Author: Servology

The backup Servology BGP router has suffered a disk fault which has caused its root filesystem to go read-only. This does not immediately impact its operation as a router as it does not need to write to disk for normal operation, but there is the risk that the router may fail entirely; for example if it suffers a power outage, it may not be able to boot up again afterwards.

Please note that even if this router does fail entirely, this should still not impact user-visible service; the router was already in backup mode, with traffic diverted away from it. It would normally only handle network traffic if there’s a fault elsewhere in the network. If the backup router failed it would mean that our backup upstream connection would be unusable, and we would be without redundancy for upstream connectivity and colocation routing, putting these services into an “at risk” condition.

Because this incident is effectively twice removed from actually affecting service, I have decided not to immediately replace the faulty disk, but instead dedicate time to the work to replace the faulty router with a newer machine, a project which has been underway for some time now and is close to completion.

I expect to announce maintenance windows soon to allow me to integrate the replacement routers into the existing iBGP, VRRP and VLAN/firewall setups, after which they should be able to take over the functions of the current routers.

00:07 Connectivity between Servology and the rest of the internet is intermittent and/or broken. There appears to be some kind of problem with BGP but I am not yet sure what. I am investigating urgently.

00:21 I have disconnected the Servology network from one of our upstream providers CityFibre, and connectivity seems to have stabilised via our other upstream provider, hSo. Both upstream connections appear to have been unstable so I am unsure what was happening. I will continue to investigate.

01:56 After a period of network stability I have reconnected our BGP sessions to CityFibre, first the IPv6 session which I monitored for about 20 minutes, then the IPv4 session. Both seem to be stable and I will continue to monitor until at least 02:30, being ready to shut down the BGP sessions via emergency remote access links if they appear to cause any more problems.

02:37 I have seen no further problems. The timeline of this outage appears to have been as follows: intermittent connectivity from around 23:28 on 1 February until about midnight. At this point I logged in via emergency remote access to investigate and shut down our CityFibre BGP sessions. This caused a short full outage lasting several minutes from approx 00:01 to 00:06 on 2 February. After that, connectivity appears to have been stable, including during the reconnection of the BGP sessions at 01:29 (IPv6) and 01:53 (IPv4).

Monitoring has alerted me to some network outages which occurred between about 12:30 and 13:00 today. Over the weekend I had made some changes to the way routing processes are restarted on my core routers if they become unresponsive, and I believe this helped keep the disruption shorter than last time. I believe I’ve finally pinned down the root cause of these outages. One of the core routers is getting short of memory and it seems this causes routing processes to run slowly. (The other core router has twice the memory and does not seem to be having the same kind of problems.) Protocol timers then expire, at which point other routers shut down and reestablish their associations with the problem router, causing an outage in the meantime. On its own this would not be too disruptive, but the router’s internal monitoring also detects the routing processes becoming unresponsive, so it shuts down and restarts those processes, which takes time. To mitigate the problems I have disabled a link between two core routers for now and I expect this will reduce the memory pressure and keep things running smoothly until I can implement a more permanent fix.

I have been working recently on configuring a pair of new core routers to replace the (aging) current ones. The new routers have much faster processors and 32 times the memory of the current ones, and I have been prioritising this work since the outage last Thursday, especially over the weekend. I expect to get these new routers into service very soon now which should fully resolve the recent network problems.

16:28: While working on configuring a new router it appears I have accidentally caused a network outage on the Servology network. I’m working to determine the extent of the problem and recover operation.

16:38: It appears the Servology network is entirely offline as of 16:23 today. There does not appear to be any working method to remotely access network devices to restore service. It looks like I am going to have to go to the datacentre to manually disconnect the faulty device. I’ll post an ETA here when I have it.

17:00: ETA at datacentre 18.30

18:40: arrived, gaining access to datacentre.

18:50 I’ve unplugged the problematic machine and monitoring statuses are going green. I believe the problem has been resolved.

20:00 I believe this problem is now fully resolved.

Starting around 16:04 today for IPv6 and 16:18 for IPv4, the Servology network has suffered routing problems which appear to be caused by a temporary routing process failure and restart on one of our BGP routers. I believe the problem is now resolved.

Connectivity to and from the Servology network was degraded from approximately 11:20 to 13:00 on Friday 22 July 2022, due to faulty hardware in our main transit provider’s network. Once I had determined that the problem was caused by a fault outside the Servology network, I resolved the problem by shutting down our connection to our main transit provider and failing over to our backup transit provider. Our main transit provider declared a Major Incident and worked to stabilise their network. I reestablished our connection to them the following day in off-peak hours once it became clear that their network was once again stable. This fault affected access to several (but not all) web sites and other services hosted at Servology.

The Servology network suffered a period of degraded performance, with high levels of packet loss, between around 19:30 and 19:45 today. I believe this is now fixed.

Due to power problems at the Sovereign House datacentre, one of the three routers Servology’s network connects to at our upstream connectivity providers is currently down. This has no immediate impact on connectivity between Servology’s network and the rest of the internet, but reduces our usual levels of redundancy. This outage is outside of the Servology network and out of our control. More details are available at the status site of the upstream provider whose router is affected.

This outage appears to be resolved. The upstream router is responding once more and our BGP connection to it has been stable for half an hour now.

Future outage notifications and updates will be posted here.