The backup Servology BGP router has suffered a disk fault which has caused its root filesystem to go read-only. This does not immediately impact its operation as a router as it does not need to write to disk for normal operation, but there is the risk that the router may fail entirely; for example if it suffers a power outage, it may not be able to boot up again afterwards.
Please note that even if this router does fail entirely, this should still not impact user-visible service; the router was already in backup mode, with traffic diverted away from it. It would normally only handle network traffic if there’s a fault elsewhere in the network. If the backup router failed it would mean that our backup upstream connection would be unusable, and we would be without redundancy for upstream connectivity and colocation routing, putting these services into an “at risk” condition.
Because this incident is effectively twice removed from actually affecting service, I have decided not to immediately replace the faulty disk, but instead dedicate time to the work to replace the faulty router with a newer machine, a project which has been underway for some time now and is close to completion.
I expect to announce maintenance windows soon to allow me to integrate the replacement routers into the existing iBGP, VRRP and VLAN/firewall setups, after which they should be able to take over the functions of the current routers.