Servology network status

Due to recently-announced exceptionally serious vulnerabilities in all versions of PHP, I have decided that my only safe option is to shut down all Servology web hosting which makes use of PHP for the time being.

I have not taken this decision lightly, and I do not know how soon I will be able to turn PHP-based web hosting back on again.

I will update this page as and when I have more information.

Please accept my apologies for this outage. All indications at the present are that this is one of the most serious security vulnerabilities in history and could result in the complete take-over of any vulnerable server. I would prefer to be safe than sorry.

For more information see the Wikipedia page on PHP2Shell and references linked from that Wikipedia page.

Please note: At the moment, almost all Servology web hosting is offline as I have not been able to identify quickly enough exactly which sites use PHP and which do not, and to separate the configurations for vulnerable PHP-based sites from unaffected sites. Further updates as information is released.

[15:20] I am reading up on the mechanisms, impact and possible mitigation strategies for the vulnerabilities.

[16:15] It looks like the sury PHP repository may have fixed versions of older PHP versions, so it may be possible to patch a number of systems despite them being older than supported, and bring sites back online.

[18:00] The proxy frontend has been mostly unaffected, so hosted redirects for several customers have not had an outage. The same server hosts a few sites which are now patched and back online. This status site has also been patched and is now back online. There are several more web hosts to update and/or mitigate, some of which may not be possible to fix. In case they cannot be fixed it may be necessary to build a new web host and migrate sites across, which may take some considerable time.

[01:30] I have seen several short outages between the Servology network and the rest of the internet since midnight tonight. So far there have been outages around 3 minutes long at approximately 00:08–00:11 and 00:25–00:28, and shorter outages at around 00:56–00:57 and 01:04–01:05.

These appear to have been caused by my maintenance on my primary transit provider’s router. The notification I received a few days ago about the maintenance, I am sure said it would be 00:00–04:00 on 30 January, but it would seem it has since been edited to say 23 January. Therefore the alarms I had set for the evening of 29 January to remind myself to shift traffic to the backup upstream connection before the maintenance was in vain and I wasn’t able to mitigate this maintenance in advance.

I have now shifted traffic to the backup transit provider for the rest of the night to hopefully mitigate any problems if the maintenance causes any further connectivity glitches. I will shift it back to the primary transit tomorrow.

[11:40] I have just moved inbound and outbound IPv4 and IPv6 traffic back to Servology’s primary transit provider. Their router doesn’t appear to have had any more connectivity problems since 01:05 this morning.

The Servology network suffered a short network outage lasting from approximately 12:55 to 12:58 today. This was caused by emergency upstream network maintenance. I only received about 12 minutes’ notice of the maintenance window, and the outage occurred less than half an hour into that window. Unfortunately I wasn’t in a position to take any action to mitigate the effects of the maintenance. My apologies for this.

[15:50] At around 15:15 today a large number of near-simultaneous connections were made to one of the former-Artonezero web servers. Due to the time taken to process each request, the server’s request queue filled up and new requests were having to wait a very long time to be processed, making the server effectively unusable. I restarted the web server process at approximately 15:36, resetting all outstanding connections and allowing the service to come back to normal. I will keep an eye on it and will try to find the cause if this happens again.

[08:15] There has been an outage on several virtual servers hosted by Servology, including several servers supporting servology’s DNS, email and web services. This appears to be due to a problem with the physical host running those virtual servers. I am currently restarting the affected servers on alternative hardware. This outage will have affected most services hosted by Servology, including DNS resolution and web and email hosting.

[10:00] I believe everything was back up and running as of approximately 08:40 this morning. As far as I can tell the outage started around 07:30, although it looks like there may have been degraded performance on Servology-hosted VMs throughout the night.

[15.30] As far as I can tell, one of the former Artonezero servers may have suffered a hardware failure and stopped working. This will affect some services for former Artonezero customers, in particular email for these customers is currently not working. Support via support@servology.co.uk is also not working. I am working on determining the extent of the problems and making plans to restart affected services on alternative hardware, but as I’m not at my desk today it may take some time. I will update this post as I make progress.

[17:30] I have begun starting virtual servers on alternative hardware. Servers should be starting to come back online. I have been prioritising email servers. Depending on the server, fsck (filesystem checks) may take a considerable time.

[18:10] My datacentre provider has advised that they were carrying out work on the cooling systems in the suite today. Despite the extra temporary cooling untis they installed in the suite during the work, it seems some machines saw an increase in temperatures, in one case beyond the emergency shutdown threshold – this would be what caused today’s problems. I’m told the work is all complete now and the suite’s usual cooling is fully back in operation, and I have restarted the failed server. This machine seems to be working normally now. I expect services to be returning to normal soon.

[18:45] I believe all servers and services used by former-Artonezero customers have come back up.

[01:30] Servology’s primary transit provider is due to perform hardware maintenance to their router to which Servology connects, between 03:00 and 05:00 today. I am therefore about to adjust settings on our BGP router to move the majority of both inbound and outbound traffic away from our primary transit provider to our secondary transit provider for the duration of the maintenance, which should help to minimise disruption if the router undergoing maintenance is not shut down cleanly.

[11:00] The upstream maintenance appears not to have gone ahead. My router did not see any disconnection or other interruption to service. The transit provider declared a major incident due to a hardware failure related to a maintenance window earlier in the night, so I can only guess they have postponed the maintenance that would have affected Servology’s connection to them. I have reverted last night’s routing changes to restore traffic to its usual paths, and will keep an eye out for future maintenance by the transit provider.

[14:30] I discovered today that there have been glitches in connectivity to one or both of the Servology DNS resolvers over the last day or two. The start of the problem appears to coincide with when I installed the latest security updates on resolver B at the weekend, but as far as I can see do not appear to have been caused by those updates – for now I am assuming the timing was coincidental. The cause of the problem seems to be a faulty VRRP process on an older router – two routers were both trying to claim the same IP address on a specific subnet where some of Servology’s internal servers are hosted. I have shut down the faulty VRRP process and this appears to have fixed the connectivity glitches.

I believe the problems were confined to a few IP addresses which are statically configured on routers and distributed via OSPF, so access to most services should not have been affected. Affected services included Servology DNS resolvers and possibly some Servology NTP and SMTP servers. The IP addresses in question most likely worked fine from outside the Servology network (my monitoring server is hosted externally and did not report any problems) – I the glitches were probably only visible from local subnets.

I will continue to monitor this in case of further problems.

[00:30] The Servology colo network just suffered a connectivity outage lasting nearly 5 minutes, from 00:10 to 00:14 today. This appears to have been caused by an outage of the router we peer with at our primary transit provider, which in turn appears to be due to scheduled maintenance. I can only guess that the router was not shut down cleanly and that this is why it took several minutes for BGP to reconverge on our backup transit provider.

I have manually shut down our BGP sessions to our primary transit provider for the time being. I am going to see if I can reverse the primary/backup configuration of the two connections for the remainder of the maintenance window, such that I can bring the BGP sessions back up again (to avoid relying on a single transit for longer than necessary) without more risk than necessary of this problem happening again.

[16:00] The BGP sessions with the primary and backup transit providers are both up and running normally. I plan to shift traffic back to the primary transit tonight shortly after 22:00. There should (I hope!) be no noticeable outage, but there may be brief periods of packet loss as routing protocols reconverge.

[23:50] Reverting last night’s localpref and prepend changes and soft-refreshing BGP sessions.

[00:25] Work complete.

01:40: We’ve had a couple of glitches in connectivity which were picked up by monitoring, one at about 00:58 which lasted for a minute or two, and another at about 01:30 which lasted a little more than five minutes. I can see that Servology’s BGP sessions have not been interrupted, so can only assume this problem happened upstream of us. I will keep an eye on it.