[11:45] Firstly, thank you to everyone for your patience on this. This incident has reminded me just how much complexity and technical debt has built up in the Servology estate over the years, including in servers taken on from previous providers. This is both the reason I can’t simply update the servers in question, and also why it’s taking this long to make workable plans to mitigate this problem.
However, I believe I have now identified all the places where I will need to implement blocks to keep the existing servers safe from direct access from the internet, and I believe I have identified the paid customer sites which I need to prioritise and safely bring back online first. For all but one of these sites it should be possible to use an external WAF (Web Application Firewall) service to protect them as a temporary measure to get them back online, until they can be migrated to up-to-date web hosts. For the one remaining site I may also be able to use the same WAF with the customer’s cooperation (in this case I do not have control of the domain).
I am continuing to work on this as much as I am able to, while also looking after my mental health and handling existing commitments. I hope to have suitable blocks put in place and the web servers restarted — but not accessible from the internet due to the blocks — today. I then plan to configure external WAF protection and apply it to customer sites either today (time permitting) or over the weekend, with the hope that these sites should all be available again by Monday.
You will probably have seen that this incident has made news headlines worldwide (it was the BBC’s top story yesterday) and has resulted in precautionary outages of a huge number of web sites. This is why I am trying not to panic or beat myself up too much over the time it’s taking to resolve this matter. I hope the inconvenience of your sites being down for this long is not causing you too many problems. That said, if you need to contact me, please use the Servology support email address as usual and I will do my best to help.
Thank you all.