30/01/2012 Monday
17:00 We are aware of a problem with some websites on this server not loading. Our technicians are working to fix this problem as quickly as possible.
17:15 Our technicians are still working on this server. It’s gradually stabilising but it’ll take a little more time to come back to normal service.
17:26 This problem is now resolved. We’re sorry for any inconvenience caused.
22/01/2012 Sunday
23:00 We are aware of a problem with some websites on this server not loading. Our technicians are working to fix this problem as quickly as possible.
23:14 Our technicians are still working on stabilising this server, however all websites should be working again.
23:18 This problem is now resolved. We’re sorry for any inconvenience caused.
Maintenance Notice – cpanel64 Server, London City datacentre
The ‘cpanel64′ web server will be under-going web server updates and will require a reboot.
Maintenance Window : Wednesday 18 January from 23:00 to 23:15 local time
Time Zones: 11pm local (UK time)
Reason: We will be installing new software and making customizations to this web server to increase performance and security. The changes will require a server reboot and therefore the server will be unavailable for approx. 5-10 minutes during the maintenance window. We apologise for any inconvenience.
If you have any questions about this please let us know.
16/01/2012 Monday
13:45 We are aware that this server has stopped responding. Our technicians are working to fix this problem as quickly as possible.
13:48 This server has stopped responding and we have rebooted it.
13:49 We are monitoring the server startup.
13:54 All services are now back online. We apologise for any inconvenience that this disruption has caused.
12/01/2012 Thursday
13:31 We are aware that this server has stopped responding. Our technicians are working to fix this problem as quickly as possible.
13:36 One of the technicians at the datacentre accidentally rebooted the server, however it is now back online and all services are responding. We apologise for any inconvenience that this disruption has caused.
05/01/2012 Thursday
12:31 We are aware of a technical problem with the cpanel54 server. Our technicians are investigating the issue.
12:40 This is also affecting the cpanel53 servers (and to a much lesser extent cpanel51 and cpanel52).
12:45 We have determined that there is a bottleneck with the SAN storage devices provided by datacentre. We are working with their technicians to resolve the problem. We’ll update you as we know more.
14:00 At present most services are running normally, but we are aware of a slow-down (in particular on the cpanel54 server). We’re working to resolve the problem and we will be sending an email to all customers with more information. We will also keep this post updated.
15:00 Thank you for your patience. Our technicians are putting every effort into stabilising servers and resolving this technical problem in the shortest possible time. At present most servers are online but we have unfortunately had to disable email/database services on some servers to prevent excessive IO (input-output) affecting the storage devices and causing them to crash
16:00 At present most servers are back online and working at near-normal levels. We are monitoring everything very closely. The vast majority of services are working normally but there is a large backlog with email delivery on some servers. If you are having any difficulties please make sure you have submitted a support ticket and we will try to respond to you as soon as possible.
Update Please note an email has been sent to all customers effected informing them of our decision to move two servers (cpanel53 and cpanel54) from our Maidenhead datacentre to two brand new servers (cpanel63 and cpanel64) in our London City datacentre. The migration will begin immediately at 5pm Thursday 5th January and continue until complete. If you have any questions or concerns, please read the full email and reply to our support team so we can assist you. It’s important that you open a support ticket for any technical problems so we can make sure they are related to this issue.
—————–
06/01/2012 Friday
08:45 The migration to the new servers has been taking place over night and it will probably continue for the next few days. So far we have concentrated on moving some of the largest websites first in order to help stabilise the old servers. We’ll be monitoring the old servers closely this morning. At present all services are working at near-normal levels. This means the servers are relatively stable at the moment and we’re not aware of any technical problems, except the two servers (cpanel53 and cpanel54) being slower than normal.
09:20 Unfortunately we are seeing intermittent access to websites/email on cpanel53 and cpanel54. We’re doing everything we can to mitigate this problem. Once your website has been migrated to the new servers (cpanel63 and cpanel64) you should notice an immediate increase in performance and speed.
10:30To give customer’s a better idea of the current problem with the SAN storage, this graph shows a server operating normally on the left and the cpanel53 server on the right over the same 6-hour period. The orange part of the graph represents IOWAIT and shows that the system is waiting almost 100% of the time for the SAN storage to service requests. The server is slow because of the wait time on the SAN storage. The cpanel54 graph is almost identical to cpanel53 and it is waiting 98% of the time.
12:00 There is intermittent access to websites/email on cpanel53 and cpanel54. We’re doing everything we can to mitigate this problem. As more websites are migrated to the new servers (cpanel63 and cpanel64), we hope that this situation will improve.
13:30 We are seeing some improvement to server loads, however please note access is likely to remain intermittent/slow and there is a large backlog on our email servers.
18:00 Both servers are slow but stable. All website/email/database services are online. The migration to the new servers will continue over-night and throughout the weekend.
—————–
07/01/2012 Saturday
16:00 The migration is 60% complete. The new servers are proving to be significantly better performance/speed. Any clients still on the old servers may notice it’s still slow but all services are stable.
—————–
08/01/2012 Sunday
14:00 The migration from cpanel54 to cpanel64 is now 100% complete. The migration from cpanel53 to cpanel63 is still in progress. We anticipate it will complete tomorrow. If you’re experiencing any technical issues with your website please make sure you’ve opened a support ticket so our technical team can assist you.
23:30 The migration from cpanel53 to cpanel63 is now 100% complete and so the overall migration is finished. If you’re experiencing any technical issues with your website please make sure you’ve opened a support ticket so our technical team can assist you.
—————–
An update will follow on Monday morning.
04/01/2012 Wednesday
11:45 The cpanel53 and cpanel54 servers are currently receiving an inbound DDOS (Distributed Denial of Service) attack which can affect network speed and latency, which means some sites may load slower than normal or time out.
12:20 We’re working to block this attack. We have blacklisted a large number of offending IP addresses (mostly in Poland and China) and we are continuing to monitor the network traffic to ensure that we are blocking them all. Unfortunately access to the servers currently remains intermittent and slow.
13:20 This attack is on-going. At this time customers should be able to access their website/email but service may still be slow and intermittent.
15:00 Unfortunately this problem is on-going. Both servers are running a lot slower than we would like and we’re working to block the attack and stabilise the servers. The attack is still large and the offending IP addresses are constantly changing which makes it difficult to mitigate.
16:00 The DDoS attack is still on-going but it has significantly decreased in the last 20 minutes (we’re blocking a very large number of offending IP addresses). Services on both cpanel53 and cpanel54 are beginning to stabilise. We have temporarily disabled the MySQL servers while other services (HTTP and email) are brought back online and stabilised. This will mean websites using MySQL will remain inaccessible (displaying a database error message) but we hope to enable MySQL access again shortly. There is a large backlog of email and all other services are extremely busy after being intermittently unavailable during the attack, and our priority is to keep the servers stable. At an appropriate time we will enable MySQL access and post an update here.
16:40 We have re-enabled MySQL access on cpanel54 server. We are monitoring the server closely.
17:20 We are enabling access to MySQL on cpanel53 server. Please note it will be slow for a while because of high demand.
17:30 We believe this issue is now resolved and we will continue to monitor the servers closely. We are very sorry for any disruption this has caused.
03/01/2012 Tuesday
12:22 We are aware that this server has stopped responding. Our technicians are working to fix this problem as quickly as possible.
12:26 This server has stopped responding and we have rebooted it.
12:27 We are monitoring the server startup.
12:35 It’s taking a little bit longer than normal to startup but we expect this server to be back online shortly.
12:42 All services are now back online. We apologise for any inconvenience that this disruption has caused.