05/01/2012 Thursday
12:31 We are aware of a technical problem with the cpanel54 server. Our technicians are investigating the issue.
12:40 This is also affecting the cpanel53 servers (and to a much lesser extent cpanel51 and cpanel52).
12:45 We have determined that there is a bottleneck with the SAN storage devices provided by datacentre. We are working with their technicians to resolve the problem. We’ll update you as we know more.
14:00 At present most services are running normally, but we are aware of a slow-down (in particular on the cpanel54 server). We’re working to resolve the problem and we will be sending an email to all customers with more information. We will also keep this post updated.
15:00 Thank you for your patience. Our technicians are putting every effort into stabilising servers and resolving this technical problem in the shortest possible time. At present most servers are online but we have unfortunately had to disable email/database services on some servers to prevent excessive IO (input-output) affecting the storage devices and causing them to crash
16:00 At present most servers are back online and working at near-normal levels. We are monitoring everything very closely. The vast majority of services are working normally but there is a large backlog with email delivery on some servers. If you are having any difficulties please make sure you have submitted a support ticket and we will try to respond to you as soon as possible.
Update Please note an email has been sent to all customers effected informing them of our decision to move two servers (cpanel53 and cpanel54) from our Maidenhead datacentre to two brand new servers (cpanel63 and cpanel64) in our London City datacentre. The migration will begin immediately at 5pm Thursday 5th January and continue until complete. If you have any questions or concerns, please read the full email and reply to our support team so we can assist you. It’s important that you open a support ticket for any technical problems so we can make sure they are related to this issue.
—————–
06/01/2012 Friday
08:45 The migration to the new servers has been taking place over night and it will probably continue for the next few days. So far we have concentrated on moving some of the largest websites first in order to help stabilise the old servers. We’ll be monitoring the old servers closely this morning. At present all services are working at near-normal levels. This means the servers are relatively stable at the moment and we’re not aware of any technical problems, except the two servers (cpanel53 and cpanel54) being slower than normal.
09:20 Unfortunately we are seeing intermittent access to websites/email on cpanel53 and cpanel54. We’re doing everything we can to mitigate this problem. Once your website has been migrated to the new servers (cpanel63 and cpanel64) you should notice an immediate increase in performance and speed.
10:30To give customer’s a better idea of the current problem with the SAN storage, this graph shows a server operating normally on the left and the cpanel53 server on the right over the same 6-hour period. The orange part of the graph represents IOWAIT and shows that the system is waiting almost 100% of the time for the SAN storage to service requests. The server is slow because of the wait time on the SAN storage. The cpanel54 graph is almost identical to cpanel53 and it is waiting 98% of the time.
12:00 There is intermittent access to websites/email on cpanel53 and cpanel54. We’re doing everything we can to mitigate this problem. As more websites are migrated to the new servers (cpanel63 and cpanel64), we hope that this situation will improve.
13:30 We are seeing some improvement to server loads, however please note access is likely to remain intermittent/slow and there is a large backlog on our email servers.
18:00 Both servers are slow but stable. All website/email/database services are online. The migration to the new servers will continue over-night and throughout the weekend.
—————–
07/01/2012 Saturday
16:00 The migration is 60% complete. The new servers are proving to be significantly better performance/speed. Any clients still on the old servers may notice it’s still slow but all services are stable.
—————–
08/01/2012 Sunday
14:00 The migration from cpanel54 to cpanel64 is now 100% complete. The migration from cpanel53 to cpanel63 is still in progress. We anticipate it will complete tomorrow. If you’re experiencing any technical issues with your website please make sure you’ve opened a support ticket so our technical team can assist you.
23:30 The migration from cpanel53 to cpanel63 is now 100% complete and so the overall migration is finished. If you’re experiencing any technical issues with your website please make sure you’ve opened a support ticket so our technical team can assist you.
—————–
An update will follow on Monday morning.
