Today at 12.04pm, our monitoring system alerted us to some issue with one of our storage nodes. The issues initially presented themselves as extremely high load on the node in question however we were unable to access the server in order to perform any checks. Following our internal procedures for such an event, we proceeded to investigate any causes for high load and were unable to find any reason for this on our application servers.

At this point, we had to consider the possibility of a more serious error on the storage node in question. We despatched a team to the datacentre to investigate the host and discovered the server was displaying the same symptons as those experienced around 200 days ago. This meant we needed to hard reboot the server which, including a disk check, took around 25 minutes to complete and the service was back to normal at 13:15pm.

Unfortunately, there were a number of points which could have been performed better which we are already working to improve:

The failure of a single storage node had too much impact on the Codebase application. In the event a failure of a storage server, Codebase is designed to handle this by disabling repositories which are located on that server to avoid congestion on our frontend web processes. Unfortunately, this did not kick in as desired and our web processes quickly became saturated with requests for data stored on the unresponsive storage node. We have already implemented procedures to ensure that un-affected repositories remain accessible in the event of any future failures of this nature.

We were misled by our monitoring when we assumed high load on the server. The process to determine the root cause of the failure was slowed down by investigations surrounding the failed node. We have implemented internal procedures to ensure that detecting issues such as these is faster.

Emergency Maintenance. We are going to be undertaking some emergency maintenance tomorrow morning from 4.30am until 6.30am (GMT+1) which will mean that repository access is unavailable for around 1 hour for some users within Codebase. When repositories are unavailable, you will receive a message saying your repository is temporarily unavailable.

I’d like to take this opportunity to apologise for any inconvenience caused by these issues and assure all our customers & users that, as always, we’re working tirelessly to ensure the Codebase platform remains fast & stable.

If you’re adding a Mercurial repository from Codebase it should automatically appear in your repositories list. If you’re configuring your repository manually then select “Mercurial” from the SCM Type menu and it will be added the same as any other Deploy project.

Simply grab the arrows on the left and drag the command into it’s new position. Your commands will now execute in this order.

If you already have a Codebase account you are entitled to a free Deploy account – you can signup for an account by logging into your Codebase account, selecting My Account and then clicking the link in the blue box provided.

We’re very excited about Deploy and hopefully you will be too. As always, if you have any questions, feel free to drop an email to our team at support@deployhq.com or follow us on Twitter. Don’t forget to keep an eye on the blog and we’ll be publishing updates whenever they occur.