Today’s issues – 10th May 2012

This post is only relevant to users of Codebase and Deploy. No other services were affected by these issues.

Today at 12.04pm, our monitoring system alerted us to some issue with one of our storage nodes. The issues initially presented themselves as extremely high load on the node in question however we were unable to access the server in order to perform any checks. Following our internal procedures for such an event, we proceeded to investigate any causes for high load and were unable to find any reason for this on our application servers.

At this point, we had to consider the possibility of a more serious error on the storage node in question. We despatched a team to the datacentre to investigate the host and discovered the server was displaying the same symptons as those experienced around 200 days ago. This meant we needed to hard reboot the server which, including a disk check, took around 25 minutes to complete and the service was back to normal at 13:15pm.

Unfortunately, there were a number of points which could have been performed better which we are already working to improve:

The failure of a single storage node had too much impact on the Codebase application. In the event a failure of a storage server, Codebase is designed to handle this by disabling repositories which are located on that server to avoid congestion on our frontend web processes. Unfortunately, this did not kick in as desired and our web processes quickly became saturated with requests for data stored on the unresponsive storage node. We have already implemented procedures to ensure that un-affected repositories remain accessible in the event of any future failures of this nature.

We were misled by our monitoring when we assumed high load on the server. The process to determine the root cause of the failure was slowed down by investigations surrounding the failed node. We have implemented internal procedures to ensure that detecting issues such as these is faster.

Emergency Maintenance. We are going to be undertaking some emergency maintenance tomorrow morning from 4.30am until 6.30am (GMT+1) which will mean that repository access is unavailable for around 1 hour for some users within Codebase. When repositories are unavailable, you will receive a message saying your repository is temporarily unavailable.

I’d like to take this opportunity to apologise for any inconvenience caused by these issues and assure all our customers & users that, as always, we’re working tirelessly to ensure the Codebase platform remains fast & stable.

CSV Ticket Export

We’ve recently added the ability to export ticket reports as CSV file. Just create a ticket report using the quick or advanced search tools as normal, then click the new link at the bottom of the table which will export all of the selected tickets and columns to a CSV file.

There you have it. Your tickets exported as a CSV file.

Updates to Time Tracking

Today we’ll be introducing some changes to the way that billed time sessions work and at the same time we’ll be deploying more flexible ways to filter your time sessions.

Billed Time Sessions

We’ve had frequent requests for billed time session groups to be account-wide, so that time sessions from different projects can be grouped together under one billed group for billing purposes.

Your time session groups will now be able to contain sessions from any project in your account.

Account Wide Filtering

As well as making time session groups account-wide, we’ve also improved the filtering ability to filter time sessions from the global report. Just hit the ‘Filter Time Sessions’ button and you’ll be able to search on Occurred On, User, Company, Ticket Number, Milestone or Billed Group.

We hope these updates make time sessions easier to use and more flexible.

Temporarily Disable Users

One of the more frequent feature requests we get for Codebase is the ability to temporarily disable users. Starting today you’ll be able to do just that. The user management page has a new “Disabled” button nestled between “Edit” and “Delete”.

Selecting this link will immediately log a user out and revoke their access to the web interface, repository acces, RSS feeds and the API. Additionally, a user will not receive any watcher notifications while they’re disabled.

To re-enable a user, just go back to the user management screen and click “Enable”.

Codebase user profile pages

One of the most requested features we’ve been asked to bring back from Codebase 3 is the user profile page. This was a page which provided information and statistics about a user. We thought it was time we included this feature in the current version of Codebase, so we have!

Starting today, all of the links for a user will direct you to their user page, rather than to their primary email address (don’t worry, if you want to find their email address you can get it from their user page).  The user page will contain a log of their recent activity, a list of their current assignments, a few brief statistics, and contact details for that user.

Some of our users have also asked us for greater privacy options in Codebase. As such, 3 new permissions have been added to user roles along with the user page.

  • View a users contact details allows you to show or hide a users e-mail address from users in that role
  • View a users full name allows you to obfuscate other users names by truncating their surnames (so Dan Wentworth would become Dan W)
  • View a users history, statistics and assignments allows you to view in depth information on a users page relating to their recent activity

All of these permissions will be enabled for existing users with the exception of those which don’t currently have any permissions (such as the default ‘Users’ role)

HipChat Integration

We’ve recently started to use HipChat as replacement to our internal IRC server for discussions between staff and it’s fantastic. It provides significantly more functionality than both IRC and Campfire allowing us to easily share information around the office and with our staff working from home.

Consequently, Codebase now supports HipChat as a notification method for repository pushes, commits, deployments as well as ticket updates. You can configure these notifications from the Notifications page which can be found from the Project Properties page. Once you have setup the notifications, you will receive messages into your desired chat room whenever an action is carried out.

The individual messages are colour coded – pushes & commits are shown in purple, deployments in red and ticket updates in green. These will not automatically notify users so they won’t constantly be bothered with notifications each time an action takes place – the message will just appear quietly in the room.

If you currently use Campfire or IRC for this, I’d strongly encourage you to take a look at it. It’s significantly more feature-rich than the alternatives.

Disclaimer: the link to HipChat above is an affiliate link – HipChat will make custom emoticons in exchange for referrals and we’ve got some ideas for icons we’d love to see in the HipChat interface.

The new project assignments interface

We’ve just overhauled the Codebase project assignments & permissions interface to make things more user friendly and easier to use. The previous system had two separate interfaces – one for setting user permissions and the other for selecting which users should be assigned to projects. The new interface allows you to assign users and select what permissions they should be granted at the same time.

If you used to find the old method a bit of a headache, I encourage you to take a look and see what you think of the improvements. Just head over to a project and select Manage Project Assignments from the yellow settings box on the right.

Export your Codebase data at will

We’ve just pushed a new feature to Codebase which will allow users to export an entire archive of all the data stored within their account. All exports are sent to you as a ZIP file containing an array of XML files and, if requested, your repository data.

In order to request an export of your data you’ll need to select Settings followed by Export Account Data from the menu at the top of the screen – to access this option you must have the appropriate permission defined as part of your access role, by default only account administrators have this level of access. Once requested, we will create the export and send you an email with a link to download the file as soon as it has been completed. To use this feature you will need a package of Small or higher.

SSL for all Codebase users

There have been a lot of discussions recently regarding the use of SSL, along with various high profile attacks against users of services such as Facebook where account credentials, and more commonly, cookies have been stolen.

Using an insecure connection to access any web application can pose two security risks. Firstly, and most commonly understood is the risk of sending passwords over an insecure connection. Secondly, the more widely exploited issue of cookies being sent over an insecure connection, even after authentication is completed and users are simply browsing a site.

The risk posed by these issues is extremely low when browsing from a trusted, wired Internet connection, however increase in use of insecure wireless hotspots means that these risks need to be addressed.

Codebase has always offered SSL connectivity as an option to paid accounts, and has allowed users to choose whether to connect with SSL or not. From today, we have made several changes to the way we offer SSL, in order to ensure that all customers are protected from the issues.

  • SSL is available to all customers, including free account holders
  • All connections will be redirected to SSL, ensuring that secure connectivity is preserved when using old bookmarks
  • The option to use SSL will be enabled on all accounts unless users explicitly choose to disable it from their account settings