Tag Archives: monitoring

How we monitor our resources

Posted on by
2

There are lots of hosted tools available which will monitor the resources you provide for your applications, however they all come with a high price tag when running any sort of serious infrastructure. At aTech we monitoring everything internally using a combination of tools including Nagios and Cacti - installing both of these tools is easy although configuring them can be a bit confusing until you get you head around how both systems work. We use these tools for two different purposes; Nagios is primarily used for making sure everything is online and notifying our team should we detect any issues whereas Cacti is used to keep track of historical statistics and plot them onto charts for viewing at a later date.

As part of our live monitoring, we monitor a number of key metrics across all our servers including:

  • Current load, memory, hard drive & swap usage on all servers
  • The status of any RAID arrays within any of our physical servers
  • The number of running processes and number of zombie processes
  • The status of key services – which actual services depends on the server being monitored – for example, on Codebase we monitor HTTP & SSH as these are the key public facing services whereas on Deliver we monitor SMTP.
In addition to these, we also monitor a number of business logic metrics which allows us to identify any issues which may arise without actually taking things offline. For example, a backlog of jobs in any of our background job runners. For more information about this, see our blog post about it.
In order to keep track of busy periods and identify potential issues we also keep track of statistics and plot them onto graphs. We have a number of different graphs:
  • The numbers and types of MySQL queres which are executed on all our database servers (see above).
  • The volume & types of jobs which are executed by any of our applications which include a background runner (see below).
  • The number of requests processed by our load balancers.
  • The power usage of all our data centre equipment.
  • All network bandwidth information from our routers, firewalls and switches.

All this information allows us to keep our fingers on the pulse of our entire infrastructure and all our applications. The information is available to all staff through our intranet and also projected onto a wall in our office allowing us to react immediately to any issues during office hours and we’ll automatically receive text alerts outside of office hours.

The screenshot above shows the display which is projected onto our office wall and includes our most common and important machines.

Monitoring more than just services

Posted on by
Reply

Last week, we had an issue within Codebase which stopped some users from being able to view tickets within their projects. Unfortunately, we weren’t immediately aware of the situation because our monitoring system was reporting everything was online and operating successfully. The only real indication to us that something was wrong here was the volume of support requests which had been submitted to our support site.

In light of this, we have now added some additional monitoring which will highlight any unusual behaviour within any of our applications, for example:

  • Any large volume of discussions submitted to our support site within an hour period
  • Any large volume of application exceptions logged by our exceptions service
  • Any build-up of unprocessed background jobs in Codebase or Deploy
  • Any build-up of undelivered or pending e-mail in Deliver

As we now monitor these, our team will be alerted by text message/SMS whenever an issue occurred ensuring that the appropriate action can be taken without delay.