At this time we've cleared up the backlog and are preparing a postmortem for today's incident.
Thanks for your patience as we worked to resolve this.
Posted Aug 31, 2015 - 22:48 UTC
Monitoring
We determined that a database deadlock was the source of the backlog. The deadlock was removed and the worker service began processing builds as expected.
We are still working through the backlog and monitoring things, we'll update again when the backlog is cleared out.
Thanks for your patience.
Posted Aug 31, 2015 - 22:22 UTC
Identified
We've identified that one of the workers for the service that is responsible for handle job status updates, which marks jobs as finished and caused new jobs to run, is not process jobs properly and we are digging into why it is failing to do so.
Posted Aug 31, 2015 - 21:57 UTC
Update
We're working to identify the cause of the delay in builds.
Posted Aug 31, 2015 - 21:29 UTC
Investigating
We are investigating slow build processing on private repositories.