Backlogs for .org builds have cleared on all our infrastructure. We have resumed full operational status.
Posted Jul 25, 2016 - 23:54 UTC
Update
The backlog on .org builds for docker and gce have cleared. We are continuing to monitor all .org OSX builds on our Mac Infrastructure.
Posted Jul 25, 2016 - 23:09 UTC
Monitoring
Our queueing bottlenecks have been resolved and scheduling times have returned to normal. Queues on all infrastructures are processing at a faster rate. We will continue to monitor the backlog.
Posted Jul 25, 2016 - 23:06 UTC
Investigating
Further investigation has shown slow response times in our AMQP message broker for both Scheduler and Hub for our .org builds. We are investigating further and have scaled out to remediate. Backlogs are continuing to process.
Posted Jul 25, 2016 - 22:03 UTC
Monitoring
Additional fixes have been deployed to our build scheduler, and backlogs continue to be processed.
Posted Jul 25, 2016 - 19:48 UTC
Identified
We have identified performance issues in our build scheduler and deployed a fix. We are starting to process the backlogs on all infrastructures.
Posted Jul 25, 2016 - 19:00 UTC
Investigating
We are investigating delays in build queueing which are impacting .org users.