Backlogs for .org builds have cleared on all our infrastructure. We have resumed full operational status.
Jul 25, 23:54 UTC
The backlog on .org builds for docker and gce have cleared. We are continuing to monitor all .org OSX builds on our Mac Infrastructure.
Jul 25, 23:09 UTC
Our queueing bottlenecks have been resolved and scheduling times have returned to normal. Queues on all infrastructures are processing at a faster rate. We will continue to monitor the backlog.
Jul 25, 23:06 UTC
Further investigation has shown slow response times in our AMQP message broker for both Scheduler and Hub for our .org builds. We are investigating further and have scaled out to remediate. Backlogs are continuing to process.
Jul 25, 22:03 UTC
Additional fixes have been deployed to our build scheduler, and backlogs continue to be processed.
Jul 25, 19:48 UTC
We have identified performance issues in our build scheduler and deployed a fix. We are starting to process the backlogs on all infrastructures.
Jul 25, 19:00 UTC
We are investigating delays in build queueing which are impacting .org users.
Jul 25, 18:02 UTC