We are back under capacity on the OS X build infrastructure and processing jobs normally once again. Thanks for your patience, and happy building.
Posted Jul 14, 2016 - 02:39 UTC
Update
The queue is still steadily draining and we are expecting to be back well within capacity once again within the upcoming hour. Thanks for sticking with us, and we'll post another update once the backlog has cleared.
Posted Jul 14, 2016 - 01:43 UTC
Update
Everything appears to be stable, but we'll continue to keep an eye on things until we get through the backlog. .com currently has no backlog, but .org has a fairly large backlog that looks like it'll continue into the early European morning. We'll post another update once the backlog is cleared. Sorry for the interruption today, and thank you for your patience.
Posted Jul 13, 2016 - 23:41 UTC
Monitoring
The new component appears to be working, so we're bringing production traffic back online while monitoring closely.
Posted Jul 13, 2016 - 22:37 UTC
Update
The switch has completed and we're working on verifying the network connectivity before bringing the workers back up.
Posted Jul 13, 2016 - 22:27 UTC
Update
We're beginning the switch to the upgraded component, and we're stopping all workers for macOS jobs in preparation for that.
Posted Jul 13, 2016 - 22:15 UTC
Update
We're continuing to work with our infrastructure provider to upgrade the overloaded network component. In the meantime, our workaround to scale down our infrastructure slightly has made things more stable, but there's still a large backlog of jobs to go through. Once we are ready to switch out the network component we're going to temporarily stop all our workers until we can verify that the new component is working correctly. We'll update the status page before that happens and once we are running jobs again afterwards.
Posted Jul 13, 2016 - 21:28 UTC
Identified
We've identified a saturation point with our network stack, which is causing frequent network disconnects and job requeues. We're working with our infrastructure provider to get the affected component upgraded. Meanwhile we're going to scale down our capacity slightly to provide some relief for the overloaded component in the hopes that this will lower the number of requeues and allow jobs to finish.
Posted Jul 13, 2016 - 19:26 UTC
Update
We're continuing to investigate OSX build delays for both .com and .org users.
Posted Jul 13, 2016 - 18:21 UTC
Investigating
We’re investigating delays in processing builds on our OSX infrastructure which are impacting .org users.