At this time everything is operating as expected and remedial automation we've deployed is working well. We will continue to monitor the situation and open a new incident in case of any further issues.
Posted Jun 20, 2015 - 05:27 UTC
Update
We have caught up on the backlog of Mac build jobs and are seeing steady processing of new jobs.
We are going to continue monitoring things closely for a while longer but do not anticipate further issues at this time.
Posted Jun 20, 2015 - 04:26 UTC
Update
We've stabilized our Mac build infrastructure and are currently working to get through our backlog of jobs. We'll continue to monitor things closely until we see our backlog has cleared out.
Posted Jun 20, 2015 - 00:28 UTC
Update
We have cleared out all running VMs once more and are starting to bring more capacity online again. Our expectation is that the changes we've pushed will mitigate the overload that caused runaway queue buildup over the past 20 hours.
Posted Jun 19, 2015 - 21:27 UTC
Update
We are shutting down currently running jobs in order to bring higher capacity online with some patches that we're expecting will result in lower impact on the underlying virtualization infrastructure. Interrupted jobs will be restarted momentarily.
Posted Jun 19, 2015 - 19:44 UTC
Update
We are still working to bring additional capacity online. For those of you with clogged build queues due to this outage that include Linux builds, we recommend canceling the Mac jobs in question and then restarting them after our Mac capacity issues are resolved.
Posted Jun 19, 2015 - 18:43 UTC
Update
We are in the process of bringing more capacity online while also working to address ongoing errors in the current capacity infrastructure. Jobs are being processed, but at a lower than normal rate, and with a backlog of ~250 jobs.
Posted Jun 19, 2015 - 16:36 UTC
Monitoring
We have completed the emergency maintenance and are monitoring while jobs start to flow again.
Posted Jun 19, 2015 - 14:08 UTC
Update
We're continuing to work on the emergency maintenance, which is now expected to exceed the original estimate of less than an hour.
Posted Jun 19, 2015 - 13:35 UTC
Identified
We've identified the issue and will be performing emergency maintenance on our Mac cluster momentarily. Running jobs will be interrupted and automatically requeued once we are back online. We expect to be back online within an hour.
Posted Jun 19, 2015 - 12:53 UTC
Investigating
We're investigating continued capacity issues with the Mac infrastructure. We've escalated this to our infrastructure provider and are working on getting full capacity back online as soon as possible.
Posted Jun 19, 2015 - 11:39 UTC
Monitoring
The production capacity is back online and we have reallocated capacity a bit to work through the largest backlogs more quickly.
Posted Jun 19, 2015 - 03:15 UTC
Identified
We have identified an issue with our Mac builds on travis-ci.org and are investigating further.