OSX infra instability. All OSX builds are stopped

Incident Report for Travis CI

Resolved

We've been able to work through the backlog and public OS X builds are now running as usual. Next update will be a detailed postmortem in the next following days. Thank you all for your patience!
Posted Aug 26, 2016 - 12:56 UTC

Update

The backlog of public OS X builds continues to decrease. We'll post another update when the queue cleared or in approximately 2 hours. Thank you.
Posted Aug 26, 2016 - 10:32 UTC

Update

The backlog for public OS X continues clearing. We are monitoring the development closely, and will provide an update in 2 hours.
Posted Aug 26, 2016 - 08:10 UTC

Update

We are making good progress in catching up on the travis-ci.org OSX backlog. We'll continue to provide updates until the backlog is cleared.
Posted Aug 26, 2016 - 00:25 UTC

Monitoring

We're continuing to process the travis-ci.org backlog and we expect to be caught up within approximately 12 hours. We'll be preparing a postmortem, including steps we are taking to prevent this kind of issue in the future, in the coming days. We'll continue to provide updates until the backlog is cleared.
Posted Aug 25, 2016 - 21:43 UTC

Update

In order to help with the OSX backlog, https://www.traviscistatus.com/incidents/4mvp857qx8bw, we will be cancelling all builds older than 12 hours. Please email support@travis-ci.com if you have any questions.
Posted Aug 25, 2016 - 19:16 UTC

Update

We apologize for the lack of an update within the promised time. At this time we've cleared the backlog for travis-ci.com OSX builds, but still have a significant backlog for travis-ci.org. We are looking into how we can help improve catching up on the .org backlog.
Posted Aug 25, 2016 - 18:42 UTC

Update

As things have remained stable overnight, we are beginning to ramp up further capacity for OSX builds. As we see the impact of this on throughput, we'll be able to begin determining and ETA for when we'll clear the backlog. We'll provide another update in 60 minutes or sooner if we have updates.
Posted Aug 25, 2016 - 14:55 UTC

Update

We've resumed builds with partial capacity and will work on processing the large backlog we've accumulated today, throughout the night, at a partial rate. Assuming things remain stable throughout the night (PST), we'll beginning ramping up throughput in the early morning (PST). At this time we don't have an ETA for when we'll be finished with the backlog.
Posted Aug 25, 2016 - 05:34 UTC

Update

We've reached a point where all of our infrastructure tests are passing as expected. We are beginning to slowly resume running queued builds. We'll provide another update in 60 minutes or sooner if we have updates.
Posted Aug 25, 2016 - 04:59 UTC

Update

We're sorry for the continued delays, getting our OSX infra working again is our highest priority right now. We are working to resolve some additional unexpected issues. We'll provide another update in 60 minutes or sooner if we have updates.
Posted Aug 25, 2016 - 04:07 UTC

Update

We are sorry for the delays and getting our OSX infra working again is our highest priority right now. We are working to resolve some additional unexpected issues. We'll provide another update in 60 minutes or sooner if we have updates.
Posted Aug 25, 2016 - 03:07 UTC

Update

We are working to resolve some additional unexpected issues. We'll provide another update in 60 minutes or sooner if we have updates.
Posted Aug 25, 2016 - 01:28 UTC

Update

Getting to the point where we can start builds is taking longer than initial anticipated. We'll provide another update in 60 minutes or sooner if we have updates.
Posted Aug 25, 2016 - 00:31 UTC

Update

We are still in the process of working to resolve instability related issues and hope to begin a partial resumption of builds within 60 minutes.
Posted Aug 24, 2016 - 23:00 UTC

Identified

We've identified issues with multiple OSX build hosts and we're working with our infrastructure provider to resolve this, as well as performing some maintenance tasks to help improve stability. Builds will continue to be paused while we work through this.
Posted Aug 24, 2016 - 19:04 UTC

Investigating

All OSX builds are stopped due to instability in our OSX infrastructure. We are investigating now
Posted Aug 24, 2016 - 18:34 UTC