OSX infra instability. All OSX builds are stopped
Incident Report for Travis CI
Resolved
We've been able to work through the backlog and public OS X builds are now running as usual. Next update will be a detailed postmortem in the next following days. Thank you all for your patience!
Posted 10 months ago. Aug 26, 2016 - 12:56 UTC
Update
The backlog of public OS X builds continues to decrease. We'll post another update when the queue cleared or in approximately 2 hours. Thank you.
Posted 10 months ago. Aug 26, 2016 - 10:32 UTC
Update
The backlog for public OS X continues clearing. We are monitoring the development closely, and will provide an update in 2 hours.
Posted 10 months ago. Aug 26, 2016 - 08:10 UTC
Update
We are making good progress in catching up on the travis-ci.org OSX backlog. We'll continue to provide updates until the backlog is cleared.
Posted 10 months ago. Aug 26, 2016 - 00:25 UTC
Monitoring
We're continuing to process the travis-ci.org backlog and we expect to be caught up within approximately 12 hours. We'll be preparing a postmortem, including steps we are taking to prevent this kind of issue in the future, in the coming days. We'll continue to provide updates until the backlog is cleared.
Posted 10 months ago. Aug 25, 2016 - 21:43 UTC
Update
In order to help with the OSX backlog, https://www.traviscistatus.com/incidents/4mvp857qx8bw, we will be cancelling all builds older than 12 hours. Please email support@travis-ci.com if you have any questions.
Posted 10 months ago. Aug 25, 2016 - 19:16 UTC
Update
We apologize for the lack of an update within the promised time. At this time we've cleared the backlog for travis-ci.com OSX builds, but still have a significant backlog for travis-ci.org. We are looking into how we can help improve catching up on the .org backlog.
Posted 10 months ago. Aug 25, 2016 - 18:42 UTC
Update
As things have remained stable overnight, we are beginning to ramp up further capacity for OSX builds. As we see the impact of this on throughput, we'll be able to begin determining and ETA for when we'll clear the backlog. We'll provide another update in 60 minutes or sooner if we have updates.
Posted 10 months ago. Aug 25, 2016 - 14:55 UTC
Update
We've resumed builds with partial capacity and will work on processing the large backlog we've accumulated today, throughout the night, at a partial rate. Assuming things remain stable throughout the night (PST), we'll beginning ramping up throughput in the early morning (PST). At this time we don't have an ETA for when we'll be finished with the backlog.
Posted 10 months ago. Aug 25, 2016 - 05:34 UTC
Update
We've reached a point where all of our infrastructure tests are passing as expected. We are beginning to slowly resume running queued builds. We'll provide another update in 60 minutes or sooner if we have updates.
Posted 10 months ago. Aug 25, 2016 - 04:59 UTC
Update
We're sorry for the continued delays, getting our OSX infra working again is our highest priority right now. We are working to resolve some additional unexpected issues. We'll provide another update in 60 minutes or sooner if we have updates.
Posted 10 months ago. Aug 25, 2016 - 04:07 UTC
Update
We are sorry for the delays and getting our OSX infra working again is our highest priority right now. We are working to resolve some additional unexpected issues. We'll provide another update in 60 minutes or sooner if we have updates.
Posted 10 months ago. Aug 25, 2016 - 03:07 UTC
Update
We are working to resolve some additional unexpected issues. We'll provide another update in 60 minutes or sooner if we have updates.
Posted 10 months ago. Aug 25, 2016 - 01:28 UTC
Update
Getting to the point where we can start builds is taking longer than initial anticipated. We'll provide another update in 60 minutes or sooner if we have updates.
Posted 10 months ago. Aug 25, 2016 - 00:31 UTC
Update
We are still in the process of working to resolve instability related issues and hope to begin a partial resumption of builds within 60 minutes.
Posted 10 months ago. Aug 24, 2016 - 23:00 UTC
Identified
We've identified issues with multiple OSX build hosts and we're working with our infrastructure provider to resolve this, as well as performing some maintenance tasks to help improve stability. Builds will continue to be paused while we work through this.
Posted 10 months ago. Aug 24, 2016 - 19:04 UTC
Investigating
All OSX builds are stopped due to instability in our OSX infrastructure. We are investigating now
Posted 10 months ago. Aug 24, 2016 - 18:34 UTC