At this time we've cleared all backlog and are processing new jobs as expected. We're not seeing any additional errors from the internal API after a restart of the services. If you see any issues please email support@travis-ci.com.
Posted Oct 17, 2015 - 08:15 UTC
Monitoring
We've enabled the full capacity for public and private OS X builds and will continue to monitor things closely until we've caught up with the backlog.
Posted Oct 17, 2015 - 07:21 UTC
Update
We've finished the aggressive cleanup and are resuming builds for both public and private builds at a reduced capacity initially and will monitoring things closely for a few minutes to ensure things are running smoothly. Thanks for your patience while we work through this.
Posted Oct 17, 2015 - 07:10 UTC
Identified
We've identified the initial issue as being due to an internal API service, that's part of our OS X virtualization stack, hitting some time-outs and returning to 504s to the service that handles the creation and build life cycle of OS X VMs. Unfortunately this particular error condition resulted in VMs being orphaned and a "VM pile up" occurred. We're in the processing of running aggressive clean up of all the orphaned VMs. Once that clean up is finished we'll resuming running builds and catch up on the backlog. We'll provide another update in 30 minutes.
Posted Oct 17, 2015 - 06:48 UTC
Investigating
We are investigating some errors in our OS X provisioning service which is causing new build VM creation for OS X builds to fail and is causing a build delay and backlog for OS X builds. We will be temporarily stopping all OS X builds as part of working to resolve this and will provide an update in 30 minutes. Thanks for your patience.