The backlog for .org OS X builds has cleared. 👍🎉 Resolving incident.
Dec 3, 05:31 UTC
The backlog on travis-ci.com has cleared, we're still monitoring the backlog on travis-ci.org.
Dec 2, 18:18 UTC
We're continuing to process through the backlog without issues, but due to the size of the backlog there are still long delays for builds to run. We're continuously monitoring the situation in order to maintain the best possible throughput.
We're truly sorry for this lengthy interruption to your builds and we will be publishing a postmortem next week.
Dec 2, 17:11 UTC
We are beginning to process backlog at full capacity and continue to monitor closely.
Dec 2, 12:03 UTC
We are slowly starting up capacity and resuming builds.
Dec 2, 11:29 UTC
We identified an issue that is preventing build VMs form booting and are working on a fix.
Dec 2, 11:14 UTC
We are experiencing some issues while restoring OSX capacity and are investigating.
Dec 2, 10:48 UTC
We are restarting the workers for our .org open source OS X builds and monitoring to ensure the system is stable.
Dec 2, 09:46 UTC
Our Infrastructure provider is placing our hypervisor host under maintenance while they perform additional clean-ups.
Dec 2, 06:46 UTC
Our infrastructure provider is doing a rolling restart of all our physical hardware.
Dec 2, 06:09 UTC
Our infrastructure provider is resolving an issue with Hypervisor hosts. All OS X Builds are stopped while they take down hosts, reload, and migrate.
Dec 2, 05:04 UTC
OS X builds are running at full capacity for .com and .org. Monitoring.
Dec 2, 04:45 UTC
Host hypervisors have been restarted, OS X builds are resuming first for .com, then .org
Dec 2, 04:27 UTC
We've stopped all OS X workers processing jobs for .com and .org while we stabilize our hypervisor hosts
Dec 2, 03:21 UTC
OS X .org workers have restarted and resuming jobs at full capacity.
Dec 2, 02:40 UTC
We are restarting the workers for our .org open source OS X builds. Please stand by as jobs will resume shortly after.
Dec 2, 02:06 UTC
OS X .com backlog has cleared. Still processing open source OS X builds.
Nov 30, 23:54 UTC
Resumed OS X builds at full capacity, monitoring performance.
Nov 30, 18:15 UTC
We have resumed partial OS X builds at reduced capacity, and placing some of our build vm hosts in maintenance mode while we continue to monitor performance issues.
Nov 30, 17:37 UTC
Temporary stoppage on .org and .com OS X builds to drain VM pool again, after which we will restart the flow of jobs at reduced capacity
Nov 30, 16:45 UTC
We are re-escalating this incident as we are experiencing high load on our hypervisor server, which is affecting all OS X build vms for .org and .com
Nov 30, 15:50 UTC
Public repos are back to full capacity. We are working through the accumulated backlog.
Nov 30, 06:12 UTC
The private repo backlog has cleared. We are continuing to run public repos at reduced capacity.
Nov 30, 04:21 UTC
We have returned to full capacity for private repos.
Nov 30, 03:28 UTC
We are still observing VM leakage, albeit very slight. We are going to remain at half capacity and reassess in 1 hour.
Nov 30, 02:44 UTC
We have restarted jobs at half capacity and are monitoring VM lifecycle metrics.
Nov 30, 01:45 UTC
We have stopped all jobs in order to drain the VM pool again, after which we will restart the flow of jobs at reduced capacity while monitoring VM leakage.
Nov 30, 00:54 UTC
We are re-escalating this incident given ongoing issues with VM lifecycle management.
Nov 30, 00:37 UTC
We have resumed all OS X builds on .org and .com at full capacity, and will start to work through the backlog while we monitor the recent changes.
Nov 29, 20:04 UTC
Temporary stoppage on .org and .com OS X builds. We're pushing fixes to our VM cloud manager and hypervisor client, and will need to restart these services.
Nov 29, 19:40 UTC
OS X builds for .org and .com are processing at 50% capacity while we continue to debug our VM cloud manager issues.
Nov 29, 17:49 UTC
All the OS X workers are back online. Jobs will be delayed until we’re able to catch up through the backlog of jobs waiting to be built.
Nov 29, 17:23 UTC
We found an issue with our virtual machine cloud manager. We’ve now brought back up part of the workers and we’re slowly resuming OS X jobs.
Nov 29, 17:02 UTC
We’ve stopped all OS X builds to investigate further and are working on restoring them as quickly as possible.
Nov 29, 16:39 UTC
We’re experiencing issues booting OS X jobs. This is causing severe build delays in our OS X infrastructure. We are currently investigating and will post an update as soon as we know some more.
Nov 29, 16:30 UTC