We've worked our way through the backlog and everything is back to normal. Thank you for your patience!
Posted Oct 18, 2015 - 06:30 UTC
Monitoring
We've been ramping up capacity since the last status page update while monitoring closely for any errors. We're now back up to full capacity and everything seems to be working. We're going to continue monitoring everything closely as the backlog on travis-ci.org drains. There is no backlog for builds on travis-ci.com.
Posted Oct 18, 2015 - 06:07 UTC
Identified
We have sorted out the fallout from the broken vSphere setup, and will be starting the worker processes back up momentarily, at which point we will begin working through the job backlog.
Posted Oct 18, 2015 - 05:23 UTC
Update
We're still working with our infrastructure provider on the vSphere glitch, which involves VMs being unable to be assigned to a distributed switch group.
Posted Oct 18, 2015 - 02:20 UTC
Investigating
We've run into some deeper issues with respect to the vSphere glitch that we encountered earlier, and are working with our infrastructure provider to resolve them. OS X builds remain paused for the time being.
Posted Oct 18, 2015 - 01:37 UTC
Update
We have stopped the worker processes and are waiting for our vCenter instance to reboot.
Posted Oct 18, 2015 - 00:16 UTC
Update
We stopped the OS X infrastructure worker processes, and narrowed down our leak issues to a glitch in vSphere, and are working with our infrastructure provider to resolve that glitch. In the meantime, we've restarted the worker processes, with a reduced pool size.
Posted Oct 17, 2015 - 23:49 UTC
Identified
We have identified another OS X build infrastructure VM leak incident and are working to resolve it.