We have resolved the API rate limit issue with our infrastructure provider, and we are operating at full capacity once again.
Posted Dec 07, 2015 - 22:38 UTC
Update
We have scaled down our capacity in response to hitting API quotas. We are working with our infrastructure provider to resolve this, and will continue to provide updates. Thanks for your patience!
Posted Dec 07, 2015 - 20:00 UTC
Update
We're continuing to slowly increase our throughput of concurrent builds on GCE.
Posted Dec 07, 2015 - 18:35 UTC
Update
We appreciate your patience during the extended pause of our GCE builds, this let us accelerate the cleanup process very aggressively. We're beginning the process of slowly ramping up the processing of our backlog and we'll provide regular updates until the backlog is caught up.
Posted Dec 07, 2015 - 17:44 UTC
Update
We have decided to keep GCE builds paused while the total VM count continues to drop. Once we are below 25% capacity, we will begin allowing jobs through again.
Posted Dec 07, 2015 - 17:02 UTC
Update
In order to help with the cleanup process, we will be pausing all GCE builds for 15 minutes. Thanks for your patience and understanding while we work through this.
Posted Dec 07, 2015 - 16:09 UTC
Identified
We've identified an issue with performance in how we're cleaning up VMs on GCE, which was causing us to hit some of our quotas. We're throttling the total number of builds we run on GCE for the moment, while we work to clean up the "leaked VMs" and adjust our quotas to better handle this particular scenarios. Thanks for your patience while we work through this.
Posted Dec 07, 2015 - 15:13 UTC
Investigating
We are investigating issues with delays in booting Trusty and Precise VMs on GCE.