We're back in healthy operating order. Thanks for your patience!
Posted Apr 05, 2018 - 00:40 UTC
Monitoring
We are now operating under capacity, although we are continuing to see errors due to quota limits which result in prolonged boot times.
Posted Apr 04, 2018 - 21:19 UTC
Update
Our configuration change didn't lead to the improvement we expected. We are now deploying a new change and we'll be again restarting our workers to make it effective. Thank you for your enduring patience.
Posted Apr 04, 2018 - 20:27 UTC
Update
We have made a configuration change and are now restarting our workers to propagate it in production. Thanks for hanging in there with us.
Posted Apr 04, 2018 - 19:45 UTC
Identified
We are seeing capacity issues in one GCE region where our VMs are running which is responsible for the longer boot times we are seeing. We are removing this specific region to prevent allocating new VMs in it to improve the boot times.
Posted Apr 04, 2018 - 19:25 UTC
Investigating
We are currently seeing delays for private builds (travis-ci.com) running on the sudo-enabled infrastructure (i.e. with `sudo: required`). We are looking into it.