Usage and backlog spike detected for `sudo: required` builds

Incident Report for Travis CI

Resolved

There is no longer any backlog for GCE users. We are still investigating requeuing internally and determining the root cause of this issue, but end users should no longer be affected.
Posted Oct 03, 2017 - 20:56 UTC

Monitoring

The backlogs have been cleared, but we continue to keep an eye on the situation to ensure all continues as intended.
Posted Oct 03, 2017 - 19:27 UTC

Identified

We continue to work to lower the queues and we are seeing a decrease in backlogs. Our .org backlog appears cleared but we are still seeing some on .com. We will update once the issue is resolved or any new information is discovered.
Posted Oct 03, 2017 - 18:25 UTC

Update

We’ve identified that internal requeues are the source of delays for sudo: required projects. We are continuing to investigate and resolve this issue and will continue posting updates here.
Posted Oct 03, 2017 - 16:41 UTC

Investigating

We are experiencing an issue caused by a sudden spike of usage on GCP. We are investigating the details and will provide updates as soon as we have them.
The previous issue regarding status updates delays has been cleared.
Posted Oct 03, 2017 - 16:21 UTC