We have recovered capacity to the previous levels, and have seen healthy operation for the past 12 hours. Thank you for your patience.
Posted Oct 31, 2017 - 19:49 UTC
Monitoring
At this stage, we have restored most of our capacity and we are monitoring closely for any further signs of capacity issues occurring, and are reaching out to our partners to help isolate and prevent further recurrences.
Posted Oct 31, 2017 - 05:13 UTC
Update
We are currently restoring services piece by piece, and are seeing positive results at this time. However, we are still monitoring closely.
Posted Oct 30, 2017 - 23:57 UTC
Update
Resources are being cleared up, and a tentative estimation is that they should be cleared up in approximately one hour. We will update as soon as we have any further updates on this issue.
Posted Oct 30, 2017 - 22:35 UTC
Update
We are currently working on clearing up available resources so that we can bring the services back online with full capacity, and are still investigating the factors that led to this issue.
Posted Oct 30, 2017 - 22:08 UTC
Update
We are currently restarting all of our GCE instances to restore them to full service. We are still investigating the contributing factors leading to this issue, and the next update will follow in 30 minutes.
Posted Oct 30, 2017 - 21:39 UTC
Update
Due to ongoing issues, we are performing emergency maintenance on our GCE (sudo: required) infrastructure. During this time, service will be significantly slower or not function. We will provide an update within 30 minutes.
Posted Oct 30, 2017 - 21:07 UTC
Investigating
We are seeing queues once more, and are continuing to investigate the issue. At this stage, we are seeing missing capacity and are working on restoring it.
Posted Oct 30, 2017 - 20:41 UTC
Monitoring
We are seeing that the queues and backlogs are clearing up, but we are still investigating the issue for further details and to confirm it has been fully resolved.
Posted Oct 30, 2017 - 19:49 UTC
Investigating
We are currently investigating a capacity issue on our open source infrastructure, related to a configuration change. This affects sudo:required (GCE) builds. We are deploying a new configuration that we hope will resolve this issue.