Shortly after 20:00 UTC on December 1, we lost the vast majority of build processing capacity with our Linux workers.

This resulted in a long back log for the builds on our Linux build environments. Unfortunately, due to the nature of the existing alerts, we did not notice the slow processing until 4 hours later.

We are sorry for the service disruption.

We identified the source of the stoppage, and deployed the hot fix at 01:03 UTC, December 2.

The full build capacity was restored shortly after.

The worker went through the queue backlog, and the build processing returned to normal around 06:40 UTC.

Please note that this degraded performance is not related to the DNS outage we experienced at the same time.

Analysis

We identified a rare condition under which only some of the worker processes (responsible for processing build requests) terminated when errors were encountered with connections to the underlying VM.

Actions

• We have identified and fixed the bug that caused worker process to terminate.
• We are working to set up a new alert to notify us of reduced processing capacity.