Build delays on .com container-based infrastructure

Incident Report for Travis CI

Postmortem

Firstly, we're very sorry for today's incident and the build delays, this is not the experience we strive to provide our users.

After reviewing the logs from our auto-scaling configuration and the system logs from the instances that were being terminated, we've determined that the instances we were getting in US-EAST-1a between 18:45 and 19:30 UTC were having issues with OOMing all processes and kernel panicking. This resulted in our auto-scaling activity to begin constantly terminating and then creating new instances, which was causing jobs to be requeued and a large backlog to be accrued. However, we saw instances created after 19:30 UTC able to run builds successfully without the OOM and panic behavior.

It's our conclusion at this time that we essentially were getting a bad batch of instances from EC2 for a short time.

We will review our auto-scaling configuration and monitoring to determine how we can better mitigate and alert around the rapid instance up/down behavior we saw today and prevent it (and the backlog it creates) in the future.

Thanks for your patience while we worked through this issue.

Posted Sep 17, 2015 - 21:37 UTC

Resolved

At this time the incident is resolved and we're running with our standard auto-scaling configuration and no backlog.

Posted Sep 17, 2015 - 21:37 UTC

Identified

At this time the backlog had cleared, we're running private container-based jobs as desired.

We're continuining to investigate the OOM error logs we saw.

Posted Sep 17, 2015 - 21:04 UTC

Update

We are sorry for the current backlog and delays this incident are causing. We've stopped the automatic terminate of EC2 instances in our private container auto-scaling config, which has stabilized the current pool of workers for those builds. Additionally, with our current workers that are online we're processing the backlog and expect to be caught up with 60-90 minutes.
We are continuing to investigate a spike in OOM errors we see in the system logs before instance health checks start to fail on the instances that got terminated. We will update as we learn more. Thanks for you patience while we work towards determining contributing factors in this incident.

Posted Sep 17, 2015 - 20:45 UTC

Update

We are seeing new EC2 instances fail to start, which is delaying our ability to add capacity for the current workload, we're investigating what is causing the instance start failures.

Posted Sep 17, 2015 - 19:49 UTC

Investigating

We're investigating build delays on the closed-source container-based build infrastructure.

Posted Sep 17, 2015 - 19:27 UTC