Builds processing delays for all repositories on travis-ci.org

Incident Report for Travis CI

Resolved

This incident has been resolved.
Posted Dec 16, 2018 - 13:08 UTC

Update

Linux capacity is back to normal. We are still waiting for Mac capacity to work through the backlog.
Posted Dec 16, 2018 - 12:42 UTC

Monitoring

Mac capacity is back online, albeit with a backlog.
Posted Dec 16, 2018 - 11:18 UTC

Update

Linux capacity is back online, although jobs queued during the RabbitMQ outage still require replay. We have not yet reconfigured Mac capacity.
Posted Dec 16, 2018 - 10:19 UTC

Update

We are in the process of configuring all apps to connect to a freshly-provisioned RabbitMQ cluster.
Posted Dec 16, 2018 - 09:35 UTC

Identified

The RabbitMQ cluster used by various systems in travis-ci.org appears to be unresponsive. We have attempted to fully restart the cluster, but this hasn't had the expected effect, so we have escalated to our upstream service provider.
Posted Dec 16, 2018 - 07:55 UTC

Investigating

We are currently investigating this issue.
Posted Dec 16, 2018 - 07:37 UTC
This incident affected: Builds Processing (Linux and Windows Builds).