Builds processing delays for all repositories on travis-ci.org

Incident Report for Travis CI

Resolved

This incident has been resolved.
Posted 7 years ago. Dec 16, 2018 - 13:08 UTC

Update

Linux capacity is back to normal. We are still waiting for Mac capacity to work through the backlog.
Posted 7 years ago. Dec 16, 2018 - 12:42 UTC

Monitoring

Mac capacity is back online, albeit with a backlog.
Posted 7 years ago. Dec 16, 2018 - 11:18 UTC

Update

Linux capacity is back online, although jobs queued during the RabbitMQ outage still require replay. We have not yet reconfigured Mac capacity.
Posted 7 years ago. Dec 16, 2018 - 10:19 UTC

Update

We are in the process of configuring all apps to connect to a freshly-provisioned RabbitMQ cluster.
Posted 7 years ago. Dec 16, 2018 - 09:35 UTC

Identified

The RabbitMQ cluster used by various systems in travis-ci.org appears to be unresponsive. We have attempted to fully restart the cluster, but this hasn't had the expected effect, so we have escalated to our upstream service provider.
Posted 7 years ago. Dec 16, 2018 - 07:55 UTC

Investigating

We are currently investigating this issue.
Posted 7 years ago. Dec 16, 2018 - 07:37 UTC
This incident affected: Builds Processing (Linux and Windows Builds).