We made it through the backlog at 11:40pm CET, and everything has been running smoothly since.
Posted Nov 18, 2014 - 23:25 UTC
Update
We've brought all capacity back up, and we're working through the backlog at this point. We'll follow up with a full post-mortem.
Posted Nov 18, 2014 - 21:59 UTC
Update
It seems one of the Linux worker machines failed to reconnect to the queue, causing us to run on reduced capacity.
Posted Nov 18, 2014 - 21:15 UTC
Update
Mac builds are all caught up and running smoothly. Our Linux queue is still backed up, though, unfortunately.
Posted Nov 18, 2014 - 20:57 UTC
Monitoring
Things are running again. We do have a large backlog to get through, though, so expect some delays.
Posted Nov 18, 2014 - 20:43 UTC
Update
Our RabbitMQ cluster has been restarted and log processing is back up, taking the rest of the system online one application at a time.
Posted Nov 18, 2014 - 20:28 UTC
Update
We're preparing for a restart of our RabbitMQ cluster, which means that we've had to disable build scheduling temporarily.
Posted Nov 18, 2014 - 19:17 UTC
Update
Log processing is still lagging behind. Processing is currently less than a 100th of our normal performance. We're trying to reach our RabbitMQ host. We will try restarting our RabbitMQ instance (have you tried turning it off and on again?) - this however involves taking most of Travis CI offline for a moment and will further impact not just logs but builds.
Posted Nov 18, 2014 - 19:01 UTC
Update
We're investigating slow RabbitMQ connectivity which seems to contribute to the current slowness we're seeing in processing build logs.
Posted Nov 18, 2014 - 18:58 UTC
Update
Builds are running normally again, but we do have a backlog to work through, causing some further delays. Log processing is still having issues.
Posted Nov 18, 2014 - 18:30 UTC
Update
We are running into connection issues with our RabbitMQ setup, our infrastructure provider has been contacted, we are investigating further.
Posted Nov 18, 2014 - 18:05 UTC
Investigating
We're looking into a slowdown in our log processing infrastructure for private repositories