On Friday at 15:55 UTC we noticed an abnormally high message count in the RabbitMQ queue for build status updates, causing a lag to updates through the web and build-related notifications.

Upon restarting the component responsible for processing these messages (travis-hub), the queue drained within 10 minutes (around 16:18 UTC). Once we looked into when the queue buildup started, we were able to correlate it with an error in the logs at 13:44 UTC showing that at least one of the processes was unable to connect to its Memcached cluster. Additionally, we found that our error reporting failed due to an HTTP timeout, leading us to suspect general network problems for the affected process. While the disconnection error had recovered shortly afterward, the rate at which messages were being consumed was abnormally low, resulting in a backlog that grew for just over 2 hours.

We haven't found anything to explain this behavior, so are setting up additional alerting around the build status queue size to help us detect problems earlier. Thanks for your patience!