Builds have stabilized on all infrastructures, and we will continue to monitor the situation. If you’re still experiencing any problems, please get in touch via email: email@example.com
Posted Jan 17, 2018 - 11:18 UTC
We have identified the cause of the problem to be our RabbitMQ cluster which needed to be upgraded and the queues purged.
We are now back to processing jobs on all infrastructures and closely monitoring the situation. As a clean-up effort, we have to cancel all jobs that were previously stuck.
Thank you for your patience and understanding.
Posted Jan 17, 2018 - 09:55 UTC
We continue working on the build back log issue. We are canceling some builds to help our recovery efforts. Thank you for your continued patience and understanding.
Posted Jan 17, 2018 - 05:17 UTC
We are resuming open source builds on the macOS and Linux container-based (i.e. sudo: false) infrastructures.
Posted Jan 17, 2018 - 04:27 UTC
We are still working on getting our infrastructure back up. Open source builds on both Linux and macOS are still stopped for the time being.
Posted Jan 17, 2018 - 04:09 UTC
All open source builds are currently stopped for both our Linux and macOS infrastructures. We are currently working on getting them back on their feet. To help us do this, we are canceling currently queued jobs. We are terribly sorry for the disruption and we will continue to provide updates in a timely manner. Thank you for your enduring patience.
Posted Jan 17, 2018 - 02:25 UTC
We are still investigating the job backlog incident.
Posted Jan 17, 2018 - 00:26 UTC
We are continuing to work on resolving this issue, and are investigating other potential causes for the delay.
Posted Jan 16, 2018 - 21:17 UTC
We have noticed an increased back log that extends to both 'sudo: required' and 'sudo: false' (container based) infrastructure. We have identified the causes that contributed to the back log, and are working on resolving them.
Posted Jan 16, 2018 - 19:18 UTC
We continue to investigate the increased back log, as well as a growing back log on our 'sudo: required' infrastructure.
Posted Jan 16, 2018 - 18:46 UTC
We are currently investigating an increased back log on our open container-based infrastructure. During this time, you may experience some delays in builds starting. We will keep you posted as soon as we have an update.