Builds have stabilized on all infrastructures, and we will continue to monitor the situation. If you’re still experiencing any problems, please get in touch via email: firstname.lastname@example.org
Jan 17, 11:18 UTC
We have identified the cause of the problem to be our RabbitMQ cluster which needed to be upgraded and the queues purged.
We are now back to processing jobs on all infrastructures and closely monitoring the situation. As a clean-up effort, we have to cancel all jobs that were previously stuck.
Thank you for your patience and understanding.
Jan 17, 09:55 UTC
We continue working on the build back log issue. We are canceling some builds to help our recovery efforts. Thank you for your continued patience and understanding.
Jan 17, 05:17 UTC
We are resuming open source builds on the macOS and Linux container-based (i.e. sudo: false) infrastructures.
Jan 17, 04:27 UTC
We are still working on getting our infrastructure back up. Open source builds on both Linux and macOS are still stopped for the time being.
Jan 17, 04:09 UTC
All open source builds are currently stopped for both our Linux and macOS infrastructures. We are currently working on getting them back on their feet. To help us do this, we are canceling currently queued jobs. We are terribly sorry for the disruption and we will continue to provide updates in a timely manner. Thank you for your enduring patience.
Jan 17, 02:25 UTC
We are still investigating the job backlog incident.
Jan 17, 00:26 UTC
We are continuing to work on resolving this issue, and are investigating other potential causes for the delay.
Jan 16, 21:17 UTC
We have noticed an increased back log that extends to both 'sudo: required' and 'sudo: false' (container based) infrastructure. We have identified the causes that contributed to the back log, and are working on resolving them.
Jan 16, 19:18 UTC
We continue to investigate the increased back log, as well as a growing back log on our 'sudo: required' infrastructure.
Jan 16, 18:46 UTC
We are currently investigating an increased back log on our open container-based infrastructure. During this time, you may experience some delays in builds starting. We will keep you posted as soon as we have an update.
Jan 16, 18:10 UTC