Resolved -
Log processing has recovered. We are investigating the possibility of some stuck log parts. If you do experience paused or stuck logs, please restart those builds. Thank you for your patience! 💛
Jun 22, 13:53 UTC
Update -
The backlog for private Mac builds has cleared. We continue to process our backlog of log parts.
Jun 22, 13:24 UTC
Update -
Job backlogs for private linux builds have cleared. We continue to process the Mac builds, as well as the log parts.
Jun 22, 12:36 UTC
Monitoring -
Job processing is recovering. We are bringing up extra capacity to process the job backlogs more quickly. We are also processing our backlog of log parts. We're continue to closely monitor the situation.
Jun 22, 12:16 UTC
Identified -
We have identified a correlating issue with one of our RabbitMQ instances. The faulty instance has been restarted, and we are waiting for capacity to fully recover. This has created job delays and backlogs for private builds on all infrastructures.
Jun 22, 11:51 UTC
Update -
We’ve manually bumped our Linux capacity to help private Linux builds process faster while we continue working on logs processing.
Jun 22, 11:27 UTC
Monitoring -
We’ve identified and fixed a memory issue caused by a high spike in our sync queues. Some build request have been lost during this process, please re-push a commit again to ensure your build is triggered.
We’re monitoring and builds should be running as expected from now on. We’re also working on processing logs and will give an update as soon as they are at 100%.
Jun 22, 11:03 UTC
Investigating -
We’re investigating API and sync connection issues, causing build delays for private repositories, log in issues and logs display and retrieval issues.
Jun 22, 10:43 UTC