Log processing has recovered. We are investigating the possibility of some stuck log parts. If you do experience paused or stuck logs, please restart those builds. Thank you for your patience! 💛
Posted Jun 22, 2017 - 13:53 UTC
Update
The backlog for private Mac builds has cleared. We continue to process our backlog of log parts.
Posted Jun 22, 2017 - 13:24 UTC
Update
Job backlogs for private linux builds have cleared. We continue to process the Mac builds, as well as the log parts.
Posted Jun 22, 2017 - 12:36 UTC
Monitoring
Job processing is recovering. We are bringing up extra capacity to process the job backlogs more quickly. We are also processing our backlog of log parts. We're continue to closely monitor the situation.
Posted Jun 22, 2017 - 12:16 UTC
Identified
We have identified a correlating issue with one of our RabbitMQ instances. The faulty instance has been restarted, and we are waiting for capacity to fully recover. This has created job delays and backlogs for private builds on all infrastructures.
Posted Jun 22, 2017 - 11:51 UTC
Update
We’ve manually bumped our Linux capacity to help private Linux builds process faster while we continue working on logs processing.
Posted Jun 22, 2017 - 11:27 UTC
Monitoring
We’ve identified and fixed a memory issue caused by a high spike in our sync queues. Some build request have been lost during this process, please re-push a commit again to ensure your build is triggered.
We’re monitoring and builds should be running as expected from now on. We’re also working on processing logs and will give an update as soon as they are at 100%.
Posted Jun 22, 2017 - 11:03 UTC
Investigating
We’re investigating API and sync connection issues, causing build delays for private repositories, log in issues and logs display and retrieval issues.