We've worked through the accumulated backlog for public macOS builds which are now performing normally. We are continuing to monitor this closely. Thanks for your patience!
Posted Sep 22, 2017 - 10:03 UTC
Update
We have worked through most of the backlog over the last few hours. We continue to investigate the root cause for instability and we’ll post an update as soon as we can.
Posted Sep 21, 2017 - 09:00 UTC
Update
We've completed the first round of job cancellations and are seeing some improvements. We're currently evaluating other changes to help reduce the backlog and improve the wait time for jobs starting. We'll provide more updates as we learn more. Thank you for your patience.
Posted Sep 20, 2017 - 15:44 UTC
Update
In an effort to clear up load on macOS servers, we are cancelling builds that have been waiting to start for more than 6 hours. This process will start at 16:15 CEST and is expected to take approximately 2 hours to complete.
We will also be rolling back to a previous version of the worker to determine if recent changes have contributed to these issues.
Posted Sep 20, 2017 - 13:47 UTC
Update
We continue to battle with stability issues on the MacOS platform, which is the root cause of severe wait times for public repository builds. We will keep you posted about any updates. We apologize for the delays this is causing.
Posted Sep 20, 2017 - 10:49 UTC
Update
We have not made any significant gains in understanding the source of instability, but changes to available capacity distribution are helping to reduce the impact of disconnections.
Posted Sep 20, 2017 - 01:33 UTC
Update
We are continuing to investigate the heightened AMQP timeout errors. The backlog for public OSX / mac OS jobs remains high.
Posted Sep 19, 2017 - 17:41 UTC
Investigating
We’re investigating a higher-than-normal AMQP timeout errors affecting the throughput of our builds. This is affecting public OSX / mac OS jobs most at the moment, as this is the highest demand queue.