We've worked through the accumulated backlog for public macOS builds which are now performing normally. We are continuing to monitor this closely. Thanks for your patience!
Sep 22, 10:03 UTC
We have worked through most of the backlog over the last few hours. We continue to investigate the root cause for instability and we’ll post an update as soon as we can.
Sep 21, 09:00 UTC
We've completed the first round of job cancellations and are seeing some improvements. We're currently evaluating other changes to help reduce the backlog and improve the wait time for jobs starting. We'll provide more updates as we learn more. Thank you for your patience.
Sep 20, 15:44 UTC
In an effort to clear up load on macOS servers, we are cancelling builds that have been waiting to start for more than 6 hours. This process will start at 16:15 CEST and is expected to take approximately 2 hours to complete.
We will also be rolling back to a previous version of the worker to determine if recent changes have contributed to these issues.
Sep 20, 13:47 UTC
We continue to battle with stability issues on the MacOS platform, which is the root cause of severe wait times for public repository builds. We will keep you posted about any updates. We apologize for the delays this is causing.
Sep 20, 10:49 UTC
We have not made any significant gains in understanding the source of instability, but changes to available capacity distribution are helping to reduce the impact of disconnections.
Sep 20, 01:33 UTC
We are continuing to investigate the heightened AMQP timeout errors. The backlog for public OSX / mac OS jobs remains high.
Sep 19, 17:41 UTC
We’re investigating a higher-than-normal AMQP timeout errors affecting the throughput of our builds. This is affecting public OSX / mac OS jobs most at the moment, as this is the highest demand queue.
Sep 19, 10:50 UTC