Update - We have worked through most of the backlog over the last few hours. We continue to investigate the root cause for instability and we’ll post an update as soon as we can.
Sep 21, 09:00 UTC

Update - We've completed the first round of job cancellations and are seeing some improvements. We're currently evaluating other changes to help reduce the backlog and improve the wait time for jobs starting. We'll provide more updates as we learn more. Thank you for your patience.
Sep 20, 15:44 UTC

Update - In an effort to clear up load on macOS servers, we are cancelling builds that have been waiting to start for more than 6 hours. This process will start at 16:15 CEST and is expected to take approximately 2 hours to complete.

We will also be rolling back to a previous version of the worker to determine if recent changes have contributed to these issues.
Sep 20, 13:47 UTC

Update - We continue to battle with stability issues on the MacOS platform, which is the root cause of severe wait times for public repository builds. We will keep you posted about any updates. We apologize for the delays this is causing.
Sep 20, 10:49 UTC

Update - We have not made any significant gains in understanding the source of instability, but changes to available capacity distribution are helping to reduce the impact of disconnections.
Sep 20, 01:33 UTC

Update - We are continuing to investigate the heightened AMQP timeout errors. The backlog for public OSX / mac OS jobs remains high.
Sep 19, 17:41 UTC

Update - We've identified that a new backend change was having a unexpected negative impact on communications with a message queue and was leading to an increased backlog. We've test out a configuration change to disable this new change and it's having the positive impact we expected, so we're continue to roll it out to all parts of our infrastructure. We'll provide updates as this rollout progress.
Sep 18, 20:03 UTC

Update - Maintenance of travis-ci.com is complete and we continuing processing of jobs.

We are now beginning maintenance on travis-ci.org.
Sep 17, 10:02 UTC

Update - The backlog for container-based (i.e. sudo: false) Linux builds has cleared. A backlog remains for Mac builds and we will update you when it's cleared. Thank you!
Sep 14, 18:54 UTC

Update - We are happy to report that the backlog has cleared for sudo-enabled Linux builds.

Small backlogs remain for container-based Linux and Mac builds.
Sep 14, 18:39 UTC

Update - The backlog has cleared for private builds. We are continuing to monitor the situation for public/open source builds. Thanks for hanging in there with us.
Sep 11, 19:52 UTC

Update - We're resuming full private macOS build capacity.
Sep 11, 16:50 UTC

Update - We're seeing some instability with some of the private macOS build capacity and so we're reducing capacity temporarily.
Sep 11, 15:47 UTC

Update - We've completed the first phase of our SAN cleanup. Things are stable and so we're working to resume full public macOS build capacity. We'll provide another update when that's complete.
Sep 11, 03:45 UTC

Update - We're now running with the previous capacity for public builds, which is still reduced from our "normal" capacity. We are continuing with SAN cleanup. We'll provide updates as things progress today. Thank you for your patience.
Sep 10, 19:56 UTC

Update - We temporarily have additional reduced capacity for public builds, as we take some actions to continue with our SAN cleanup. We'll provide another update when that capacity has been restored.
Sep 10, 19:45 UTC

Update - We've processed a backlog of approximately 9,600 macOS jobs for public repositories since re-enabling public macOS builds at 07:00 PDT yesterday. As we're still at reduced capacity and working on cleaning the SAN, we still have a backlog of ~150-200 jobs and continue to actively process them. We'll provide updates as things progress today. Thank you for your patience.
Sep 10, 16:45 UTC

Update - We're continuing to process the public backlog while running SAN cleanup. We may still need to reduce or suspend public builds later in the weekend, depending on SAN progress. Thank you for your patience.
Sep 9, 15:11 UTC

Update - Capacity for macOS public repositories has been back online for ~1 hr. We're bumping additional capacity to work through the backlog.
Sep 9, 14:32 UTC

Update - The backlog for private repository builds has been clear for ~4h. We are planning to bring partial capacity for public repositories back online shortly.
Sep 9, 13:04 UTC

Update - We've resumed running private builds at this time. We'll provide further updates on the overall progress tomorrow morning PDT. Thank you for your patience.
Sep 9, 03:13 UTC

Update - We ran into an issue with booting Xcode 8.x images, so all builds are suspended again. We'll update when private builds are running.
Sep 9, 01:28 UTC

Update - In order to help things become stable and reliable going forward, we're undertaking intense cleanup of our SAN filesystem. This cleanup is likely to take all weekend. Because of this, we're only able to resume a portion of our capacity for private builds and will not be resuming shared public builds yet. We do not currently have an ETA for when we'll be able to resume shared public builds. We will provide our next update in the morning PDT. We are very sorry for the delays and will update this incident when we know more. Thank you for your patience.
Sep 9, 01:04 UTC

Update - We're working on stabilization cleanup for our SAN storage. At the moment we do not have an ETA for when we will resume builds. We are very sorry for the delays and will update this incident when we know more. Thank you for your patience.
Sep 8, 21:29 UTC

Update - We're continuing to work on getting things into a stable state where we can potentially start running builds. At the moment we do not have an ETA for when we will resume builds. We are very sorry for the delays and will update this incident when we know more. Thank you for your patience.
Sep 8, 19:52 UTC

Update - We've rebooted our vCenters and continue to work on stabilizing things. All macOS builds remain stopped thank you for patience.
Sep 8, 16:52 UTC

Update - macOS jobs for public and private repositories builds are stopped.
We're currently working with our infrastructure provider to reboot one of our vCenter instances to work out unresponsive SAN issues.
Sep 8, 15:01 UTC

Update - We are stopping all macOS jobs because we have run out of space on our datacenter's SAN.
Sep 8, 12:34 UTC