Sep 20, 19:12 UTC
The public macOS build backlog has reached normal peak levels and things are remaining stable. We're closing the incident at this time. A postmortem blog post will be published in the next few days and we'll share it on Twitter when it's published. Thank you everyone for your patience and understanding during this extended incident.
Sep 11, 21:11 UTC
The backlog has cleared for private builds. We are continuing to monitor the situation for public/open source builds. Thanks for hanging in there with us.
Sep 11, 19:52 UTC
We're resuming full private macOS build capacity.
Sep 11, 16:50 UTC
We're seeing some instability with some of the private macOS build capacity and so we're reducing capacity temporarily.
Sep 11, 15:47 UTC
We've resumed full build capacity for public builds. We will be monitoring things overnight and will provide further updates in the morning PDT. Thank you for your patience.
Sep 11, 03:57 UTC
We've completed the first phase of our SAN cleanup. Things are stable and so we're working to resume full public macOS build capacity. We'll provide another update when that's complete.
Sep 11, 03:45 UTC
We're now running with the previous capacity for public builds, which is still reduced from our "normal" capacity. We are continuing with SAN cleanup. We'll provide updates as things progress today. Thank you for your patience.
Sep 10, 19:56 UTC
We temporarily have additional reduced capacity for public builds, as we take some actions to continue with our SAN cleanup. We'll provide another update when that capacity has been restored.
Sep 10, 19:45 UTC
We've processed a backlog of approximately 9,600 macOS jobs for public repositories since re-enabling public macOS builds at 07:00 PDT yesterday. As we're still at reduced capacity and working on cleaning the SAN, we still have a backlog of ~150-200 jobs and continue to actively process them. We'll provide updates as things progress today. Thank you for your patience.
Sep 10, 16:45 UTC
We're continuing to process the public backlog while running SAN cleanup. We may still need to reduce or suspend public builds later in the weekend, depending on SAN progress. Thank you for your patience.
Sep 9, 15:11 UTC
Capacity for macOS public repositories has been back online for ~1 hr. We're bumping additional capacity to work through the backlog.
Sep 9, 14:32 UTC
The backlog for private repository builds has been clear for ~4h. We are planning to bring partial capacity for public repositories back online shortly.
Sep 9, 13:04 UTC
We've resumed running private builds at this time. We'll provide further updates on the overall progress tomorrow morning PDT. Thank you for your patience.
Sep 9, 03:13 UTC
We ran into an issue with booting Xcode 8.x images, so all builds are suspended again. We'll update when private builds are running.
Sep 9, 01:28 UTC
In order to help things become stable and reliable going forward, we're undertaking intense cleanup of our SAN filesystem. This cleanup is likely to take all weekend. Because of this, we're only able to resume a portion of our capacity for private builds and will not be resuming shared public builds yet. We do not currently have an ETA for when we'll be able to resume shared public builds. We will provide our next update in the morning PDT. We are very sorry for the delays and will update this incident when we know more. Thank you for your patience.
Sep 9, 01:04 UTC
We're working on stabilization cleanup for our SAN storage. At the moment we do not have an ETA for when we will resume builds. We are very sorry for the delays and will update this incident when we know more. Thank you for your patience.
Sep 8, 21:29 UTC
We're continuing to work on getting things into a stable state where we can potentially start running builds. At the moment we do not have an ETA for when we will resume builds. We are very sorry for the delays and will update this incident when we know more. Thank you for your patience.
Sep 8, 19:52 UTC
We've rebooted our vCenters and continue to work on stabilizing things. All macOS builds remain stopped thank you for patience.
Sep 8, 16:52 UTC
macOS jobs for public and private repositories builds are stopped.
We're currently working with our infrastructure provider to reboot one of our vCenter instances to work out unresponsive SAN issues.
Sep 8, 15:01 UTC
We are stopping all macOS jobs because we have run out of space on our datacenter's SAN.
Sep 8, 12:34 UTC
We've identified an issue with some of our Xcode image hosts, causing macOS requeues on both public and private repositories. We're working together with our upstream provider to sort this out while we continue investigating macOS build timeouts.
Sep 8, 11:07 UTC
We continue investigating mac OS requeues and build timeouts for both public and private repositories. This seems to be related to SAN performance, we'll continue posting updates as we work to get a more stable performance.
Sep 8, 10:32 UTC
The stability of our macOS builds seems to have improved. We will continue to monitor the rate of errored builds. Thank you for you patience.
Sep 8, 08:56 UTC
Repositories running on travis-ci.org and travis-ci.com are experiencing an increase in errored builds. We are investigating and will update as soon as we can.
Sep 7, 09:36 UTC