Emergency maintenance: Reduction of macOS capacity for both private and open source builds

Incident Report for Travis CI

Resolved

The backlog for macOS public repositories has cleared.
Posted Mar 29, 2018 - 02:28 UTC

Update

Our infrastructure provider has completed all host reboots. We are returning all macOS jobs to 100% capacity.
Posted Mar 29, 2018 - 01:35 UTC

Update

Due to SAN load, the rolling restart of all the macOS hosts is still ongoing, and will continue for another estimated 2-4 hours. In order to help our provider complete the task without any downtime, we’ve agreed to reduce load by reduced capacity by roughly 1/3rd. Backlog persists for macOS public repos.
Posted Mar 28, 2018 - 21:24 UTC

Update

Our infrastructure provider is continuing the rolling restart, delays continue for macOS builds on public repos.
Posted Mar 28, 2018 - 19:05 UTC

Update

Our macOS backlog for .com site for private repos is clear. Our infrastructure provider is performing rolling restarts of all hosts for new SAN configurations to take effect. The backlog for .org public repos still persists.
Posted Mar 28, 2018 - 18:08 UTC

Update

We are seeing improvements to performance, but at this time we sill remain at reduced capacity.
Posted Mar 28, 2018 - 17:30 UTC

Identified

Our macOS infrastructure provider has noticed a misconfiguration on our SAN and is working to change it. In order to help load on the SAN, we are reducing our job throughput by 30% hence some delays are expected. Thank you for your understanding.
Posted Mar 28, 2018 - 16:55 UTC