Mac Builds Network Outage
Incident Report for Travis CI
Postmortem

Our macOS infrastructure hosting provider has shared an update with us on Thursday, Feb 9th's brief Internet connectivity outage:


Update from MacStadium

> Further investigation found the BGP issues to be rooted with Level3 who was having routing issues on a regional basis at this time. Level3 is the preferred carrier for a significant amount of traffic in and out of MacStadium Atlanta including the preferred carrier to popular network peers like AWS.

> During this incident, we found no network transport problems inside MacStadium Atlanta, or its interconnections to our Tier 1 network providers in the region. Our Internet BGP sessions with Level 3 remained up, however traffic stopped flowing to/from Level3 due to an issue inside their network at the time. Because the BGP session never went down, manual intervention was required to force Internet traffic onto other BGP peers we have in Atlanta.

> We have still not received a full explanation from Level 3 as to what occurred, but we do feel that it is necessary to close out this incident at this time as we have seen no other effects or symptoms related to this event in over 24 hours.


Please email support@travis-ci.com if you have any further questions or concerns.

Posted Feb 10, 2017 - 21:04 UTC

Resolved
All MacOS builds have resumed and processing at full capacity. Seemed to have been an upstream provider hiccup. The backlog should be clearing momentarily for .com users.
Posted Feb 09, 2017 - 19:21 UTC
Investigating
We are currently investigating a network outage on our macOS infrastructure.
Posted Feb 09, 2017 - 19:07 UTC