We've caught up on the backlog from this incident and are processing builds as expected now. We apologize for the delay in realizing this issue, this particular issue was not caught by our current monitoring for this piece of our infrastructure. We'll be reviewing system logs and identifying additional monitoring we can add to better detect this in the future. Thank you for your patience while we resolved this issue.
Posted Nov 24, 2016 - 04:57 UTC
Update
We are beginning to resume OS X VMs for both public and private builds.
Posted Nov 24, 2016 - 04:50 UTC
Update
We've restored the cluster to service and we're doing some additional checks on things before we resume builds again. Thank you for your patience.
Posted Nov 24, 2016 - 04:41 UTC
Identified
We've determined that one of our clustered VM firewalls was in an inconsistent state that was causing outbound network traffic to fail in an unexpected way that did not result in other parts of the cluster properly handling the traffic. We're working to restore this cluster to service.
Posted Nov 24, 2016 - 04:35 UTC
Investigating
We are currently investigating why our OS X VMs cannot resolve github.com. This is currently preventing `git clone` from working and makes OS X builds fail.