At this time we've worked through the backlog due to the additional capacity we added.
Our preliminary investigation into the network timeouts did not lead us to any direct conclusions at this time, but in investigating we did realize that our NAT instance sizing for this environment does need to be increased to match our public Container build environment, and so we suspect that the spike in builds which caused build delays also resulted in our NAT instance being overloaded.
As an initial step to help prevent our NAT instance for this environment from being overloaded like this again, we will be scheduling planned maintenance soon to upgrade the NAT instance, and we're working on identifying additional metrics and monitoring we can add to get better insight into the state and capacity of our NAT instances.
Thank you for your patience during today's incident.
Posted Jul 20, 2015 - 20:34 UTC
Update
Our backlog for private repositories builds on the Linux container infrastructure has been caught up at this time.
We are continuing to investigate the network timeouts that we saw with some builds.
Posted Jul 20, 2015 - 19:15 UTC
Identified
We're adding additional capacity to the Linux container infrastructure for private repositories to help with the backlog.
We are still investigating why network connections are timing out and failing for connections to things like Github and RubyGems
Posted Jul 20, 2015 - 19:02 UTC
Investigating
We are actively investigating network timeouts and a spike in delayed builds on our Linux container infrastructure.