We have reduced the maximum number of concurrent jobs for both public and private repositories to ease pressure on our existing SAN while we plan for a SAN upgrade. We have been operating below capacity with no queue delays for nearly 2 hours for public repositories and a little over 1 hour for private repositories.
Posted Jul 21, 2015 - 01:59 UTC
Identified
We believe we've hit a saturation point with our SAN, which is resulting in exponential boot timeouts. We are working with our infrastructure provider to move to a faster SAN, as well as adjusting block sizes to smooth out latency from IOPS.
Posted Jul 21, 2015 - 01:45 UTC
Update
We've discovered that the boot errors are due to boots hitting the timeout set for them, and are working with our infrastructure provider to figure out the cause of these elevated boot times.
Posted Jul 21, 2015 - 00:40 UTC
Investigating
We're investigating elevated boot errors on our Mac infrastructure, causing Mac jobs to be delayed.