At this time the backlog has been cleared and we are processing new builds as they come in.
This morning's backlog was a result of some assumptions in our current metrics and monitoring setup around when we run a backlog and what our capacity is. We're reviewing what alerts we have configured and will update them to ensure that we get alerted to the failure conditions we saw this morning, should it occur again.
Thanks for your patience while we resolved this issue.
Posted Sep 28, 2015 - 16:25 UTC
Monitoring
OS X private build delays. From 13:50 UTC onward we were running at partial capacity with the OS X private builds. This was due to one of the services which processes builds for this platform failing to recover gracefully from a service restarting during automatic log rotation. We've recovered the service and are working at our current full capacity to catch up on the backlog that was created. We'll provide another update once the backlog has cleared.