We processed the build backlog at 20:15 UTC and have been processing all new builds as they come in. We've also modified our alerting to better detect for a partial capacity failure. Thank you for your patience while we resolved this incident.
Posted Oct 15, 2015 - 22:25 UTC
Monitoring
We've tracked it down to a worker process going inactive a few hours ago. After restarting the worker process, we're seeing normal job processing. We're monitoring the situation closely, and will be investigating further into why the process died.
Posted Oct 15, 2015 - 19:26 UTC
Investigating
We are currently investigating why we are not running at full capacity on our OS X infrastructure. Thank you for your patience.