Normal operations have been restored for log output. Thanks for your patience!
Posted Jan 12, 2016 - 23:11 UTC
Update
We have pushed an additional change to our log aggregation queries, and are now tweaking the limit and interval in order to get through the backlog quickly.
Posted Jan 12, 2016 - 21:23 UTC
Monitoring
The new index is ready and in use. We are now monitoring the log aggregation process.
Posted Jan 12, 2016 - 20:25 UTC
Update
The migration containing the new index applied successfully in staging, so we are now applying it to production.
Posted Jan 12, 2016 - 19:25 UTC
Identified
We have a backlog in log aggregation that is resulting in build logs for public repositories to appear blank. We have identified a query that no longer executes within the necessary time window, which has caused a snowball effect via an ever growing data set. We are planning to add a new index concurrently to address the issue, which we expect will begin to correct the problem within the next hour.