At this time we have caught up on the backlog for all platforms except OS X, which has a fixed capacity and so it will continue to have backlog for a while more. Everything is looking stable and we're continuing to monitor things closely. Thank you for your patience and understanding while we worked through this issue.
Posted Dec 09, 2015 - 18:07 UTC
Monitoring
We're starting to see builds move from being queued to running but there will be backlog for a while more. We'll continue to provide updates as we monitor the backlog and ensure things remain stable.
Posted Dec 09, 2015 - 17:28 UTC
Update
We were able to identify that a deploy of our JavaScript frontend, with primarily internal changes to use some new API end points) we did earlier today (approx 14:00 UTC) is having a larger than expected impact on the queries generated by the API, which is resulting in the increased load on Postgres. This increased database load is the source of the build delays. We are in the process of rolling back the frontend to the previous version, which should start reducing the load. We'll update once we see the queue backlogs start to be processed again.
Posted Dec 09, 2015 - 16:59 UTC
Identified
We've identified a large spike in the load on travis-ci.com's primary Postgres database and we are examining the query workload to identify the source of the increased load. Thank you for your patience while we look into this.
Posted Dec 09, 2015 - 16:56 UTC
Investigating
We are investigating why we're seeing delayed build queues for private builds on all platforms. We have all eyes on this and will work to get this solved as soon as possible.