Builds delayed for public repos

Incident Report for Travis CI

Resolved

We are back to normal operation.
Posted Feb 26, 2015 - 04:12 UTC

Monitoring

We are fully functional, including web and API. Please note that builds were being processed while we were under maintenance. If you see problems, please email us at support@travis-ci.com.
Posted Feb 26, 2015 - 02:37 UTC

Update

We are nearing completion of database maintenance and preparing to bring API and Web back online.
Posted Feb 26, 2015 - 01:38 UTC

Update

travis-ci.org will be going into maintenance mode while we investigate the database performance issues we are experiencing
Posted Feb 25, 2015 - 22:53 UTC

Update

We are working with our database infrastructure provider to further investigate remaining issues.
Posted Feb 25, 2015 - 20:54 UTC

Update

We are working on resolving normal operation of one remaining component.
Posted Feb 25, 2015 - 19:26 UTC

Identified

We believe a concurrent index build that we introduced ~20 minutes prior to the first noticeable slowdown is to blame. We have cancelled the index build query and are keeping a close watch on database performance.
Posted Feb 25, 2015 - 17:30 UTC

Update

We are continuing to investigate. Thanks for your patience!
Posted Feb 25, 2015 - 16:58 UTC

Update

The database deadlock is now resolved and we are continuing to investigate the increased latency of GitHub webhook handling.
Posted Feb 25, 2015 - 16:34 UTC

Update

We have identified a deadlock in the database and are working to resolve it. It is unclear at this point how this is related to the latency we noticed earlier.
Posted Feb 25, 2015 - 16:30 UTC

Investigating

We are seeing abnormally high delays between GitHub webhook deliveries and builds appearing on Travis CI.
Posted Feb 25, 2015 - 16:18 UTC