Emergency database upgrades for the build logs service, for both travis-ci.com and travis-ci.org

Incident Report for Travis CI

Resolved

This incident has been resolved.
Posted Jan 21, 2016 - 02:12 UTC

Update

At this time both travis-ci.org and travis-ci.com logs are being processed as they come in and there should be no delays in build processing for any platforms. Thank you for your patience during this lengthy incident. We'll be preparing a public post-mortem that we should publish soon.
Posted Jan 21, 2016 - 02:11 UTC

Monitoring

We're nearly caught up, we're monitoring things closely and will resolve this issue when we're caught up.
Posted Jan 21, 2016 - 01:58 UTC

Update

We continue to steadily catching up on the backlog of travis-ci.org build logs and will provide an update when we've caught up.
Posted Jan 21, 2016 - 01:48 UTC

Update

We are steadily catching up on the backlog of travis-ci.org build logs and will provide an update when we've caught up.
Posted Jan 21, 2016 - 00:50 UTC

Update

We've completed the initial failover for travis-ci.org and are working to resume the logging services
Posted Jan 21, 2016 - 00:23 UTC

Update

We are beginning the process of failing over the logs database for travis-ci.org. This will result in a temporary pause to new logs showing up and some delay for build jobs being run.
Posted Jan 21, 2016 - 00:06 UTC

Update

At this time travis-ci.com logs are being processed as they come in and there should be no delays in build processing. We're preparing to upgrade travis-ci.org's database shortly.
Posted Jan 20, 2016 - 23:50 UTC

Update

We've completed the initial failover and are working to resume the logging services
Posted Jan 20, 2016 - 23:37 UTC

Update

We are beginning the process of failing over the logs database for travis-ci.com. This will result in a temporary pause to new logs showing up and some delay for build jobs being run.
Posted Jan 20, 2016 - 23:13 UTC

Update

We've determined that upgrading the size of each database for the build log service for both travis-ci.com and travis-ci.org. We're preparing this process and will be starting with travis-ci.com shortly. We'll post another update when we begin.
Posted Jan 20, 2016 - 22:22 UTC

Update

We are investigating why there's been an unexpected large jump in our logging backlog and how to mitigate and resolve this.
Posted Jan 20, 2016 - 21:52 UTC

Identified

All builds are running as expected. Only display of the builds is affected. We've seen some sudden spikes in the amount of logs in our backlog for log processing and investigating why.
Posted Jan 20, 2016 - 20:36 UTC

Investigating

Delays in build logs being shown in the web UI. We are investigating this.
Posted Jan 20, 2016 - 20:26 UTC