Emergency database upgrades for the build logs service, for both travis-ci.com and travis-ci.org
Incident Report for Travis CI
Resolved
This incident has been resolved.
Posted over 3 years ago. Jan 21, 2016 - 02:12 UTC
Update
At this time both travis-ci.org and travis-ci.com logs are being processed as they come in and there should be no delays in build processing for any platforms. Thank you for your patience during this lengthy incident. We'll be preparing a public post-mortem that we should publish soon.
Posted over 3 years ago. Jan 21, 2016 - 02:11 UTC
Monitoring
We're nearly caught up, we're monitoring things closely and will resolve this issue when we're caught up.
Posted over 3 years ago. Jan 21, 2016 - 01:58 UTC
Update
We continue to steadily catching up on the backlog of travis-ci.org build logs and will provide an update when we've caught up.
Posted over 3 years ago. Jan 21, 2016 - 01:48 UTC
Update
We are steadily catching up on the backlog of travis-ci.org build logs and will provide an update when we've caught up.
Posted over 3 years ago. Jan 21, 2016 - 00:50 UTC
Update
We've completed the initial failover for travis-ci.org and are working to resume the logging services
Posted over 3 years ago. Jan 21, 2016 - 00:23 UTC
Update
We are beginning the process of failing over the logs database for travis-ci.org. This will result in a temporary pause to new logs showing up and some delay for build jobs being run.
Posted over 3 years ago. Jan 21, 2016 - 00:06 UTC
Update
At this time travis-ci.com logs are being processed as they come in and there should be no delays in build processing. We're preparing to upgrade travis-ci.org's database shortly.
Posted over 3 years ago. Jan 20, 2016 - 23:50 UTC
Update
We've completed the initial failover and are working to resume the logging services
Posted over 3 years ago. Jan 20, 2016 - 23:37 UTC
Update
We are beginning the process of failing over the logs database for travis-ci.com. This will result in a temporary pause to new logs showing up and some delay for build jobs being run.
Posted over 3 years ago. Jan 20, 2016 - 23:13 UTC
Update
We've determined that upgrading the size of each database for the build log service for both travis-ci.com and travis-ci.org. We're preparing this process and will be starting with travis-ci.com shortly. We'll post another update when we begin.
Posted over 3 years ago. Jan 20, 2016 - 22:22 UTC
Update
We are investigating why there's been an unexpected large jump in our logging backlog and how to mitigate and resolve this.
Posted over 3 years ago. Jan 20, 2016 - 21:52 UTC
Identified
All builds are running as expected. Only display of the builds is affected. We've seen some sudden spikes in the amount of logs in our backlog for log processing and investigating why.
Posted over 3 years ago. Jan 20, 2016 - 20:36 UTC
Investigating
Delays in build logs being shown in the web UI. We are investigating this.
Posted over 3 years ago. Jan 20, 2016 - 20:26 UTC