travis-ci.org logs database outage.
Incident Report for Travis CI
Resolved
This incident has been resolved.
Posted Apr 10, 2017 - 16:37 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 10, 2017 - 15:34 UTC
Update
We have completed the data restoration process. We will be re-enabling mutability of the affected records shortly.
Posted Apr 10, 2017 - 15:28 UTC
Update
We are continuing work on restoring historical data, which we expect will take between 1-3 days. We will be updating this incident once the restoration process is complete.
Posted Apr 07, 2017 - 14:52 UTC
Identified
We apologize for the delay in updating this incident.

Due to database infrastructure issues we have been unable to successfully extract all data from our previous primary database. Any jobs that completed prior to the incident cannot be effectively restarted for updated logs.

We continue working towards restoring the relevant records. This is estimated to be a ~5 day process due to the size of the database tables.

We are extremely sorry and just as frustrated with this as you probably are. We will provide another update once we have more information.

Please email support@travis-ci.com if you have any questions in the mean time.
Posted Mar 31, 2017 - 16:12 UTC
Update
We apologize for the delay in updating this incident. Our earlier database maintenance included needing to put a new, empty logs database into production.

The previous database still contains the only copy of parts of some build logs, written in the few minutes prior to when the database was no longer able to accept writes. We've been working towards extracting that data this afternoon; so that we can get it archived into S3 and make it accessible again.

However, due to the effects of the original problem, we need the assistance of our database infrastructure provider in performing some actions on the previous primary database. We've escalated this with them, but do not currently have an ETA on when this work will be completed.

We will provide another update once we have an update from them. Please email support@travis-ci.com if you have any questions in the mean time.
Posted Mar 30, 2017 - 01:42 UTC
Monitoring
If you restart jobs that completed before 18:00 UTC, you will continue to see the previous job's output or inconsistent logs displayed. We are working on a resolution for this but it may be a few hours before this is done.
Posted Mar 29, 2017 - 20:01 UTC
Update
We are resuming build processing. New build logs should appear, but recent build logs may be unavailable until further notice.

We also expect some delay in build processing while we catch up with the back logs.
Posted Mar 29, 2017 - 19:50 UTC
Update
We are continuing our database maintenance.
Posted Mar 29, 2017 - 19:23 UTC
Update
We are performing emergency database maintenance. We hope to resume processing build requests shortly.
Posted Mar 29, 2017 - 18:52 UTC
Update
We are stopping accepting builds while we perform emergency maintenance.
Posted Mar 29, 2017 - 18:22 UTC
Identified
Logs database is having problems with writing logs. We are working with our service provider to resolve this issue.
Posted Mar 29, 2017 - 18:06 UTC
Investigating
We are investigating an .org logs database problem. Logs are unavailable at this time.
Posted Mar 29, 2017 - 18:00 UTC