- We apologize for the delay in updating this incident. Our earlier database maintenance included needing to put a new, empty logs database into production.
The previous database still contains the only copy of parts of some build logs, written in the few minutes prior to when the database was no longer able to accept writes. We've been working towards extracting that data this afternoon; so that we can get it archived into S3 and make it accessible again.
However, due to the effects of the original problem, we need the assistance of our database infrastructure provider in performing some actions on the previous primary database. We've escalated this with them, but do not currently have an ETA on when this work will be completed.
We will provide another update once we have an update from them. Please email firstname.lastname@example.org
if you have any questions in the mean time.
Mar 30, 01:42 UTC
Monitoring - If you restart jobs that completed before 18:00 UTC, you will continue to see the previous job's output or inconsistent logs displayed. We are working on a resolution for this but it may be a few hours before this is done.
Mar 29, 20:01 UTC
Update - We are resuming build processing. New build logs should appear, but recent build logs may be unavailable until further notice.
We also expect some delay in build processing while we catch up with the back logs.
Mar 29, 19:50 UTC
Update - We are continuing our database maintenance.
Mar 29, 19:23 UTC
Update - We are performing emergency database maintenance. We hope to resume processing build requests shortly.
Mar 29, 18:52 UTC
Update - We are stopping accepting builds while we perform emergency maintenance.
Mar 29, 18:22 UTC
Identified - Logs database is having problems with writing logs. We are working with our service provider to resolve this issue.
Mar 29, 18:06 UTC
Investigating - We are investigating an .org logs database problem. Logs are unavailable at this time.
Mar 29, 18:00 UTC