Slow booting Linux builds
Incident Report for Travis CI
Resolved
We are considering this incident to be resolved at this time.
We'll be conducting an incident review and publishing the findings on our blog soon.
Thank you for your patience.
Posted Mar 29, 2019 - 01:22 UTC
Update
We are happy to report that the backlog on travis-ci.org has now cleared as well. Your builds should now be starting more promptly depending on your account's capacity vs its accumulated backlog. We'll still be monitoring things closely to ensure the infrastructure continues to be stable.
Posted Mar 29, 2019 - 00:11 UTC
Update
The travis-ci.org backlog has been steadily decreasing since the last update. We estimate it should be cleared in the next 30 minutes.
Posted Mar 28, 2019 - 23:25 UTC
Update
We are continuing to monitor for any further issues.
Posted Mar 28, 2019 - 20:49 UTC
Update
We are making steady progress on processing the travis-ci.org backlog. It has decreased by close to 40% since the last update.
Posted Mar 28, 2019 - 20:45 UTC
Update
We are happy to report that the backlog on travis-ci.com has cleared. We have proceeded with increasing the capacity on travis-ci.org to help with the remaining backlog.
Posted Mar 28, 2019 - 19:59 UTC
Monitoring
travis-ci.com's backlog has been reduced by at least 80% since the last update so we are hopeful to clear it really soon. travis-ci.org's backlog was reduced by close to 20%.
Posted Mar 28, 2019 - 19:09 UTC
Update
We are making good progress on clearing the backlog on travis-ci.com. It's been reduced by at least 50% since the last update. travis-ci.org's backlog is also decreasing but at a lower pace to ensure we don't run into API rate limit again. Thank you again for your enduring patience.
Posted Mar 28, 2019 - 18:12 UTC
Update
We are bringing back capacity on both sites.
Posted Mar 28, 2019 - 17:08 UTC
Update
We made great progress on cleaning up stale VMs in our infrastructure and we are preparing to add back capacity.
Posted Mar 28, 2019 - 16:21 UTC
Update
We are stopping workers on travis-ci.com and travis-ci.org to hopefully alleviate the current situation.
Posted Mar 28, 2019 - 15:18 UTC
Update
The backlog on both travis-ci.org and travis-ci.com is decreasing steadily. We will continue to keep you posted on their state.
Posted Mar 28, 2019 - 13:11 UTC
Investigating
We are seeing an increase in the build backlog and we are looking into it. Thank you for your patience.
Posted Mar 28, 2019 - 09:16 UTC
Update
Backlog on .com is decreasing steadily. We are still monitoring it closely.
Posted Mar 28, 2019 - 04:05 UTC
Update
We have increased the capacity on .com. We'll be looking into increasing it further if things stay stable.
Posted Mar 28, 2019 - 03:37 UTC
Update
There’s still a backlog on .com and we’ll be looking at increasing the capacity over there soon.
Posted Mar 28, 2019 - 02:37 UTC
Update
Backlog has cleared on .org. We are still seeing a backlog on .com that's been decreasing at a more slower pace.
Posted Mar 28, 2019 - 01:25 UTC
Update
Queues are being processed slowly. We expect this process to speed up as we free up more resources.
Posted Mar 27, 2019 - 22:05 UTC
Update
We are still seeing a downward trend on the .com backlog. It’s about 25% lower since the last update. We’ve proceeded to restart the workers on .org to give more headroom and the backlog is a little bit higher over there. Thanks for hanging in there.
Posted Mar 27, 2019 - 20:40 UTC
Update
We've stopped some workers to mitigate an API rate limit we were running into. This had a positive effect and worker's capacity is now back up. The queue is still large but we are seeing a downward trend. We'll keep you posted on the state of the backlog
Posted Mar 27, 2019 - 19:40 UTC
Identified
We believe we have identified sources of the issue and we are working on cleaning orphaned VMs that are preventing new ones from being created.
Posted Mar 27, 2019 - 18:09 UTC
Update
We are in contact with our cloud provider to help us troubleshoot this situation further.
Posted Mar 27, 2019 - 17:08 UTC
Update
We are still investigating. We will post an update as soon as we have more information. Thank you for your patience.
Posted Mar 27, 2019 - 15:57 UTC
Investigating
We are currently receiving reports of Linux builds not booting promptly. We are currently looking into it.
Posted Mar 27, 2019 - 14:50 UTC
This incident affected: Builds Processing (Linux and Windows Builds).