Travis CI
Update - We apologize for the delay in updating this incident. Our earlier database maintenance included needing to put a new, empty logs database into production.

The previous database still contains the only copy of parts of some build logs, written in the few minutes prior to when the database was no longer able to accept writes. We've been working towards extracting that data this afternoon; so that we can get it archived into S3 and make it accessible again.

However, due to the effects of the original problem, we need the assistance of our database infrastructure provider in performing some actions on the previous primary database. We've escalated this with them, but do not currently have an ETA on when this work will be completed.

We will provide another update once we have an update from them. Please email support@travis-ci.com if you have any questions in the mean time.
Mar 30, 01:42 UTC
Monitoring - If you restart jobs that completed before 18:00 UTC, you will continue to see the previous job's output or inconsistent logs displayed. We are working on a resolution for this but it may be a few hours before this is done.
Mar 29, 20:01 UTC
Update - We are resuming build processing. New build logs should appear, but recent build logs may be unavailable until further notice.

We also expect some delay in build processing while we catch up with the back logs.
Mar 29, 19:50 UTC
Update - We are continuing our database maintenance.
Mar 29, 19:23 UTC
Update - We are performing emergency database maintenance. We hope to resume processing build requests shortly.
Mar 29, 18:52 UTC
Update - We are stopping accepting builds while we perform emergency maintenance.
Mar 29, 18:22 UTC
Identified - Logs database is having problems with writing logs. We are working with our service provider to resolve this issue.
Mar 29, 18:06 UTC
Investigating - We are investigating an .org logs database problem. Logs are unavailable at this time.
Mar 29, 18:00 UTC
API Operational
Web App Operational
Builds Processing Degraded Performance
Mac Builds ? Degraded Performance
Linux Builds (container-based) ? Degraded Performance
Linux Builds (Trusty and Precise on GCE) ? Degraded Performance
Background Processing Operational
Log Processing Degraded Performance
User Sync ? Operational
Notifications ? Operational
Third Party Services Operational
GitHub Operational
DNSimple Name Servers Operational
Heroku Operational
Pusher Pusher REST API Operational
Pusher WebSocket client API Operational
Pusher Presence channels Operational
Pusher Webhooks Operational
AWS ec2-us-east-1 Operational
Help Scout Email Processing Operational
Help Scout Web App Operational
PagerDuty Notification Delivery Operational
Google Cloud Platform Google Compute Engine Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
On Sunday April 2nd, 2017, from 17:00 to 21:00 UTC, we will be performing network maintenance on our Mac infrastructure to improve and test the redundancy in our network stack. The maintenance will be performed in two stages, the first starting at 17:00 UTC and the second starting at 20:00 UTC. We're not expecting any user-visible impact apart from brief packet loss as we perform failover tests.
Posted on Mar 24, 09:30 UTC
System Metrics Month Week Day
API Uptime (api.travis-ci.org)
Fetching
API Uptime (api.travis-ci.com)
Fetching
Active Linux Builds for Open Source projects (container-based)
Fetching
Backlog Linux Builds for Open Source projects (container-based)
Fetching
Active Linux Builds for Open Source projects
Fetching
Backlog Linux Builds for Open Source projects
Fetching
Active OS X Builds for Open Source Projects
Fetching
Backlog OS X Builds for Open Source Projects
Fetching
Past Incidents
Mar 29, 2017
Resolved - Systems operating normally. Thanks for bearing with us!
Mar 29, 14:09 UTC
Monitoring - Maintenance has been completed and services appear to be stable. We will continue to monitor.
Mar 29, 13:50 UTC
Identified - We mitigated the issue and API error rates have recovered. We are continuing with maintenance to make sure the API is stable.
Mar 29, 13:26 UTC
Investigating - We are investigating reports of the higher rate of API errors for .org.
Mar 29, 13:04 UTC
Mar 28, 2017
Resolved - This issue has been resolved and logs are working as expected now. Thanks for your patience!
Mar 28, 11:45 UTC
Monitoring - We've deployed a fix for this issue and logs should be rendering now properly. We're currently monitoring. Please email support@travis-ci.com if you notice any log hiccups.
Mar 28, 11:03 UTC
Identified - We’ve identified an issue that is causing some logs for both public and private repositories to not render properly. We’re working on a fix for this.
Mar 28, 10:18 UTC
Investigating - We are investigating reports that logs are not always displayed in web.
Mar 28, 07:52 UTC
Mar 27, 2017
Completed - The scheduled maintenance has been completed.
Mar 27, 21:35 UTC
Verifying - We've completed the update of the API components on both travis-ci.org and travis-ci.com. We are continuing to monitor closely.
Mar 27, 20:52 UTC
Update - We've finished the update of the API components on travis-ci.org and we are monitoring things closely. We shall begin updating travis-ci.com shortly.
Mar 27, 19:43 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 27, 18:30 UTC
Scheduled - We will be performing a maintenance on our API for more efficient access to our logs database. We will first update the API for travis-ci.org and then travis-ci.com with a least an hour in between for monitoring. We don't expect any visible downtime or user impact during the maintenance. Please contact support [at] travis-ci [dot] com if you see something.
Mar 27, 18:08 UTC
Resolved - Backlog for both open-source-builds and private-builds has now cleared. Thank you for your patience.
Mar 27, 08:26 UTC
Update - The backlog for our open-source-builds has cleared. The private-builds backlog continues to drop.
Mar 27, 08:22 UTC
Monitoring - We have identified the issue, applied a fix, and are monitoring as things recover.
Display of newer logs may be delayed until we process the backlog.
Mar 27, 08:02 UTC
Identified - We have identified the issue, applied a fix, and are monitoring as things recover.
Display of newer logs may be delayed until we process the backlog.
Mar 27, 08:02 UTC
Investigating - We’re investigating issues with logs not being displayed.
Mar 27, 07:40 UTC
Mar 26, 2017

No incidents reported.

Mar 25, 2017

No incidents reported.

Mar 24, 2017
Resolved - Webhook delivery delays have been resolved on GitHub. We have confirmed we're receiving build request events immediately.
Mar 24, 12:36 UTC
Identified - We are experiencing delayed build requests due to webhook delivery delays on GitHub (up to ~3min). We are monitoring the situation. Also see https://status.github.com.
Mar 24, 12:27 UTC
Mar 23, 2017
Resolved - We have confirmed the effectiveness of the short-term fix. We will continue investigating for a longer-term resolution. Thank you for your patience.
Mar 23, 15:32 UTC
Monitoring - We have identified the problematic component. While we continue investigating a long-term fix, we have applied a short-time one to have the logs load correctly.
Mar 23, 14:55 UTC
Investigating - We’re investigating issues displaying public logs while builds are running.
Mar 23, 14:13 UTC
Mar 22, 2017
Resolved - The issue was confirmed fixed by our infrastructure provider, and we're no longer seeing errors while booting instances. Jobs are running normally across all our infrastructures at this time.
Mar 22, 16:59 UTC
Identified - Our infrastructure provider has identified an issue and is working on a fix.
Mar 22, 16:29 UTC
Investigating - We're investigating elevated error rates while booting instances for sudo-enabled Linux builds, which is causing jobs to take longer to start.
Mar 22, 15:46 UTC
Mar 21, 2017

No incidents reported.

Mar 20, 2017
Resolved - Things have been reliable and stable, so we are resolving this issue.
Mar 20, 23:43 UTC
Monitoring - The unresponsive host has been removed from the set of DNS records. The site should be fully reachable again. We continue to monitor the situation.
Mar 20, 16:59 UTC
Identified - We are receiving reports of travis-ci.org being unreachable for some users. One of the 8 edge server IPs appears to be unreachable via TCP. We have escalated the issue to our upstream provider.
Mar 20, 16:29 UTC
Mar 19, 2017

No incidents reported.

Mar 18, 2017

No incidents reported.

Mar 17, 2017
Resolved - User reports of issues have cleared up as of approximately Noon PST. We apologize for the delays in resolving this issue. Please email support@travis-ci.com if you see any further issues.
Mar 17, 02:45 UTC
Update - We are seeing some reports of similar intermittent slowness from both nodejs.org and registry.yarnpkg.com from other users outside of Travis CI's infrastructure, which seems to indicate a potential upstream issue.
Restarting affected builds can help and this is recommended for the moment.
We are continuing to investigate.
Mar 16, 15:30 UTC
Investigating - We’re currently investigating slowness and connectivity issues on `sudo: false`, container-based builds. This seems to be specially affecting Node.js installations via nvm.
Mar 16, 14:20 UTC
Mar 16, 2017
Resolved - At this time things are now running at our current full capacity for public macOS builds. Please email support@travis-ci.com if you have any questions. Thank you for your patience while we worked through some underlying improvements that required reduced capacity today.
Mar 16, 02:40 UTC
Update - We are beginning to add capacity back for public macOS jobs and will provide an update when this process is finished.
Mar 16, 01:29 UTC
Identified - We are in the process of reducing capacity for public macOS builds, in order to support moving some of our hardware capacity for infrastructure improvements. We'll be at reduced capacity for a few hours and we'll provide periodic updates. Please email support@travis-ci.com if you have any questions.
Mar 15, 18:08 UTC