Travis CI
All Systems Operational
API   Operational
90 days ago
99.97 % uptime
Today
Web App   Operational
90 days ago
99.97 % uptime
Today
Builds Processing Operational
90 days ago
98.66 % uptime
Today
Mac Builds   ? Operational
90 days ago
96.01 % uptime
Today
Linux Builds (container-based)   ? Operational
90 days ago
99.97 % uptime
Today
Linux Builds (Trusty and Precise on GCE)   ? Operational
90 days ago
100.0 % uptime
Today
Background Processing Operational
90 days ago
100.0 % uptime
Today
Log Processing   Operational
90 days ago
100.0 % uptime
Today
User Sync   ? Operational
90 days ago
100.0 % uptime
Today
Notifications   ? Operational
90 days ago
100.0 % uptime
Today
Third Party Services Operational
GitHub   Operational
DNSimple Name Servers   Operational
Heroku   Operational
Pusher Pusher REST API   Operational
Pusher WebSocket client API   Operational
Pusher Presence channels   Operational
Pusher Webhooks   Operational
AWS ec2-us-east-1   Operational
Help Scout Email Processing   Operational
Help Scout Web App   Operational
PagerDuty Notification Delivery   Operational
Google Cloud Platform Google Compute Engine   Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
System Metrics Month Week Day
API Uptime (api.travis-ci.org)
Fetching
API Uptime (api.travis-ci.com)
Fetching
Active Linux Builds for Open Source projects
Fetching
Backlog Linux Builds for Open Source projects
Fetching
Active macOS Builds for Open Source projects
Fetching
Backlog macOS Builds for Open Source projects
Fetching
Active Linux Builds for Open Source projects (container-based)
Fetching
Backlog Linux Builds for Open Source projects (container-based)
Fetching
Past Incidents
Sep 23, 2017

No incidents reported today.

Sep 22, 2017
Resolved - We have worked through the accumulated backlog for private macOS builds which are now performing normally. We are continuing to monitor this closely. Thanks for your patience!
Sep 22, 10:58 UTC
Monitoring - In an effort to stabilize our macOS infrastructure, yesterday at 17:00 UTC we also reduced the capacity available for macOS private repositories. We've now been able to address this and since 08:40 UTC, macOS Private builds are now running as expected. Thank you for your understanding.
Sep 22, 09:42 UTC
Resolved - We've worked through the accumulated backlog for public macOS builds which are now performing normally. We are continuing to monitor this closely. Thanks for your patience!
Sep 22, 10:03 UTC
Update - We have worked through most of the backlog over the last few hours. We continue to investigate the root cause for instability and we’ll post an update as soon as we can.
Sep 21, 09:00 UTC
Update - We've completed the first round of job cancellations and are seeing some improvements. We're currently evaluating other changes to help reduce the backlog and improve the wait time for jobs starting. We'll provide more updates as we learn more. Thank you for your patience.
Sep 20, 15:44 UTC
Update - In an effort to clear up load on macOS servers, we are cancelling builds that have been waiting to start for more than 6 hours. This process will start at 16:15 CEST and is expected to take approximately 2 hours to complete.

We will also be rolling back to a previous version of the worker to determine if recent changes have contributed to these issues.
Sep 20, 13:47 UTC
Update - We continue to battle with stability issues on the MacOS platform, which is the root cause of severe wait times for public repository builds. We will keep you posted about any updates. We apologize for the delays this is causing.
Sep 20, 10:49 UTC
Update - We have not made any significant gains in understanding the source of instability, but changes to available capacity distribution are helping to reduce the impact of disconnections.
Sep 20, 01:33 UTC
Update - We are continuing to investigate the heightened AMQP timeout errors. The backlog for public OSX / mac OS jobs remains high.
Sep 19, 17:41 UTC
Investigating - We’re investigating a higher-than-normal AMQP timeout errors affecting the throughput of our builds. This is affecting public OSX / mac OS jobs most at the moment, as this is the highest demand queue.
Sep 19, 10:50 UTC
Sep 21, 2017
Resolved - This incident has been resolved.
Sep 21, 15:52 UTC
Monitoring - We have incurred a backlog on our sudo-enabled private repository queue while performing a graceful restart. We expect the backlog to clear once the full capacity is back online after finishing the longest-running jobs.
Sep 21, 15:28 UTC
Sep 18, 2017
Resolved - This incident has been resolved.
Sep 18, 21:31 UTC
Monitoring - We can see that the backlog for sudo-enabled builds running on GCE has cleared. We are continuing to roll out the fix to our other infrastructures.
Sep 18, 20:56 UTC
Update - We've identified that a new backend change was having a unexpected negative impact on communications with a message queue and was leading to an increased backlog. We've test out a configuration change to disable this new change and it's having the positive impact we expected, so we're continue to roll it out to all parts of our infrastructure. We'll provide updates as this rollout progress.
Sep 18, 20:03 UTC
Investigating - We’re investigating an anomaly in demand that is causing backlog on public Linux `sudo: required` builds running at travis-ci.org
Sep 18, 17:17 UTC
Sep 17, 2017
Completed - The maintenance is complete, thanks for bearing with us! 💛
Sep 17, 10:50 UTC
Update - Maintenance of travis-ci.com is complete and we continuing processing of jobs.

We are now beginning maintenance on travis-ci.org.
Sep 17, 10:02 UTC
In progress - We are beginning our scheduled maintenance on travis-ci.com and travis-ci.org.
Sep 17, 08:58 UTC
Scheduled - We are performing some scheduled maintenance on travis-ci.org and travis-ci.com on Sunday, 18 September, 2017 From 09.00 AM UTC to 12:00 PM UTC.

We expect the API and web interface to be unavailable for some of that time window on both platforms. Processing of public and private builds is also expected to be delayed.
Sep 11, 16:19 UTC
Sep 16, 2017

No incidents reported.

Sep 15, 2017

No incidents reported.

Sep 14, 2017
Resolved - AWS S3 issues have been resolved and all services are operating normally.
Sep 14, 20:34 UTC
Identified - AWS is reporting issues with S3 in us-east-1, 11:58 AM PDT We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region. WE can confirm we're seeing these issues as well. While S3 is unstable you'll see errors from build caching/artifacts activities and may have trouble accessing older build logs, which are stored in S3 long term. We will provide updates as we learn more.
Sep 14, 19:08 UTC
Resolved - A tiny backlog remains for private Mac builds but it should be cleared in the next 30 minutes. Hence, we are resolving this incident for now. Thank you for your enduring patience!
Sep 14, 20:08 UTC
Update - The backlog for container-based (i.e. sudo: false) Linux builds has cleared. A backlog remains for Mac builds and we will update you when it's cleared. Thank you!
Sep 14, 18:54 UTC
Update - We are happy to report that the backlog has cleared for sudo-enabled Linux builds.

Small backlogs remain for container-based Linux and Mac builds.
Sep 14, 18:39 UTC
Monitoring - We are sorry to inform you that the previous incident (https://www.traviscistatus.com/incidents/4gy46v0t3vrq), although it's fixed, resulted in a backlog for private builds. Hence you might experience some delays with your builds. Sorry for the inconvenience.

We are monitoring things closely and we will update with the state of the backlog on our different infrastructures in a timely manner.

Thank you for your patience!
Sep 14, 18:23 UTC
Resolved - Travis API for .com private builds is back up. Our redis service had to be migrated due to a hardware failure on AWS in the us-east-1 region.
Sep 14, 17:36 UTC
Identified - Our API for .com builds is down due to our Redis instances being unavailable. We are working with our 3rd party redis hosting service.
Sep 14, 17:21 UTC
Sep 13, 2017
Resolved - Log processes have stabilized for private builds on travis-ci.com
Sep 13, 19:06 UTC
Investigating - We are investigating delays in log processing for private builds on travis-ci.com. Log streaming is unaffected.
Sep 13, 15:30 UTC
Sep 12, 2017

No incidents reported.

Sep 11, 2017
Postmortem - Read details
Sep 20, 19:12 UTC
Resolved - The public macOS build backlog has reached normal peak levels and things are remaining stable. We're closing the incident at this time. A postmortem blog post will be published in the next few days and we'll share it on Twitter when it's published. Thank you everyone for your patience and understanding during this extended incident.
Sep 11, 21:11 UTC
Update - The backlog has cleared for private builds. We are continuing to monitor the situation for public/open source builds. Thanks for hanging in there with us.
Sep 11, 19:52 UTC
Update - We're resuming full private macOS build capacity.
Sep 11, 16:50 UTC
Update - We're seeing some instability with some of the private macOS build capacity and so we're reducing capacity temporarily.
Sep 11, 15:47 UTC
Monitoring - We've resumed full build capacity for public builds. We will be monitoring things overnight and will provide further updates in the morning PDT. Thank you for your patience.
Sep 11, 03:57 UTC
Update - We've completed the first phase of our SAN cleanup. Things are stable and so we're working to resume full public macOS build capacity. We'll provide another update when that's complete.
Sep 11, 03:45 UTC
Update - We're now running with the previous capacity for public builds, which is still reduced from our "normal" capacity. We are continuing with SAN cleanup. We'll provide updates as things progress today. Thank you for your patience.
Sep 10, 19:56 UTC
Update - We temporarily have additional reduced capacity for public builds, as we take some actions to continue with our SAN cleanup. We'll provide another update when that capacity has been restored.
Sep 10, 19:45 UTC
Update - We've processed a backlog of approximately 9,600 macOS jobs for public repositories since re-enabling public macOS builds at 07:00 PDT yesterday. As we're still at reduced capacity and working on cleaning the SAN, we still have a backlog of ~150-200 jobs and continue to actively process them. We'll provide updates as things progress today. Thank you for your patience.
Sep 10, 16:45 UTC
Update - We're continuing to process the public backlog while running SAN cleanup. We may still need to reduce or suspend public builds later in the weekend, depending on SAN progress. Thank you for your patience.
Sep 9, 15:11 UTC
Update - Capacity for macOS public repositories has been back online for ~1 hr. We're bumping additional capacity to work through the backlog.
Sep 9, 14:32 UTC
Update - The backlog for private repository builds has been clear for ~4h. We are planning to bring partial capacity for public repositories back online shortly.
Sep 9, 13:04 UTC
Update - We've resumed running private builds at this time. We'll provide further updates on the overall progress tomorrow morning PDT. Thank you for your patience.
Sep 9, 03:13 UTC
Update - We ran into an issue with booting Xcode 8.x images, so all builds are suspended again. We'll update when private builds are running.
Sep 9, 01:28 UTC
Update - In order to help things become stable and reliable going forward, we're undertaking intense cleanup of our SAN filesystem. This cleanup is likely to take all weekend. Because of this, we're only able to resume a portion of our capacity for private builds and will not be resuming shared public builds yet. We do not currently have an ETA for when we'll be able to resume shared public builds. We will provide our next update in the morning PDT. We are very sorry for the delays and will update this incident when we know more. Thank you for your patience.
Sep 9, 01:04 UTC
Update - We're working on stabilization cleanup for our SAN storage. At the moment we do not have an ETA for when we will resume builds. We are very sorry for the delays and will update this incident when we know more. Thank you for your patience.
Sep 8, 21:29 UTC
Update - We're continuing to work on getting things into a stable state where we can potentially start running builds. At the moment we do not have an ETA for when we will resume builds. We are very sorry for the delays and will update this incident when we know more. Thank you for your patience.
Sep 8, 19:52 UTC
Update - We've rebooted our vCenters and continue to work on stabilizing things. All macOS builds remain stopped thank you for patience.
Sep 8, 16:52 UTC
Update - macOS jobs for public and private repositories builds are stopped.
We're currently working with our infrastructure provider to reboot one of our vCenter instances to work out unresponsive SAN issues.
Sep 8, 15:01 UTC
Update - We are stopping all macOS jobs because we have run out of space on our datacenter's SAN.
Sep 8, 12:34 UTC
Identified - We've identified an issue with some of our Xcode image hosts, causing macOS requeues on both public and private repositories. We're working together with our upstream provider to sort this out while we continue investigating macOS build timeouts.
Sep 8, 11:07 UTC
Investigating - We continue investigating mac OS requeues and build timeouts for both public and private repositories. This seems to be related to SAN performance, we'll continue posting updates as we work to get a more stable performance.
Sep 8, 10:32 UTC
Monitoring - The stability of our macOS builds seems to have improved. We will continue to monitor the rate of errored builds. Thank you for you patience.
Sep 8, 08:56 UTC
Investigating - Repositories running on travis-ci.org and travis-ci.com are experiencing an increase in errored builds. We are investigating and will update as soon as we can.
Sep 7, 09:36 UTC