Intermittent DNS issues resolving our website domains

Incident Report for Travis CI

Postmortem

At around 19:15 UTC on December 1, 2014, our DNS service provider started receiving a massive Distributed Denial of Service (DDoS) attack.

The attack was volumetric, and lasted 13 hours.

Impact of DDoS attack

DDoS overwhelms a server's capacity to respond to requests, so that other legitimate requests are left unanswered. In this scenario, DNS queries for our services at *.travis-ci.org and *.travis-ci.com timed out, rendering our services unreachable for the most part.

During the past service interruptions, we posted updates on http://status.travis-ci.com, and this incident was no different. However, this service was also unreachable for the same reason that our services were unreachable. Because of this, our communication with our users was limited to updates on Twitter.

What we are doing about it

This was a particularly prolonged DDoS attack. Our fantastic DNS provider DNSimple fought against this malicious attack valiantly. You can read their postmortem here.

We have set up a new domain, traviscistatus.com to communicate the status of our services. This domain is served by a different DNS provider.

We are investigating a secondary DNS provider, which will allow for a better failover in cases where the primary is unavailable.

If you have further questions, please feel free to get in touch with us via email.

Posted Dec 03, 2014 - 17:16 UTC

Resolved

The DDoS attack has ceased and all sites should be reachable again. We'll follow up with more information and possible remediation steps once we have them. We're sorry for the extended outage.
Posted Dec 02, 2014 - 08:46 UTC

Update

Hostnames are resolving again and, according to our DNS provider, the large scale DDoS attack on their servers has ceased.
Posted Dec 02, 2014 - 07:38 UTC

Update

As a response for future DNS issues and to improve communication visibility, we've moved our status page to http://www.traviscistatus.com.
Posted Dec 02, 2014 - 07:13 UTC

Update

Our DNS provider reports that their data centers are beginning to resolve requests correctly.
Posted Dec 02, 2014 - 05:14 UTC

Update

Our DNS provider is still working on routing issues to Amsterdam and Tokyo data centers. This may be causing problems for some European and Asia Pacific users.
Posted Dec 02, 2014 - 02:11 UTC

Monitoring

DNS server is coming back online. There is a huge backlog in build queue, which we are working to resolve.
Posted Dec 02, 2014 - 00:24 UTC

Update

Our DNS provider has identified a solution and is working on rolling this out.
Posted Dec 01, 2014 - 23:33 UTC

Update

Our DNS provider is working to mitigate DDoS attacks.
Posted Dec 01, 2014 - 22:04 UTC

Update

Our DNS provider is working on mitigating DDoS attacks
Posted Dec 01, 2014 - 20:52 UTC

Identified

Our DNS provider has identified DDoS attack, and is working to resolve the issue.
Posted Dec 01, 2014 - 19:56 UTC

Investigating

Our upstream provider is experiencing issues resolving names, which can cause DNS resolutions to fail.
Posted Dec 01, 2014 - 19:19 UTC