Jobs are processing normally. Thank you for your patience.
Posted Mar 01, 2017 - 03:40 UTC
Update
We are processing normally, though there is still a job processing backlog. We are monitoring stability closely while the backlog clears.
Posted Mar 01, 2017 - 02:12 UTC
Monitoring
We are currently processing a large job backlog as part of fallout from the S3 incident. All services are functioning normally, but you may still notice delays until the backlogs clear.
Posted Mar 01, 2017 - 00:43 UTC
Update
We are seeing some services recover after the S3 outage, and jobs are processing. We are watching recovery closely, and will keep updating if anything goes awry.
Posted Feb 28, 2017 - 23:14 UTC
Update
We are still waiting for Amazon S3 to recover.
Posted Feb 28, 2017 - 21:11 UTC
Identified
AWS has confirmed that S3 is experiencing issues. We've taken some actions to maintain current container-based capacity and are monitoring the S3 status and overall healthy of our infrastructure closely.
Posted Feb 28, 2017 - 18:10 UTC
Investigating
We are investigating issues related to the S3 outage in AWS. Currently build logs older than a few hours will fail to load. Build caches for container-based builds are unavailable. Builds that depend on resources like Docker Hub, Quay.io, or other S3 dependent third party services will fail with errors related to being unable to access the resources.