Amazon continues to faces connectivity issues after widespread AWS outage
AWS continues to face issues despite initial fix
Earlier today, Amazon Web Services (AWS) suffered an outage that took down many services that relied on its servers. This included games like Fortnite and Roblox, banking services, and a huge number of websites and apps. Now, Amazon has applied “multiple mitigations” to address these issues, though some problems remain.
Many websites and applications are returning to normal operating conditions. However, not everything at AWS is running smoothly. Amazon has confirmed that there still are “significant API errors and connectivity issues” on their web services. Currently, new EC2 instances are throttled, preventing Amazon’s “Elastic Compute Cloud” users from accessing additional resources when required. In other words, these services are not “Elastic”, at least currently.
While games like Fortnite and many other popular apps have returned online, it may be another while before Amazon’s Web Services return to normal.
Operational issue – Increased Error Rates and Latencies
Oct 20 6:42 AM PDT We have applied multiple mitigations across multiple Availability Zones (AZs) in US-EAST-1 and are still experiencing elevated errors for new EC2 instance launches. We are rate limiting new instance launches to aid recovery. We will provide an update at 7:30 AM PDT or sooner if we have additional information.
Oct 20 7:14 AM PDT We can confirm significant API errors and connectivity issues across multiple services in the US-EAST-1 Region. We are investigating and will provide further update in 30 minutes or soon if we have additional information.
Oct 20 7:29 AM PDT We have confirmed multiple AWS services experienced network connectivity issues in the US-EAST-1 Region. We are seeing early signs of recovery for the connectivity issues and are continuing to investigate the root cause.
Oct 20 8:04 AM PDT We continue to investigate the root cause for the network connectivity issues that are impacting AWS services such as DynamoDB, SQS, and Amazon Connect in the US-EAST-1 Region. We have identified that the issue originated from within the EC2 internal network. We continue to investigate and identify mitigations.
Oct 20 8:43 AM PDT We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations.
Oct 20 9:13 AM PDT We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT.
It may be a while before Amazon/AWS fully recovers from its outage. For now, the severity of the issues facing AWS has been lessened. Even so, this outage highlights the problem that AWS’ huge scale enables. Amazon accounts for 30% of the global cloud computing market, with Microsoft Azure and Google Cloud accounting for an additional 20% and 13% respectively. That’s over 60% of the world’s cloud services controlled by three companies.
You can join the discussion on Amazon’s continued web services issues on the OC3D Forums.
