Midaxo Cloud US Instance was unavailable from 7:28 PM to 7:48 PM UTC due to increased error rates in the Amazon S3 service in the AWS Ohio region.

# Midaxo Cloud US Production Outage Post-Mortem Report ## Incident Overview On October 7th, 2024, from 7:28 PM to 7:48 PM UTC, Midaxo Cloud US instance experienced a service disruption lasting approximately 20 minutes. During this period, all customers were unable to access Midaxo Cloud services. ## Root Cause The outage was caused by a cascading failure originating from Amazon Web Services \(AWS\) S3 service in the Ohio region \(US-EAST-2\). A software bug within AWS S3 triggered a series of failures that impacted over 30 AWS services, including those critical to Midaxo Cloud operations. ## Timeline of Events * **Oct. 7th, 7:28 PM UTC**: Midaxo Cloud US production environment began experiencing issues. * **7:28 PM UTC**: Automated monitoring systems detected the outage and triggered alerts. * **7:28 PM UTC**: Public status page at [http://status.midaxo.com/](http://status.midaxo.com/) updated to show service disruption. * **7:48 PM UTC**: AWS mitigated the outage, restoring availability for all affected services. * **7:54 PM UTC**: Automated monitoring systems reported recovery from the outage. * **Oct. 8th, ~2:00 AM UTC**: AWS provided an update on the identification and resolution of the underlying issue. ## Impact All Midaxo Cloud customers were unable to access the service during the 20-minute outage period. ## Resolution and Recovery Our team quickly identified that the issue originated from AWS after being alerted on the incident. As the problem was on AWS's end, our recovery was dependent on their resolution of the S3 service issues and we focused on monitoring the situation. AWS was able to mitigate the outage within 20 minutes, at which point our services were restored to full functionality. ## Corrective and Preventative Measures While this incident was outside of our direct control, we are taking the following steps to improve our response and resilience: 1. Enhancing our communication protocols during third-party service outages. 2. Evaluating our automated monitoring and alert systems to ensure they continue to provide timely and accurate information. ## Additional Information AWS provided the following details about the incident on their [status page](https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-2::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_67084_0A6A325AD2F): * The issue began at 12:27 PM PDT \(7:27 PM UTC\) with increased error rates for S3 GET/PUT requests in the US-EAST-2 Region. * The root cause was identified as a latent software issue in a subsystem of S3 responsible for assessing metadata during S3 PUT and GET operations. * AWS has implemented mitigations, including removing the software issue in the S3 request path. * New testing procedures have been identified to detect similar software issues in the future. We appreciate your patience and understanding during this service disruption. Midaxo is committed to providing reliable service and will continue to work on improving our systems' resilience. If you have any questions or concerns, please don't hesitate to contact our support team.