Cornerstone Status
Major Issue - System availability issues on EU (FRA) Production

Started

Resolved

Duration

3 hrs 29 min

Postmortemmajor impactVendor link
Affected:Uptime

Update timeline

Resolved

This issue has been fully resolved, and all corrective actions have been completed successfully. No further impact is expected.

Monitoring

A fix for this issue was implemented, and the issue has been mitigated. We are monitoring the swimlane performance before concluding this incident as resolved.

Investigating

The swimlane is experiencing some latency issues. Clients with portals in this swimlane may experience delays or issues accessing certain pages in the application. This is our top priority and we are working to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Postmortem—

**Root Cause Analysis**   **Incident Overview:** On October 19th, 2026, at approximately 2:16 AM PT, a service disruption impacted the EU region \(FRA SL1\) preventing customers from accessing the application login page. Requests to the affected services returned HTTP 504 Gateway Timeout errors. During the incident, origin endpoints URLs were inaccessible across all three environments, disrupting access to the affected services. Service availability recovered automatically at approximately 2:45 AM PST, restoring access to the affected production environments. **Impact:** During the incident, customers experienced login failures and were unable to access the application login page. Requests returned HTTP 504 Gateway Timeout errors because the Content Delivery Network \(CDN\) was unable to reach the origin endpoints. **Root Cause:** The incident was attributed to an AWS-side service disruption that affected connectivity between the CDN and the origin endpoints. During the investigation, AWS confirmed that the CDN was unable to reach the origin endpoints. This prevented requests from reaching the application services and resulted in HTTP 504 Gateway Timeout errors. Internal investigations confirmed that shared components, including databases and DNS, remained operational during the incident. A sharp decline in incoming traffic was also observed, consistent with the reported service accessibility issues. **Preliminary Findings from AWS:** After the incident was detected, AWS acknowledged an outage on its end that contributed to the disruption. Service availability recovered automatically at approximately 2:45 AM PST, without manual intervention from the operations team. A detailed RCA from AWS is still pending to confirm the underlying failure mechanism, the services and components affected, and the corrective and preventive measures being implemented to minimize the risk of recurrence. **Resolution and Recovery:** Service availability recovered automatically at approximately 2:45 AM PST on October 19th, 2026. Following recovery, access to the affected production environments was restored. No manual intervention was required to trigger service recovery. **Corrective and Preventive Actions:** The following actions are planned to complete the investigation and identify opportunities to improve service resilience: 1. **Obtain the final AWS RCA:** Continue working with AWS to obtain the completed event analysis, including confirmation of the suspected KMS API errors and the sequence of events that led to the regional service disruption. 2. **Assess monitoring and alerting:** Review existing monitoring for CDN-to-origin connectivity, HTTP 504 errors, origin endpoint availability, and sudden declines in incoming traffic to identify opportunities for earlier detection and faster diagnosis. 3. **Evaluate service resilience:** Based on AWS's final findings, assess whether additional monitoring, failover, or recovery mechanisms are required within our environment to reduce customer impact during similar infrastructure disruptions. 4. **Update the incident documentation:** Incorporate AWS's confirmed findings into the incident record and update this RCA with the final technical explanation and any additional corrective or preventive actions. **Current Status and Next Steps:** The incident is resolved, and service availability has been restored. Based on AWS's initial assessment, an increase in KMS API errors is the suspected trigger for a broader cascading impact across multiple AWS services in the region. This may have contributed to the connectivity failure between the CDN and the origin endpoints. However, the precise failure mechanism and the sequence of downstream impacts remain under investigation by AWS. **Further details will be provided as soon as AWS completes its investigation and shares the detailed RCA.** The final findings will be reviewed to determine whether any additional corrective or preventive measures are required within our environment.

Major Issue - System availability issues on EU (FRA) Production — Cornerstone Incident Timeline & Status — DevHelm