Spacelift Status
US Customers MQTT Service Disruption

Started

Resolved

Duration

1 hr 22 min

Postmortemmajor impactVendor link

Update timeline

Resolved

This incident has been resolved.

Monitoring

We deployed the fix and monitoring.

Identified

We have the fix implemented and we will release it shortly.

Investigating

We’re seeing an increased number of errors affecting the worker polling API for customers in our US region. Our team is actively working on mitigation and will share updates as they become available.

Postmortem—

**Root Cause** A change in a new launcher version caused it to connect by default to the EU endpoint if the region was not explicitly configured. Workers registered in other regions were then rejected at authentication. This was a human error. Detection was delayed for three reasons: * Our pre-release testing did not cover workers outside the EU region well enough. * Launcher authentication errors usually come from individual customer misconfiguration, so the increase was not flagged as a platform issue. * MQTT disconnections coincided with our planned MQTT-to-polling migration, which made the drop look expected. **Mitigations** * **Immediate:** We rolled back to a working version of the launcher * Before every deployment, we will now boot fresh private workers in other regions and confirm that they connect and complete a run. A failure like this should be caught before a release. * We are improving alerting so that authentication failures across multiple customers are flagged automatically as a potential platform issue. **Conclusion** A broken binary caused launchers without a configured region to target the wrong endpoint. Gaps in regional test coverage and alerting meant we did not catch it before customers did. Additional automated checks are being put in place so that this failure would be detected before release along with improved alerting so that we are notified sooner for issues that make it past our automated checks.

US Customers MQTT Service Disruption — Spacelift Incident Timeline & Status — DevHelm