Turbot Status
Pipes Authentication Errors

Started

Resolved

Duration

15 hrs 57 min

Resolvedmajor impactVendor link
Affected:Core Services

Update timeline

Monitoring

On July 20, between approximately 14:00 and 21:00 UTC, the scheduled process that refreshes temporary credentials for cloud connections stopped completing. As existing credentials expired (beginning around 15:00 UTC), live queries against affected connections, primarily AWS connections using cross-account roles, returned authentication errors such as ExpiredToken. Cached datatank and snapshot queries were not affected. Engineering identified and resolved the issue at approximately 21:00 UTC. Temporary credentials have been rotated across all affected connections, and live queries are working normally. A sweep is running to backfill any datatank syncs that were missed during the impact window, and we are continuing to monitor. We apologize for the disruption. A root cause analysis is underway, and we will follow up with affected customers as needed.

Resolved

Monitoring indicates the the issue has been resolved.

Investigating

We are currently investigating an issue impacting pipes ability to authenticate to AWS cloud providers.

Pipes Authentication Errors — Turbot Incident Timeline & Status — DevHelm