This incident has been resolved. Between 12:08 and 14:23 UTC, customers were unable to access the Cloud Dashboard and the Analytics Services APIs. The cause was a control plane failure in our last remaining legacy Kubernetes cluster, which took out a dependency of those services. Our user-facing services (RTC, SIP, and others) were unaffected, as they run redundantly across our modern clusters and have no dependency on the legacy cluster. We have no indication of data loss. Recovery took longer than we would like, as mitigation required working within the legacy cluster. Work was already underway to make the affected dependency redundant across our modern clusters, and that cluster is scheduled for decommissioning, which removes this class of failure. We will follow up with a postmortem.

A fix has been deployed as of 14:23 UTC, and access to the Cloud Dashboard and Analytics Services APIs is returning to normal. We are continuing to monitor before marking this resolved. Recovery may take a few additional minutes for some clients as the change fully propagates; retrying or reconnecting will restore access.

We have identified the cause of the issue affecting the Cloud Dashboard and Analytics Services APIs, and recovery work is underway. Customers may continue to see errors accessing the Cloud Dashboard and Analytics Services APIs during this time. Room APIs, RTC APIs, and all other services remain healthy and unaffected. We have no indication of data loss from this incident. We will post another update as recovery progresses.

We are continuing to investigate this issue.

We are continuing to investigate this issue affecting LiveKit Cloud dashboard, Analytic Services APIs and Billing APIs. Users may be unexpectedly signed out and unable to sign back in, and API requests may return 522 errors.

We are investigating an issue where users may be unexpectedly signed out of the LiveKit Cloud dashboard and unable to sign back in.