Small subset of artifact and cache requests in Illinois region failing — Namespace Incident Timeline & Status

Try DevHelm Uptime

Namespace Status

Small subset of artifact and cache requests in Illinois region failing

Started

Sep 24, 5:16 PM UTC

Resolved

Sep 24, 6:45 PM UTC

Duration

1 hr 29 min

Resolvedminor impact

Affected:Artifacts Bazel Gradle HTTP Cache Illinois, US Turborepo

Update timeline

ResolvedSep 24, 6:45 PM

Error rates have recovered, all systems operational.

MonitoringSep 24, 6:37 PM

We disabled all affected replicas.

IdentifiedSep 24, 6:00 PM

The new replica showed the same stalling pattern. We identified a networking misconfiguration that's causing the issue.

IdentifiedSep 24, 5:38 PM

We identified and stopped a single replica that was causing the stalls.

InvestigatingSep 24, 5:16 PM

We were alerted to an issue with the system backing storage for Artifacts and Cache storage in our Illinois region. A small number of requests is stalling and timing out.