After taking remedial action, Real Device test error rates have returned to normal in the US-West-1 data center. All services are fully operational.

The root cause of the elevated error rates for Real Device tests in the US-West-1 data center has been identified. Remediation has been applied, and we are actively monitoring the situation.

We are currently seeing elevated error rates for automated real device tests in US-West-1 data center. We are currently investigating.

### **Dates:** Monday, September 28th 2026, 14:19 UTC - 16:52 UTC. ### **What happened:** Real device tests in US West hung and failed to output test artifacts due to slow uploads. ### **Why it happened:** A degraded interconnect link suffered high packet loss from optical errors, but BGP stayed up, preventing automatic failover to healthy paths. ### **How we fixed it:** Manually disabled the degraded port on the edge router, forcing traffic onto healthy alternate interconnects. ### **What we are doing to prevent it from happening again:** Automating link draining based on CRC error thresholds, improving protocols for isolating flapping vendor links, and hardening upload resilience.