Replicate Status
Delayed scaling due to node failure

Started

Resolved

Duration

36 minutes

Resolvedminor impact

Update timeline

Resolved

All affected models have been scaling correctly for more than 30 minutes at this point, and we see no residual prediction queues. Thank you for your patience!

Monitoring

Scaling decisions were delayed by nearly 1 hour after the controller responsible for emitting queue metrics failed to schedule on a soft-failed node. We have since cordoned and drained the node, and the controller is emitting queue metrics again.

Delayed scaling due to node failure — Replicate Incident Timeline & Status — DevHelm