A significant fraction of Clickhouse replicas marked as unhealthy
Started
Resolved
Duration
1 hr 20 min
Resolvedminor impact
Affected:API EndpointsAPI EndpointsAPI EndpointsAPI EndpointsAPI EndpointsAPI EndpointsAPI EndpointsAPI EndpointsBI ConnectorBI ConnectorCSV ImportsCSV ImportsCSV ImportsCSV ImportsClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceClickHouse HTTP InterfaceCopyCopyCopyCopyCopyCopyCopyCopyDynamoDB ConnectorDynamoDB ConnectorDynamoDB ConnectorEvents APIEvents APIEvents APIEvents APIEvents APIEvents APIEvents APIEvents APIGCS ConnectorGCS ConnectorGCS ConnectorGCS ConnectorGCS ConnectorGCS ConnectorGCS ConnectorGCS ConnectorKafkaKafkaKafkaKafkaKafkaKafkaKafkaKafkaLoginLoginLoginLoginLoginLoginLoginLoginQuery APIQuery APIQuery APIQuery APIQuery APIQuery APIQuery APIQuery APIS3 ConnectorS3 ConnectorS3 ConnectorS3 ConnectorS3 ConnectorS3 ConnectorS3 ConnectorS3 ConnectorSinkSinksSinksSinksSinksSinksSinksSinks
Update timeline
Resolved
We've confirmed that service has been restored fully.
Monitoring
We have narrowed the root cause to an automatic Ubuntu package upgrade triggering the restart of our database services on many servers, and those being marked unhealthy as a result. Service should be generally restored already.
Monitoring
We've marked as healthy the database replicas that were incorrectly marked as unhealthy. We're confirming the cause and have a few suspects. We keep monitoring the situation.
Investigating
Some database replicas have been marked as unhealthy incorrectly. This is affecting how traffic is distributed among healthy replicas, and in some clusters it's resulting in degraded performance and some errors.