Piano Software Status
[Piano Analytics] Delay in generating yesterday's data

Started

Resolved

Duration

1 hr 47 min

Postmortemmajor impactVendor link

Update timeline

Resolved

The issue has been resolved and the delay in data generation has now been caught up. All scheduled exports have been sent.

Monitoring

We are continuing to monitor for any further issues.

Monitoring

A fix has been implemented and data recovery is currently underway (already completed for around 90% of customers).

Investigating

We are continuing to investigate this issue.

Investigating

We are currently experiencing a delay in data generation for yesterday on Piano Analytics. Daily scheduled exports may be still pending to be sent. Hourly and dataflow exports are not affected by the problem. Our technical teams are actively working to resolve the issue. Please excuse us for any inconvenience caused.

Postmortem—

**Incident explanation** * Due to inconsistencies in our delivery process, two services with incompatible versions were deployed to production during a full redelivery of all services, aimed at quickly resolving a security vulnerability * These services are part of our data ingestion pipeline, which became blocked. As all consolidation and aggregation jobs depend on this step, they could not be computed, delaying data availability. ‌ **Action performed** * Both services were rolled back to their last stable versions, which unblocked the process. ‌ **Actions to come** * Strengthen validation of this part of the pipeline during deliveries. Set up an on-call rotation for the D-1 data feed \(details to be defined\).

[Piano Analytics] Delay in generating yesterday's data — Piano Software Incident Timeline & Status — DevHelm