This incident has been resolved.

All new servers have been fixed. We are still increasing data density in the servers and monitoring any latency spikes.

We are continuing to work on a fix for this issue.

The fix improved nvme behavior during the day and today we applied it on half of the concerned rack. As data must be removed first to apply the fix, migrations operations are processing slowly to avoid worsening impact on latencies. Migration operations will continue through the week-end.

We applied a fix on one of the concerned servers and monitoring it, we will have more significant metrics tomorrow.

Since monday 26 october on the morning, we noticed increased periodic spike of latencies following the integration of a new rack in one of our cluster. Those spikes are more noticeable during the day and are linked to writes done on some of the new disks. This issue also impact MRDB product. We limited the impact and currently we are working on a fix. An update will be posted at the end of the day.