Started
Resolved
Duration
1 day
Update timeline
The issue that prevented ODC users across multiple regions from previewing data in ODC Studio, and that may also have caused errors in application runtime, has been resolved. A fix was applied and our team has monitored the platform to confirm full recovery. Data preview is working as expected and all services are operating normally. We apologise for any disruption caused and thank you for your patience.
The fix has now been fully applied across all affected regions, and error rates have returned to normal levels. Users should once again be able to preview data in ODC Studio and access their applications as expected. We are continuing to monitor the platform closely and are confirming full recovery with affected users. We will provide a further update once we have confirmed the incident is fully resolved.
We have identified the cause of this issue as a recent database authentication update that failed validation during rollout. The update has been reverted, and a fix is currently being deployed. As the fix is applied across affected regions, users may still experience difficulty previewing application data in ODC Studio or encounter errors when accessing their applications. We will provide an update as soon as more information becomes available.
We are investigating an issue affecting ODC users across multiple regions. Users are currently unable to preview data in ODC Studio and may also encounter errors in application runtime. The issue is associated with the error message: "FATAL: PAM authentication failed for user "osruntime"". Our engineering team is actively investigating the issue, and we will provide an update as soon as more information is available.
On September 29th, starting at approximately 8:53 AM UTC, we experienced an issue affecting ODC users across multiple regions. Users were unable to preview data in ODC Studio. Despite early suspicions of impact in runtime, our investigation did not find evidence that running applications were actually affected. The issue was detected through customer reports. Our investigation identified a recent software update related to database authentication as the source of the issue. In certain circumstances, the update caused database authentication to fail when users previewed data in ODC Studio. The conditions that triggered the problem were not present in our earlier test environments, so it was not caught before the update reached production. Rolling the update back restored normal operation, and we are finalising confirmation of the precise mechanism as part of our post-incident review. Error rates fell sharply from approximately 11:24 UTC, once the fix was applied. Recovery was then confirmed with affected users later that day. We continued to monitor the platform and marked the incident as resolved on September 30th, with no further impact. To prevent a recurrence, a corrected version of the update has been developed and is being reviewed, and related changes will not proceed until it has been validated. We apologise for any disruption caused and thank you for your patience.