The incident has been resolved. Please do not hesitate to reach out to support@redoxengine.com if you are continuing to run into anything unexpected.

We applied a fix at 2:23pm Central and processing appears to be operating as normal. We are still seeing delays in log visibility. We will continue to monitor to verify that the issues are fully resolved with our 3rd party vendor.

Our automated monitoring tools detected a major outage affecting the API and main site/dashboard at 1:34pm Central. Our team is currently investigating the cause and working on a solution. Please work with your teams to implement downtime procedures. If you have any additional questions, please notify us at support@redoxengine.com.

## Summary On September 8, an outage at a third-party infrastructure provider we rely on for message processing disrupted our platform for approximately 40 minutes, delaying asynchronous processing and returning errors on synchronous requests. Additionally, this slowed down our processing of transaction artifacts resulting in delayed visibility in our dashboard. Our team executed an emergency failover to backup infrastructure and restored normal processing the same afternoon. No data was lost. ## What Happened In the early afternoon of September 8, a third-party infrastructure provider that Redox relies on for message processing experienced an outage. This caused approximately 40 minutes where our platform was not processing traffic. Our team immediately began an emergency failover, redirecting message processing to backup infrastructure to restore service while the third-party issue was ongoing. This failover was successful. Processing was restored, and by 4:05 PM Central our team confirmed the platform had caught up and declared the incident resolved. ## Impact Customers experienced approximately 40 minutes of delayed asynchronous processing and errors on synchronous requests while the third-party outage was active. Additionally, our dashboard and log visibility was delayed for approximately 2 and a half hours. ## How We Resolved It Our team redirected message processing to backup infrastructure within minutes of confirming the outage. Processing was fully restored and the dashboard/log visibility was caught up by 4:05 PM Central the same day. ## What We're Doing About This * Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine * We are actively migrating away from this third-party provider We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team. ‌ _The emergency failover used to resolve this incident led to a second, separate issue affecting dashboard and log visibility. See \[Incident 2: Dashboard & Log Visibility Delay\] for details._