Background job delays impacting sync and email delivery
Started
Resolved
Duration
1 hr 6 min
Affected:AWS IAM Dynamic SecretsAWS MySQL RotationAWS Parameter StoreAWS Secrets ManagerAWS User Key RotationAzure App ServiceAzure Dev OpsAzure Key VaultCircleCICodefreshDashboard (dashboard.doppler.com)Fly.ioGCP Secret ManagerGitHubHerokuLaravel ForgeNetlifyRailwayRenderSendGrid RotationSplunkTwilio RotationVercelVercel
Update timeline
Resolved
This incident has been resolved.
Monitoring
A fix has been implemented and we're monitoring the situation. All unprocessed jobs have now been processed.
Identified
The issue has been identified and a fix is being implemented.
Investigating
We are continuing to investigate the issue.
Investigating
We're currently investigating an issue involving delays in background job processing that's impacting integration syncs, email delivery, activity log delivery, and webhook delivery (amongst other things). Impacted integration syncs will show as failed with "Out of sync". If expected emails aren't being received, it's likely related to this as well.