GitHub returning 403 for all Git Sync repositories
Started
Resolved
Duration
1 hr 57 min
Affected:AWS Australia - prod-au-southeast-1AWS Australia - prod-au-southeast-2AWS Brazil - prod-sa-east-1AWS Canada - prod-ca-east-0AWS Canada prod-ca-central-1AWS East (VA) prod-us-east-1AWS Germany - prod-eu-west-2AWS Germany - prod-eu-west-4AWS Germany prod-eu-central-1AWS India - prod-ap-south-1AWS Indonesia - prod-ap-southeast-2AWS Indonesia - prod-ap-southeast-3AWS Ireland - prod-eu-west-6AWS Ireland eu-west-1AWS Japan - prod-ap-northeast-0AWS Singapore - prod-ap-southeast-1AWS Sweden - prod-eu-north-0AWS Sweden eu-north-1AWS Switzerland - prod-eu-central-0AWS Switzerland eu-central-2AWS UAE - prod-me-central-1AWS UK - prod-gb-south-1AWS US East (OH) - prod-us-east-0AWS US East (OH) - prod-us-east-2AWS US East (VA) - prod-us-east-3AWS US West - prod-us-west-0AWS West prod-us-west-2Azure Malasia - azure-malaysiawest-0Azure Netherlands - prod-eu-west-3Azure UAE - prod-me-central-2Azure US Central - us-central2Federal Cloud - AWS US Gov WestGCP Australia - prod-au-southeast-0GCP Belgium - prod-eu-west-0GCP Brazil - prod-sa-east-0GCP India - prod-ap-south-0GCP Saudi Arabia - prod-me-central-0GCP Singapore - prod-ap-southeast-0GCP UK - prod-gb-south-0GCP US Central - prod-us-central-0GCP US Central - prod-us-central-3GCP US Central - prod-us-central-4GCP US Central - prod-us-central-7GCP US East - prod-us-east-1play.grafana.org
Update timeline
Resolved
The fix has been deployed to all production clusters and we are observing a continued period of stability. At this time, we are marking the incident as resolved.
Monitoring
The fix has landed in 60% of production. We are currently monitoring as it lands in the remaining clusters.
Identified
Since ~21:40 UTC on Nov 3, GitHub started rejecting the default User-Agent our Git Sync client sends (nanogit/0), so nearly every GitHub-backed repo shows as unhealthy ("not authorized") and syncs are being skipped. Our Engineering team has rolled out a fix to development and validated that the fix works. We will roll out the fix to production shortly and will provide another update.