37signals Status
Basecamp 5 and HEY are down

Started

Resolved

Duration

24 minutes

Postmortemcritical impactVendor link
Affected:Basecamp 5HEY

Update timeline

Resolved

Basecamp 5 and HEY are back up. We'll continue to monitor it to make sure it's stable. So sorry for the interruption today, folks.

Investigating

Basecamp 5 and HEY are back up. We'll continue to monitor it to make sure it's stable. So sorry for the interruption today, folks.

Investigating

Basecamp 5 and HEY are coming back online. We're actively monitoring both.

Investigating

We've discovered the root cause and are actively working on a fix. No ETA yet, but we're on the case. Thanks for bearing with us, folks.

Investigating

We're still investigating the trouble with Basecamp and HEY. We'll keep you posted here. We're so sorry, folks.

Investigating

Basecamp and HEY are having trouble loading. We're looking into the root cause and will keep you posted here. We're so sorry for the interruption, everyone.

Postmortem—

This morning, power maintenance in our Ashburn datacenter caused our load balancers there to briefly stop serving customer traffic. That meant Basecamp and HEY were down for some customers in the U.S. and parts of Europe. Our three other datacenters remained online during the incident. After fixing the root cause, we noticed some lingering network instabilities. This led to increased errors with using Basecamp and HEY. Our Ops team quickly rolled out a fix for these as well. This was an avoidable problem. These devices should be resilient to this type of maintenance but were misconfigured. We’ve corrected the issue that caused today's incident to prevent this problem from recurring. We’re very sorry for interrupting your work today. If you have any questions, or if we can help in any way, please [reach out to our support crew](mailto:support@basecamp.com).

Basecamp 5 and HEY are down — 37signals Incident Timeline & Status — DevHelm