Hippo Status
S2 - Hippo CMMS Experiencing unusual slowness and intermittent availability

Started

Resolved

Duration

less than a minute

PostmortemVendor link

Update timeline

Resolved

The issue with Hippo CMMS has been identified and a fix is being implemented.

Postmortem—

**Hippo Detailed Root Cause Analysis \(RCA\) – Severity 2 Event May 11th, 2024** ‌ We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. ‌ **Description:** On May 11, 2024, 1:43 PM we received reports from several customers that the Hippo front-end web application has been unresponsive to the point of being unusable. Primary impact is to Canadian customers but additional regions starting to report issues. ‌ **Type of Event:** S2 event - Service disruption. slowness. ‌ **Services\\Modules Impacted:** Hippo CMMS application ‌ **Remediation:**` `Once the issue was identified, immediate steps were taken to fix the issue. In this case, the service was restarted. ` `Insufficient resources have been allocated to the production instance of Hippo, causing severely degraded performance. Some delays in identifying this issue were caused by insufficient knowledge of the platform, causing this to be misidentified as not being resource-related. ` `**Timeline:**` `May 11, 2024, 1:43 PM S1 Ticket logged May 11, 2024, 1:55 PM Investigation started May 11, 2024, 2:10 PM agent Reduced priority as per the other cases in queue May 11, 2024, 2:11 PM DC-0393 created May 11, 2024, 2:13 PM agent included Hippo T2 in conversation - linked existing JIRA CM-80947 May 11, 2024, 6:56 PM agent increased priority to S2 May 11, 2024, 6:57 PM Service restarted May 11, 2024, 7:01pm contacted customer ‌ **Total Duration of Event:** 5 hours 28 minutes ‌ **Root Cause Analysis:** Insufficient System Resources ‌ **Preventative Action:** Our Cloud Ops team took the preventive measure that they have been testing for the past few days to automatically scale the app service to have more instances when the service experiences high CPU.

S2 - Hippo CMMS Experiencing unusual slowness and intermittent availability — Hippo Incident Timeline & Status — DevHelm