PM Work Orders should now be generating as expected. Thank you for your patience as we worked to resolve this issue.

We are continuing to monitor the issue throughout the day.

We are continuing to monitor the issue throughout the day.

We will continue to monitor this issue throughout the night and an update will be provided @ 10 AM CST.

PM Work Orders are now being processed, however, it may take time to generate based on your set schedules. Thank you for your patience, we will continue to monitor this issue throughout the day.

The issue has been identifed and a fix has been implemented.

We are currently investigating an issue with Work Orders. Our Engineering team is currently investigating to determine the cause of the disruption. Next update will be posted at 4 CST

We are currently investigating an issue with Hippo CMMS. We will update you when we have more information.

**Hippo Detailed Root Cause Analysis \(RCA\) – Severity 2 Event July 29th, 2024**  We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Description:** On July 29th, 2024, at 9:35 AM CST we received reports from several customers that PM Work Orders were not getting processed. **Type of Event:** S2 event - Service disruption. The cmms service was returning 500 error codes **Services\\Modules Impacted:** PM and Work Order module **Remediation:** Once the issue was identified, immediate steps were taken to fix the issue. In this case, the cmms service was restarted. **Timeline:** 2024-07-29 9:35 AM – Customer reports PMs are not generating 2024-07-29 10:30 AM – Support responds to incoming client cases 2024-07-29 11:01 AM – Fire Alarm Pulled 2024-07-29 11:04 AM - Jira ticket is created 2024-07-29 11:15 AM – Support manually generated PMs for impacted clients as a temp work around. 2024-07-29 2:26 PM - Cloud Ops restarts the cmms service 2024-07-29 3:02 PM - Cloud Ops reports the service was throwing 500 errors until the restart fixed it. We had to wait for the service to hit the schedules the next day to confirm if the restart resolved the issue.  2024-07-30 9:40 AM – Support follows up with clients to verify issue is resolved 2024-07-30 10:38 AM - Confirmed the majority of clients have started generating PMs 2024-07-30 4:35 PM – All clear called - Severity is lowered to an S3 due to work around for remaining clients. **Total Duration of Event:** 1 day and 7 hours **Root Cause Analysis:** We determined that the cmms service was using shared resources and was not able to handle the load causing it to error out. **Preventative Action:** Our Cloud Ops team provisioned a dedicated instance of cmms service to handle the load which has improved overall performance of the product.