Podio search, re-calculations, workflows, real time item updates, chat and notifications event – February 26, 2025 

Summary of Impact

On February 26th 2025, Podio customers experienced issues in searching items, calculations on fields, real time item updates and webhook events causing delays and disruptions in workflow execution. 

Root Cause 

On February 26, 2025, at 07:37 EDT, our alert system noticed a sudden increase in activity in one of our queues that handles item searches, updates, and other tasks. The messages in this queue were not being processed, which added to the existing workload. The problem was traced back to a failure in our messaging system, likely caused by a recent software update, which was completed on February 19, 2025. This issue made the queue stop working, leading to delays in processing messages and affecting various services like item searches, calculations, and real-time updates. We worked closely with our third-party database provider to troubleshoot and resolve the issue. 

Mitigation 

After identifying the root cause, the Engineering team took the following steps: 

• Stopped all incoming traffic to the queue and reverted the AMQ upgrade. 

All issues were confirmed resolved and fixed as of Feb 26, 2025 23:58 EDT. 

Next Steps 

1. Conduct load testing of the AMQ upgrade to version 3.13 in our lower environments to identify and fix any bottlenecks before attempting the next upgrade in our production system. 
2. Further enhance alerts and monitoring to notify us when our event queue reaches a certain threshold, allowing us to proactively address the issue.