Podio Service Event - Oct 26 - 27 2024

Summary of Impact

On October 26th and 27th, 2024, Podio customers encountered performance issues, including login disruptions and general unavailability. This impacted the ability of users to access and utilize Podio, affecting workflows and client engagements.

Root Cause

On October 26th, we conducted a scheduled maintenance event from 9:30 AM to 11:30 AM IST to upgrade our database infrastructure. Upon transitioning traffic to the upgraded database post-maintenance, we noticed a substantial increase in CPU utilization alongside an elevated number of active sessions. This surge caused slow performance, preventing many users from successfully logging in. The continued high demand ultimately led to temporary service interruptions.

Mitigation

Immediate Actions: Our team, in collaboration with third party support, investigated the potential sources of the performance lag and identified that multiple composite indexes in use were contributing to slow query execution times.

Resolution Timeline: By 1:30 PM IST on October 26th, we optimized several queries to improve response times. We communicated this update on our status page, noting improvements. However, some users experienced issues with category/date filters in Podio apps, which we resolved by 3:00 PM IST on October 27th.

Full Restoration: By 3:30 PM IST on October 27th, all identified issues were resolved, restoring Podio services to normal operation.

Next Steps

• Third Party Collaboration: Continue to work closely with third parties to understand the increased query execution times experienced in the new database environment.
• Rollback Strategy: We already have a general strategy in place, but moving forward, we will implement a more robust rollback plan for database upgrades to further reduce risks and ensure seamless experience. 
• Query Optimisation: We are prioritising ongoing analysis and optimisation of high-demand queries to ensure improved performance and stability.

Our Commitment to Service, Reliability and Performance

Thank you for your patience and understanding as we work to enhance the reliability of our services. Please feel free to reach out to our support team via 24/7 active chat support.