Latency in calculations and triggered events
Incident Report for Podio

On 9/19/17, reports of elevated wait times in Globiflow-based processes were reported to Podio Ops at 9:55am EDT, and monitoring tools recorded API errors. As task load increased unexpectedly on the affected instance, subsequent processes became queued. After a manual throttle of the affected tasks was unsuccessful, the API server was restarted at 10:59am, leading to an almost immediate reduction in queue time and overall performance, effectively resolving this incident.

Further investigation revealed that the impacted instance had been scheduled for maintenance earlier today, however a discrepancy in the backing instance configuration resulted in the instance remaining deployed, and contributed to the inability of the system to gracefully failover as intended.

Our Operations team has made a number of recommendations for long-term remediation, particularly as it applies to the backing services configuration, and the failover mechanics for instances that are no longer deployed.

Posted about 1 month ago. Sep 19, 2017 - 15:55 EDT

Resolved
All services operational.
Posted about 1 month ago. Sep 19, 2017 - 13:11 EDT
Update
An unexpected increase in workload activity created an operation backlog; we have taken steps to block some of that activity, which has mitigated the latency issue.

Separately, we are investigating occurrences of the front end web application being intermittently unavailable, and will continue to post updates here as that work progresses.

Again, thank you for your patience.
Posted about 1 month ago. Sep 19, 2017 - 11:42 EDT
Investigating
Our operations and development teams are aware of latency issues in both Podio and Globiflow, resulting in delays and/or timeouts when events/flows are triggered, as well as in populating calculated fields.

Our team has identified the potential root cause, and is working to make changes on the backend to remedy the latency.

We apologize for any inconvenience this may have caused, and will update this page as information is available.
Posted about 1 month ago. Sep 19, 2017 - 10:41 EDT