Podio is temporarily not accessible
Incident Report for Podio Status Page
Postmortem

Citrix Podio Incident – September 15/16th, 2022

On 15th and 16th September 2022, Citrix Podio experienced two significant service outages. We apologize
for any interruption this may have caused for our customers and their business workflows. Due to the
proximity of these incidents, and common underlying causes, we wanted to share the details with you to
ensure that you know what happened and what we are doing to prevent them from reoccurring.

Impact Summary:

  • The first incident started on 15th September at 06:33 A.M. EDT (10:33 A.M. UTC) when internal alerts notified the Podio team that performance was degraded. Multiple mitigation actions to address issues on infrastructure were deployed, without being a full fix for the underlying issue.
  • We were able to get Podio up and running again by 10:11 A.M. EDT (15:11 P.M. UTC).
  • The Podio team worked through Thursday 15th September to understand the cause and stabilize the Podio infrastructure to avoid the issue from reoccurring.
  • We scheduled an urgent maintenance window for Thursday afternoon (15:00 P.M. - 16:29 P.M. EDT) in an attempt to improve the infrastructure, which was considered successful.
  • On Friday 16th September at 05:32 A.M. EDT (09:32 A.M. UTC), the second outage occurred in the Podio platform, due to a similar issue.
  • Podio was up and running again at 06:21 A.M. EDT (10:21 A.M. UTC) on Friday 16th September.

What Happened:

Citrix Podio employs a third-party message-brokering component. On September 15th, internal monitoring
detected delays in processing tasks in the queue of this component. Citrix Engineering determined the
message-broker system was in an unhealthy state and the node was unable to process requests
impacting all systems. We saw a reoccurrence of this issue on September 16th.

Mitigation Efforts:

On September 15th, initially the memory was increased which showed some temporary recovery of
queue processing. Unfortunately, this did not mitigate the issue. Citrix Engineering subsequently
performed restarts of backend resources, along with purging of queued operations, which allowed the
system to recover.

On September 16th, similar actions were taken to restore the health of our third-party message-brokering
system. Mitigation and service recovery was more efficient on September 16th and therefore the time to
recover was lower.

Corrective Action and Next Steps:

We take such outages very seriously at Podio. Please be assured that Podio's performance, stability and
reliability remain our #1 priority. Citrix Engineering teams are investigating the cause of the brokering
system failures and will be putting additional safeguards in place to assure adequate capacity and
resiliency in the future.

We apologize for the impact these incidents have had on you and your teams. Since our customers use
Podio daily, we know how impactful these outages can be. As a result, we have been working on fixing
these issues with high priority and urgency. In addition, we are committed to learning from these
incidents, so we can improve our system's stability, issue detection, and mitigation capabilities with an
aim to avoid such outages in the future.

Posted Sep 16, 2022 - 15:25 EDT

Resolved
At this time, we believe that the issues have been fully resolved.

We will continue to monitor closely. If any further issues are seen, please contact Citrix Technical Support.
Posted Sep 15, 2022 - 11:10 EDT
Update
Podio is now up and running with some degraded features. Customers may still be experiencing issues with real-time updates for viewing. In addition, updating items is currently unavailable.

We continue to work towards mitigating the issue completely.

More information will be provided as it becomes available.
Posted Sep 15, 2022 - 10:50 EDT
Update
The mitigation actions continue to alleviate some of the issues seen by users. The Citrix engineering team has seen improvement in some customers.


We continue to work towards mitigating the issue completely.


More information will be provided as it becomes available.
Posted Sep 15, 2022 - 10:43 EDT
Update
At this point, mitigation actions seem to be alleviating some of the issues seen by users.

The Citrix engineering team has seen improvement in some customers. We are continuing to work towards mitigating the issue completely.

More information will be provided as it becomes available.
Posted Sep 15, 2022 - 10:16 EDT
Update
The team has tried some potential fixes to resolve the issue but nothing has worked so far. We continue to work on a resolution.
At present we are unable to give an ETA for delivery of the resolution. We will provide further update will be provided once we have fix in place or sooner if needed.

We will conduct a full Root Cause Analysis to avoid this issue from occurring in the future after this incident is resolved. Right now, the team is fully dedicated to fixing Podio.
Posted Sep 15, 2022 - 10:02 EDT
Update
We are continuing to work on a fix for this issue.
Posted Sep 15, 2022 - 09:23 EDT
Update
The team is actively working on a resolution. Customers may still not be able to access Podio.
We will keep Podio users updated on this page.
Posted Sep 15, 2022 - 08:42 EDT
Update
The Citrix Engineering team is engaged and working on a fix. We regret the inconvenience caused and are trying our best to get Podio up and running again
Posted Sep 15, 2022 - 08:22 EDT
Identified
The issue has been identified and a fix is being implemented.
Posted Sep 15, 2022 - 08:15 EDT
Update
We are continuing to investigate this issue.
Posted Sep 15, 2022 - 07:45 EDT
Investigating
The delays could not be mitigated with the fix, we are continuing to investigate further.
Posted Sep 15, 2022 - 07:16 EDT
Update
The delays could not be mitigated with the fix, we are continuing to investigate further.
Posted Sep 15, 2022 - 07:16 EDT
Monitoring
A fix has been deployed. Delays should now be resolved.
In case your calculations, imports, exports or workflows failed, you might need to re-trigger them.
We will continue to monitor the situation for sometime.
Posted Sep 15, 2022 - 07:03 EDT
Update
A fix has been deployed. Delays should now be resolved.
In case your calculations, imports, exports or workflows failed, you might need to re-trigger them.
We will continue to monitor the situation for sometime.
Posted Sep 15, 2022 - 07:03 EDT
Identified
The issue has been identified and the Citrix Engineering team is working on a fix
Posted Sep 15, 2022 - 06:50 EDT
Update
We are continuing to investigate this issue.
Posted Sep 15, 2022 - 06:35 EDT
Update
We are continuing to investigate this issue.
Posted Sep 15, 2022 - 06:34 EDT
Investigating
We are currently investigating the issue.
Posted Sep 15, 2022 - 06:34 EDT
This incident affected: Web, API, Email, and Advanced Workflow Automation.