Podio is temporarily inaccessible
Incident Report for Citrix Podio Status Page
Postmortem

Citrix Podio Incident – September 15/16th, 2022

On 15th and 16th September 2022, Citrix Podio experienced two significant service outages. We apologize
for any interruption this may have caused for our customers and their business workflows. Due to the
proximity of these incidents, and common underlying causes, we wanted to share the details with you to
ensure that you know what happened and what we are doing to prevent them from reoccurring.

Impact Summary:

  • The first incident started on 15th September at 06:33 A.M. EDT (10:33 A.M. UTC) when internal alerts notified the Podio team that performance was degraded. Multiple mitigation actions to address issues on infrastructure were deployed, without being a full fix for the underlying issue.
  • We were able to get Podio up and running again by 10:11 A.M. EDT (15:11 P.M. UTC).
  • The Podio team worked through Thursday 15th September to understand the cause and stabilize the Podio infrastructure to avoid the issue from reoccurring.
  • We scheduled an urgent maintenance window for Thursday afternoon (15:00 P.M. - 16:29 P.M. EDT) in an attempt to improve the infrastructure, which was considered successful.
  • On Friday 16th September at 05:32 A.M. EDT (09:32 A.M. UTC), the second outage occurred in the Podio platform, due to a similar issue.
  • Podio was up and running again at 06:21 A.M. EDT (10:21 A.M. UTC) on Friday 16th September.

What Happened:

Citrix Podio employs a third-party message-brokering component. On September 15th, internal monitoring
detected delays in processing tasks in the queue of this component. Citrix Engineering determined the
message-broker system was in an unhealthy state and the node was unable to process requests
impacting all systems. We saw a reoccurrence of this issue on September 16th.

Mitigation Efforts:

On September 15th, initially the memory was increased which showed some temporary recovery of
queue processing. Unfortunately, this did not mitigate the issue. Citrix Engineering subsequently
performed restarts of backend resources, along with purging of queued operations, which allowed the
system to recover.

On September 16th, similar actions were taken to restore the health of our third-party message-brokering
system. Mitigation and service recovery was more efficient on September 16th and therefore the time to
recover was lower.

Corrective Action and Next Steps:

We take such outages very seriously at Podio. Please be assured that Podio's performance, stability and
reliability remain our #1 priority. Citrix Engineering teams are investigating the cause of the brokering
system failures and will be putting additional safeguards in place to assure adequate capacity and
resiliency in the future.

We apologize for the impact these incidents have had on you and your teams. Since our customers use
Podio daily, we know how impactful these outages can be. As a result, we have been working on fixing
these issues with high priority and urgency. In addition, we are committed to learning from these
incidents, so we can improve our system's stability, issue detection, and mitigation capabilities with an
aim to avoid such outages in the future.

Posted Sep 16, 2022 - 15:25 EDT

Resolved
While Podio is up and running again and we continue to monitor the situation, the Podio team deeply regrets the inconvenience caused to our users and their businesses due to the downtime seen in the past two days.

Please know our teams are working hard as you read this to ensure such incidents do not occur again in the near future. We take such outages very seriously at Podio and will share a Postmortem report containing the timeline overview of the issue and the plan of action to remediate such incidents in the future by EOD today.

Again, please accept our deepest apologies for the inconvenience caused today and yesterday.
Posted Sep 16, 2022 - 08:33 EDT
Update
While Podio is up and running again and we continue to monitor the situation, the Podio team deeply regrets the inconvenience caused to our users and their businesses due to the downtime seen in the past two days.

Please know our teams are working hard as you read this to ensure such incidents do not occur again in the near future. We take such outages very seriously at Podio and will share a Postmortem report containing the timeline overview of the issue and the plan of action to remediate such incidents in the future by EOD today.

Again, please accept our deepest apologies for the inconvenience caused today and yesterday.
Posted Sep 16, 2022 - 07:13 EDT
Monitoring
Podio is up and running and we are monitoring to ensure all the services are available to end-users.
Posted Sep 16, 2022 - 06:35 EDT
Identified
The issue has been identified and the Citrix Engineering team is working on a fix
Customers may still not be able to access Podio. We will keep Podio users updated on this page.
Posted Sep 16, 2022 - 06:29 EDT
Update
We are continuing to investigate this issue.
The Citrix Engineering team is engaged and working on a fix. We regret the inconvenience caused and are trying our best to get Podio up and running again
Posted Sep 16, 2022 - 06:16 EDT
Update
We are continuing to investigate this issue.
Posted Sep 16, 2022 - 05:36 EDT
Investigating
We are currently investigating this issue.
Posted Sep 16, 2022 - 05:32 EDT
This incident affected: Web, API, Email, Advanced Workflow Automation, and Advanced Workflow Automation Failover Queue.