On 15th and 16th September 2022, Citrix Podio experienced two significant service outages. We apologize
for any interruption this may have caused for our customers and their business workflows. Due to the
proximity of these incidents, and common underlying causes, we wanted to share the details with you to
ensure that you know what happened and what we are doing to prevent them from reoccurring.
Citrix Podio employs a third-party message-brokering component. On September 15th, internal monitoring
detected delays in processing tasks in the queue of this component. Citrix Engineering determined the
message-broker system was in an unhealthy state and the node was unable to process requests
impacting all systems. We saw a reoccurrence of this issue on September 16th.
On September 15th, initially the memory was increased which showed some temporary recovery of
queue processing. Unfortunately, this did not mitigate the issue. Citrix Engineering subsequently
performed restarts of backend resources, along with purging of queued operations, which allowed the
system to recover.
On September 16th, similar actions were taken to restore the health of our third-party message-brokering
system. Mitigation and service recovery was more efficient on September 16th and therefore the time to
recover was lower.
We take such outages very seriously at Podio. Please be assured that Podio's performance, stability and
reliability remain our #1 priority. Citrix Engineering teams are investigating the cause of the brokering
system failures and will be putting additional safeguards in place to assure adequate capacity and
resiliency in the future.
We apologize for the impact these incidents have had on you and your teams. Since our customers use
Podio daily, we know how impactful these outages can be. As a result, we have been working on fixing
these issues with high priority and urgency. In addition, we are committed to learning from these
incidents, so we can improve our system's stability, issue detection, and mitigation capabilities with an
aim to avoid such outages in the future.