Wednesday March 20, 2024

Issue summary:
On March 20, 2024 from around 8:05 AM PDT until 10:33 PM PDT, some users experienced various issues with Slack such as loading channel membership lists, joining a channel, setting or clearing their user statuses, loading profile images, incorrect huddle status, and inconsistent user states if a user was recently deactivated or reactivated.

A recent change to resolve an unrelated traffic issue we observed with our backend servers inadvertently caused connections to database caching services to timeout.

At around 12:00 PM PDT, we had identified the cause of the timeout failures and began gradually deploying a fix. We continued to closely monitor our health metrics for any abnormalities as our remediation progressed.

By 10:33 PM PDT, the deploy successfully reached all affected users, these users would have immediately received the fix through a hard refresh of their Slack client (Cmd/Ctrl + Shift + R). By 8:49 AM PDT on March 21, 2024, most affected users received the fix without a required hard refresh.

We are continuing to investigate this issue and implement preventative measures to reduce the likelihood of this occurring again in the future.

5:06 PM PST

The metrics have recovered back to normal levels. The fix deploy is still rolling out to 100% of users clients however, in the meantime users are able to hard reload Slack which will achieve the same result.

To hard reload Slack, please use 'Cmd'/'Ctrl' + Shift + 'R'

Thank you for your patience during this investigation.

2:39 AM PST

The fix rollout has made significant progress and we're seeing our metrics are recovering back to normal levels. We're continuing to monitor progress of the fix rollout and will update once the fix is fully implemented.
Thank you for your continued patience.

1:42 AM PST

A fix is rolling out and is gradually reaching all affected users. Our metrics are showing healthy recovery rates.

We're continuing to monitor progress of the fix rollout and will continue to provide updates here when they are available to us.

We apologize for any disruption these issues may have caused.

5:43 PM PST

We’ve determined that this issue may also be preventing some users from viewing profile photos and message activity. Fortunately we’re beginning to see signs of improvement, but are still working towards a full resolution. We’re keeping a close eye and will continue to provide updates as they become available.

1:54 PM PST

We have identified the cause of the issue and are working on getting things back to normal as soon as possible. We're continuing to monitor the situation closely and will be back with an update as soon as we have any new information.

1:15 PM PST

We've identified further inconsistencies with user experiences when interacting with huddles. Users may inconsistently appear to be joining a huddle from another device. The huddle sidebar indicator may also appear inconsistent for affected users. Our investigation continues and we'll be back with another update soon.

12:48 PM PST

Some users are reporting issues loading & joining channels, viewing channel members, and clearing their user status. We're investigating these issues and will share an update once we have more information.

12:17 PM PST