Wednesday September 11, 2024

Issue Summary:
From September 11th, 2024 at 4:30 PM PDT to September 13th 4:00 PM PDT, some users experienced loading issues with items such as user names, avatars, channel names and their Slack Status.

We found the issue was caused by a Workflow Migrations job which flooded the real-time service with events and created a large number of bot users.

We initiated a backend cache refresh which resolved the issue for all impacted users, and the issue is now completely resolved.

3:05 PM PST

We’ve released a fix and all impacted features are up and running once again. We appreciate your patience while we sorted this out! Please send a note to feedback@slack.com if you continue to have any trouble. We’ll follow up with an issue summary in the next 24 hours.

10:46 AM PST

We are continuing to work on deploying fixes for this issue. We do not have a timeline for when this issue will be completely mitigated, but we are making progress and will post once we know more. Thank you for your continued patience in the meantime.

5:41 PM PST

The fix mentioned in our previous update is continuing to roll out. Thank you for your patience while we work to resolve this issue. We'll be back with another update when we have something to share.

10:04 AM PST

We have deployed a potential fix and are closely monitoring its progress. This fix will take some time to roll out, and we'll be back with an update as soon as we have more news to share.

8:30 PM PST

We do not have additional news to share at this time, but we’re focused on getting things back to normal as quickly as we can.

Thank you for your patience and we apologize for the continued trouble.

10:28 AM PST

Some users may still be experiencing issues with their user status and profile photos not saving.

We are continuing to work on this and will provide updates once we have any information to share.

We apologize for the continued disruption. Thank you for your patience.

12:36 AM PST

Users who were previously experiencing loading issues with items such as user names, avatars, and channel names should now seen all functionality return to normal.

If you’re still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux). Thank you for your patience while we worked to resolve these issues.

8:56 PM PST

We are continuing to see improvements, though we are still waiting for error rates to fully stabilise, and are aiming to have solid confirmation that user impact is fully mitigated.

We'll provide another update in 30 minutes, or as soon as new information is available.

8:15 PM PST

We have implemented a potential fix and are seeing signs of improvement. We're continuing to monitor the situation and will be back with further updates in 30 minutes.

7:45 PM PST

Our team is currently investigating multiple avenues to narrow down the root cause of these issues. We'll provide another update in 30 minutes, or as soon as new information is available.

7:16 PM PST

We are continuing to investigate this issue, but we don’t have any new information to share at this time. We’ll be back with another update in 30 minutes.

6:45 PM PST

We are still working to identify the root cause of these issues. We'll be back with another update in 30 minutes, or as soon as we have more information to share.

6:15 PM PST

We are pursuing several avenues of investigation at the moment, and we'll be back with more information in 30 minutes.

5:42 PM PST

A small number of users may be experiencing loading user names, avatars, and channel names. We're on the case and working to resolve this.

Thank you for bearing with us. We'll be back with another update in 30 minutes.

Note: We've updated the title and the note above to better reflect the impact of this issue.

5:05 PM PST