Tuesday July 2, 2024

Issue summary:
From 7:13 PM PDT on July 2, 2024 to 2:38 PM PDT on July 3, 2024, some users experienced issues loading threads in Slack.

We traced the issue to one of our databases. Repeated API calls related to threads from one org caused the database responsible for storing that thread to become overwhelmed. This overload affected other users accessing the same database, causing congestion, which prevented them from using it effectively and created further unexpected timeouts.

As an immediate mitigation step, we halted all traffic from users using that API call in order to retrieve the thread from the database which allowed our systems to catch-up. Once traffic started flowing at a reasonable rate again, we reinstated the API call for all users and disallowed the thread that initially caused an issue. This resolved the issue for all impacted users.

We've now added additional code to the problematic thread to bypass this particular API so that any unwanted traffic congestion can be mediated.

8:41 PM PST

The issue has been resolved and threads should now load as expected. Thank you for bearing with us, we appreciate your patience.

9:54 PM PST

We are still working to resolve the issue with threads. No additional information to share just yet, but we'll be back with another update in 30 minutes.

9:25 PM PST

Some users may not be able to load threads at the moment. We apologize for the disruption. We're on the case, and we'll be back with more information in 30 minutes or as soon as we have news to share.

9:01 PM PST