Tuesday July 2, 2024

Issue summary:
On July 2, 2024 from 5:09 PM PDT to 7:00 PM PDT, users may have experienced issues uploading files, sending messages with attached files, downloading files, and viewing files in conversations.

A code change inadvertently introduced a logic error in our file handling processes. We reverted this code change, resolving the issue for all affected users.

10:24 PM PST

The fix has been deployed and error rates have subsided. All files actions should be back to normal.

We apologize any disruption to your work.

7:15 PM PST

We've deployed a fix that is rolling out to everyone gradually. We are monitoring the outcome, and we'll be back once we're able to confirm that the issue has been resolved.

6:58 PM PST

Some users may be experiencing issues with file uploads and with existing files in conversations. We're investigating and we'll be back with an update as soon as we have more information.

Thank you for bearing with us.

6:04 PM PST

Issue summary:
From 7:13 PM PDT on July 2, 2024 to 2:38 PM PDT on July 3, 2024, some users experienced issues loading threads in Slack.

We traced the issue to one of our databases. Repeated API calls related to threads from one org caused the database responsible for storing that thread to become overwhelmed. This overload affected other users accessing the same database, causing congestion, which prevented them from using it effectively and created further unexpected timeouts.

As an immediate mitigation step, we halted all traffic from users using that API call in order to retrieve the thread from the database which allowed our systems to catch-up. Once traffic started flowing at a reasonable rate again, we reinstated the API call for all users and disallowed the thread that initially caused an issue. This resolved the issue for all impacted users.

We've now added additional code to the problematic thread to bypass this particular API so that any unwanted traffic congestion can be mediated.

8:41 PM PST

The issue has been resolved and threads should now load as expected. Thank you for bearing with us, we appreciate your patience.

9:54 PM PST

We are still working to resolve the issue with threads. No additional information to share just yet, but we'll be back with another update in 30 minutes.

9:25 PM PST

Some users may not be able to load threads at the moment. We apologize for the disruption. We're on the case, and we'll be back with more information in 30 minutes or as soon as we have news to share.

9:01 PM PST