Tuesday March 19, 2024

Issue summary:
On March 19, 2024 from 2:50 PM PDT to 8:16 PM PDT, less than 1% of users were unable to load Slack.

A corrupted table in our database infrastructure prevented one of our databases from responding to requests, causing connection issues for customers whose workspaces were hosted on that database. We manually triggered a backup of the affected database. While we always have backups in place, we were concerned that the existing replicas would be affected by the same bug. We used this manual backup to provision new databases, spinning up several fresh replicas.

Once the replicas were up and running, we directed traffic from the impacted workspaces to these new databases. This restored access for affected customers. We continued working to address the bug that caused the original issue to prevent this from occurring to any other database.

8:40 PM PST

We have taken steps to resolve the issue for all affected users. We appreciate your patience as this was remediated and apologize for any interruptions to your day.

9:01 AM PST

We’re seeing signs of improvement, but we’re still monitoring the situation. We’ll confirm once the issue is fully resolved or we have another update to share.

8:15 AM PST

We’re still actively investigating this issue, but we don’t have any new information to share at this time. Thanks for sticking with us as we continue to work towards a fix.

7:45 AM PST

Our team is continuing to work to remediate this issue. We’re focused on getting Slack working again for affected users as soon as possible. Thank you for your continued patience.

7:14 AM PST

Our engineering team have narrowed down the issue to a specific database and are currently exploring a potential resolution.

The error identified impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.

The team continues to investigate this issue, and we'll report back as more information becomes available.

6:57 AM PST

Our team has identified an issue with a specific database and are currently looking into viable solutions.

The error identified impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.

The team continues to investigate this issue, and we'll report back as more information becomes available.

6:27 AM PST

Our team is continuing to identify the root cause of the issue.

An error was identified which impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.

We're actively looking into this, and we'll report back as more information becomes available.

5:42 AM PST

We've identified an error which impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.

We're actively looking into this, and we'll report back as more information becomes available.

4:26 AM PST

Users have been reporting that they are seeing a Slack server error when trying to load Slack and are having issues signing in.

3:53 AM PST