Tuesday March 19, 2024

Issue Summary:
From Monday March 18, 2024 at 9:07 PM PDT to Tuesday March 19, 2024 until 12:06 AM PDT, some users saw blurry image previews.

A change was made to a backend service that's responsible for the handling of files in workspaces that have International Data Residency (IDR) enabled. This change inadvertently affected the ability for Slack to correctly generate previews for image files. On Tuesday March 19 around 12:06 AM PDT, we reverted the change to stop any new files to be loaded with blurry image previews.

Thereafter, we conducted an in-depth investigation to find any images that were uploaded during the impacted duration to correct its image clearness. On Wednesday March 20, 12:50 PM PDT, all images were restored.

Thank you for bearing with us while we worked through this case.

7:38 PM PST

We have implemented a fix and all impacted images should be returned to normal. We appreciate your patience while we sorted this out. If you continue to experience issues, please reach out to feedback@slack.com

9:56 AM PST

We’re still actively investigating this issue, but we don’t have any new information to share at this time.

Thanks for sticking with us as we continue to work towards a fix.

10:46 AM PST

We’re still investigating issues with file previews. No additional information to share just yet, but we’ll keep you posted as soon we have an update.

10:17 AM PST

Customers on workspaces enrolled in an International Data Residency region may be experiencing blurred file previews on desktop. We have resolved the underlying issue, and we're now working to restore the affected file previews. We'll provide an update once we have more details to share.

12:06 AM PST

Issue summary:
On March 19, 2024 from 2:50 PM PDT to 8:16 PM PDT, less than 1% of users were unable to load Slack.

A corrupted table in our database infrastructure prevented one of our databases from responding to requests, causing connection issues for customers whose workspaces were hosted on that database. We manually triggered a backup of the affected database. While we always have backups in place, we were concerned that the existing replicas would be affected by the same bug. We used this manual backup to provision new databases, spinning up several fresh replicas.

Once the replicas were up and running, we directed traffic from the impacted workspaces to these new databases. This restored access for affected customers. We continued working to address the bug that caused the original issue to prevent this from occurring to any other database.

8:40 PM PST

We have taken steps to resolve the issue for all affected users. We appreciate your patience as this was remediated and apologize for any interruptions to your day.

9:01 AM PST

We’re seeing signs of improvement, but we’re still monitoring the situation. We’ll confirm once the issue is fully resolved or we have another update to share.

8:15 AM PST

We’re still actively investigating this issue, but we don’t have any new information to share at this time. Thanks for sticking with us as we continue to work towards a fix.

7:45 AM PST

Our team is continuing to work to remediate this issue. We’re focused on getting Slack working again for affected users as soon as possible. Thank you for your continued patience.

7:14 AM PST

Our engineering team have narrowed down the issue to a specific database and are currently exploring a potential resolution.

The error identified impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.

The team continues to investigate this issue, and we'll report back as more information becomes available.

6:57 AM PST

Our team has identified an issue with a specific database and are currently looking into viable solutions.

The error identified impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.

The team continues to investigate this issue, and we'll report back as more information becomes available.

6:27 AM PST

Our team is continuing to identify the root cause of the issue.

An error was identified which impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.

We're actively looking into this, and we'll report back as more information becomes available.

5:42 AM PST

We've identified an error which impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.

We're actively looking into this, and we'll report back as more information becomes available.

4:26 AM PST

Users have been reporting that they are seeing a Slack server error when trying to load Slack and are having issues signing in.

3:53 AM PST