Tuesday March 19, 2024
Incident
Issues with file previews
Issue Summary:
From Monday March 18, 2024 at 9:07 PM PDT to Tuesday March 19, 2024 until 12:06 AM PDT, some users saw blurry image previews.
A change was made to a backend service that's responsible for the handling of files in workspaces that have International Data Residency (IDR) enabled. This change inadvertently affected the ability for Slack to correctly generate previews for image files. On Tuesday March 19 around 12:06 AM PDT, we reverted the change to stop any new files to be loaded with blurry image previews.
Thereafter, we conducted an in-depth investigation to find any images that were uploaded during the impacted duration to correct its image clearness. On Wednesday March 20, 12:50 PM PDT, all images were restored.
Thank you for bearing with us while we worked through this case.
7:38 PM PST
We have implemented a fix and all impacted images should be returned to normal. We appreciate your patience while we sorted this out. If you continue to experience issues, please reach out to feedback@slack.com
9:56 AM PST
We’re still actively investigating this issue, but we don’t have any new information to share at this time.
Thanks for sticking with us as we continue to work towards a fix.
10:46 AM PST
We’re still investigating issues with file previews. No additional information to share just yet, but we’ll keep you posted as soon we have an update.
10:17 AM PST
Customers on workspaces enrolled in an International Data Residency region may be experiencing blurred file previews on desktop. We have resolved the underlying issue, and we're now working to restore the affected file previews. We'll provide an update once we have more details to share.
12:06 AM PST
Features affected
Files
Status
Incident
Incident
Issues loading Slack for less than 1% of customers
Issue summary:
On March 19, 2024 from 2:50 PM PDT to 8:16 PM PDT, less than 1% of users were unable to load Slack.
A corrupted table in our database infrastructure prevented one of our databases from responding to requests, causing connection issues for customers whose workspaces were hosted on that database. We manually triggered a backup of the affected database. While we always have backups in place, we were concerned that the existing replicas would be affected by the same bug. We used this manual backup to provision new databases, spinning up several fresh replicas.
Once the replicas were up and running, we directed traffic from the impacted workspaces to these new databases. This restored access for affected customers. We continued working to address the bug that caused the original issue to prevent this from occurring to any other database.
8:40 PM PST
We have taken steps to resolve the issue for all affected users. We appreciate your patience as this was remediated and apologize for any interruptions to your day.
9:01 AM PST
We’re seeing signs of improvement, but we’re still monitoring the situation. We’ll confirm once the issue is fully resolved or we have another update to share.
8:15 AM PST
We’re still actively investigating this issue, but we don’t have any new information to share at this time. Thanks for sticking with us as we continue to work towards a fix.
7:45 AM PST
Our team is continuing to work to remediate this issue. We’re focused on getting Slack working again for affected users as soon as possible. Thank you for your continued patience.
7:14 AM PST
Our engineering team have narrowed down the issue to a specific database and are currently exploring a potential resolution.
The error identified impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.
The team continues to investigate this issue, and we'll report back as more information becomes available.
6:57 AM PST
Our team has identified an issue with a specific database and are currently looking into viable solutions.
The error identified impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.
The team continues to investigate this issue, and we'll report back as more information becomes available.
6:27 AM PST
Our team is continuing to identify the root cause of the issue.
An error was identified which impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.
We're actively looking into this, and we'll report back as more information becomes available.
5:42 AM PST
We've identified an error which impacts loading Slack on the app, web browsers and logging into Slack, for less than 1% of Slack customers.
We're actively looking into this, and we'll report back as more information becomes available.
4:26 AM PST
Users have been reporting that they are seeing a Slack server error when trying to load Slack and are having issues signing in.
3:53 AM PST
Features affected
Login/SSO
Connectivity
Messaging
Status
Incident