Wednesday February 26, 2025

From 6:45 AM PST to 4:13 PM PST on February 26, 2025, a large percentage of Slack users experienced issues with various features including sending and receiving messages, using workflows, loading channels or threads, and logging into Slack. These features may have been degraded or in some cases fully unusable.  

We traced this issue to a maintenance action in one of our database systems, which, combined with a latency defect in our caching system, caused an overload of heavy traffic to the database. As a result, approximately 50% of instances relying on this database became unavailable. 

We took several actions to reduce the heavy load on the database system and implemented a fix for the source of the load. At around 9:32 AM PST, we observed improved health metrics for some affected Slack features and at around 4:13 PM PST, our health metrics returned to normal rates, indicating the issue was resolved for all affected users.

Edit: After the main impact of this issue was resolved, users experienced issues with our Events API - specifically related to custom applications, integrations and bots not working. This issue is now resolved and these are functioning as expected. You can review specific updates on this resolution here: https://slack-status.com/2025-02/d41e4bfd1ccae26a.

12:31 PM PST

We've restored full functionality to all affected Slack features such as sending messages, workflows, threads and other API-related features. 

If you're still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux).

Thank you for your patience while we worked to resolve this issue.

4:42 PM PST

We're still working to restore functionality to affected Slack features such as sending messages, workflows, threads and other API-related features.  We'll be back with more updates as soon as they're available.

4:26 PM PST

No additional news to share just yet from our last update, but we’re focused on getting things back to normal as quickly as we can. We apologize for the continued trouble.

3:46 PM PST

We're still working to restore functionality to affected Slack features such as sending messages, workflows, threads and other API-related features.  We'll be back with more updates as soon as they're available.

3:18 PM PST

We're continuing to diligently work on database shard repair and have made progress on restoring affected replicas, though users may still be experiencing impact.

Slash commands will be temporarily disrupted for a few minutes as we work on restoring service. Users who boot during this time may experience issues.

Thank you for your patience.

2:47 PM PST

No additional news to share just yet from our last update, but we’re focused on getting things back to normal as quickly as we can. We apologize for the continued trouble.

2:07 PM PST

We're continuing to diligently work on database shard repair and have made progress on restoring affected replicas, though users may still be experiencing impact. 

We are continuing to operate urgently to restore full functionality and we appreciate your continued patience.

1:35 PM PST

Users may still be experiencing issues loading Slack, using workflows, sending messages or loading threads. API-related actions may also be degraded in speed. While some users may notice improvements, we remain focussed on resolving the issues and are working on this with the highest urgency.

Remediation work involves repairing affected database shards, which are causing feature degradation issues. This has become a diligent process to ensure we're prioritizing the database replicas with the most impact. 

We are working to restore functionality to all of the database shards, and will continue to share updates here as we have more.

1:04 PM PST

We identified the cause of the issue and are continuing to monitor our health metrics, but we are still not fully resolved. Some users may still reproduce symptoms such as issues loading their workspaces.

Our work is still ongoing to restore affected backend services to returning functionality to impacted features.

12:34 PM PST

We're still working to restore functionality to affected Slack features such as sending messages, workflows, threads and other API-related features.  We'll be back with more updates as soon as they're available.

12:05 PM PST

No additional news to share just yet, but we’re focused on getting things back to normal as quickly as we can. We apologize for the continued trouble.

11:35 AM PST

We’re still looking into the cause of the issue and working on restoring functionality to affected features such as workflows, sending messages, threads and API-related features. Users may also continue to experience issues when attempting to log-in. We’ll provide new information as soon as it’s available.

11:05 AM PST

We’re still actively investigating this issue, but we don’t have any new information to share at this time. Thanks for sticking with us as we continue to work towards a fix. We’ll keep you posted as soon we have an update.

10:34 AM PST

We're still working to restore functionality to affected Slack features such as sending messages, workflows, threads and other API-related features.  We'll be back with more updates as soon as they're available.

10:03 AM PST

Our investigation is still in progress with regard to degraded functionality for Slack features such as workflows, threads, sending messages and API-related features. We'll be back with more updates as soon as they're available.

9:26 AM PST

We're continuing our efforts to restore functionality to affected features such as workflows, sending messages, threads and API-related features. Users may also experience issues when attempting to log-in. We appreciate your continued patience. More updates will be shared as soon as possible.

8:56 AM PST

We're still working to restore functionality to affected Slack features such as sending messages, workflows, threads and other API-related features. Users may also experience issues when attempting to log-in. Thank you for your continued patience as we continue investigating. We'll be following up with further updates as they become available.

8:30 AM PST

We're still working to restore functionality to affected Slack features but have also discovered some workflows may also be experiencing issues. We appreciate your patience as we sort this out and will be back with another update soon.

8:08 AM PST

Our investigation is still ongoing but have determined a variety of API endpoints, sending/receiving messages, and some threads loading are also impacted. We'll be back with another update as soon as possible.

7:48 AM PST

We're investigating reports of trouble connecting or loading Slack, we'll provide another update once we have news to share.

7:27 AM PST

We continue to see custom applications, integrations and workflows working as expected. We consider the Events API issue resolved, and would like to share some additional detail:

Due to mitigation efforts, our Events API built up a considerable backlog of events that were not delivered as expected to apps. Once the database tier became available, we re-enabled these queues and by 11:27pm PST on 2025-02-26, all events were being properly delivered as expected. 

We chose to not replay the large number of events for delivery that were buffered from the incident because app event delivery semantics often depend on timely delivery for correctness. In addition we wanted to avoid potentially overloading either our systems or those of third-party systems with the surge of queued events which had been paused throughout the outage.

We thank you for your patience as we worked through resolution efforts, and we apologize for the disruption caused.

7:28 PM PST

Our work related to events is still in motion. We'll provide an update as new information becomes available. We appreciate your patience along the way.

10:12 AM PST

The work we're doing related to the events that queued during the downtime yesterday is still ongoing. We'll provide an update on this as we continue making progress, but it is still expected to take some time.

The issue users were seeing regarding Slack Connect @ mention trouble has now been resolved. Users may need to refresh Slack (Cmd/Ctrl + Shift + R) for the fix to appear. Thank you for your continued patience

8:57 AM PST

We are continuing to work on events that were created during the downtime yesterday - these are queued and currently paused. We expect this to take time and will update hourly on progress made. Apps and integrations continue to function as normal for new submissions and events.

We have zeroed in on the Slack Connect @ mention issue. When typing an @ mention in the message box, the users name may not appear. We are working to resolve this issue and will update when we have more to share.

7:17 AM PST

We would like to provide an update regarding the current issues in Slack:

• Custom applications, workflows and integrations have had their full functionality restored. Any new events that are created via the Events API will be delivered as expected.
  
  
• Any events that were created during the downtime yesterday are queued and currently paused. Our engineering teams are working on methods to resolve this. It will take some additional time to parse through although we expect to have an update in the near future.
  
  
• We continue to actively investigate reported issues with Slack Connect @ mentions.

We would like to apologise for the disruption and will continue to communicate any updates moving forwards.

6:36 AM PST

The issues related to Slack API's and Workflows should now be resolved however we are still working on replaying older events.

We are continuing to investigate additional issues, including @mentions in Slack Connect not working properly, and will surface more information as it becomes available.

2:51 AM PST

The re-processing of the backend job to attempt to serve queued Events API requests continues, and we expect this to take a couple of hours. 

If you are experiencing the separate symptom relating to Slack Connect where some users are having issues @-mentioning channel members, please reset your cache and refresh by following the below steps:

1. Open the Slack desktop app, then click Help in your computer's top menu bar. (If you’re on Windows 10, click the ☰ three lines icon in the top left of the Slack app.)
2. Select Troubleshooting.
3. Click Clear cache and restart.

We will continue to provide any new information available along with confirmation once the re-processing of the backend job is complete. 

7:28 PM PST

We've began re-processing the backend job to attempt to serve queued Events API requests over the next several hours. Once complete, new requests should begin to trigger as expected.

We will continue to provide relevant updates and confirm once this process is completed.

6:45 PM PST

Users may still be experiencing issues with some custom apps/bots or Slack Connect not working as expected. We have identified the cause of the issue as a symptom of an earlier incident, and are working on a solution with the highest priority.

We will update with more information when available. Thank you for your ongoing patience while we work on this.

6:15 PM PST

We're receiving reports that some custom apps/bots may not be working as expected in Slack. This is related to Events API and is a symptom of an earlier incident about trouble connecting to or loading Slack. A separate symptom related to Slack Connect is also occurring where some users are having issues @-mentioning channel members.

We're on the case and will follow up once we know more. We appreciate your patience.

5:44 PM PST