Wednesday February 26, 2025

From around 6:45 AM PST on February 26, 2025 to 8:30 AM PST on February 27, 2025, Slack users experienced issues with our Events API. Specifically, custom applications, integrations, and bots may not have worked as expected. Additionally, some users may have experienced issues with @mentioning Slack Connect channel members.

This issue was a result of the mitigation measures we took while working to resolve a concurrent incident. To help us address the initial incident, we paused the Events API job queue and rate-limited a non-critical API endpoint that was generating an outsized load. 

While the queue was paused, incoming Events API requests were placed in a backlog. Once we stabilized the database tier and restored critical Slack functionality, we re-enabled the Events API queue at around 11:27 PM PST on February 26, 2025. From this point onwards, all new events were processed as expected. We made the decision not to replay the backlogged events to avoid confusion over significantly delayed app responses, and to avoid a surge of queued events for both our own systems and those of third-party providers. 

We continued working to address lingering issues with Slack Connect @mentions, fully resolving the incident for all impacted users at 8:30 AM PST on February 27, 2025.

9:12 PM PST

We continue to see custom applications, integrations and workflows working as expected. We consider the Events API issue resolved, and would like to share some additional detail:

Due to mitigation efforts, our Events API built up a considerable backlog of events that were not delivered as expected to apps. Once the database tier became available, we re-enabled these queues and by 11:27pm PST on 2025-02-26, all events were being properly delivered as expected. 

We chose to not replay the large number of events for delivery that were buffered from the incident because app event delivery semantics often depend on timely delivery for correctness. In addition we wanted to avoid potentially overloading either our systems or those of third-party systems with the surge of queued events which had been paused throughout the outage.

We thank you for your patience as we worked through resolution efforts, and we apologize for the disruption caused.

7:28 PM PST

Our work related to events is still in motion. We'll provide an update as new information becomes available. We appreciate your patience along the way.

10:12 AM PST

The work we're doing related to the events that queued during the downtime yesterday is still ongoing. We'll provide an update on this as we continue making progress, but it is still expected to take some time.

The issue users were seeing regarding Slack Connect @ mention trouble has now been resolved. Users may need to refresh Slack (Cmd/Ctrl + Shift + R) for the fix to appear. Thank you for your continued patience

8:57 AM PST

We are continuing to work on events that were created during the downtime yesterday - these are queued and currently paused. We expect this to take time and will update hourly on progress made. Apps and integrations continue to function as normal for new submissions and events.

We have zeroed in on the Slack Connect @ mention issue. When typing an @ mention in the message box, the users name may not appear. We are working to resolve this issue and will update when we have more to share.

7:17 AM PST

We would like to provide an update regarding the current issues in Slack:

• Custom applications, workflows and integrations have had their full functionality restored. Any new events that are created via the Events API will be delivered as expected.
  
  
• Any events that were created during the downtime yesterday are queued and currently paused. Our engineering teams are working on methods to resolve this. It will take some additional time to parse through although we expect to have an update in the near future.
  
  
• We continue to actively investigate reported issues with Slack Connect @ mentions.

We would like to apologise for the disruption and will continue to communicate any updates moving forwards.

6:36 AM PST

The issues related to Slack API's and Workflows should now be resolved however we are still working on replaying older events.

We are continuing to investigate additional issues, including @mentions in Slack Connect not working properly, and will surface more information as it becomes available.

2:51 AM PST

The re-processing of the backend job to attempt to serve queued Events API requests continues, and we expect this to take a couple of hours. 

If you are experiencing the separate symptom relating to Slack Connect where some users are having issues @-mentioning channel members, please reset your cache and refresh by following the below steps:

1. Open the Slack desktop app, then click Help in your computer's top menu bar. (If you’re on Windows 10, click the ☰ three lines icon in the top left of the Slack app.)
2. Select Troubleshooting.
3. Click Clear cache and restart.

We will continue to provide any new information available along with confirmation once the re-processing of the backend job is complete. 

7:28 PM PST

We've began re-processing the backend job to attempt to serve queued Events API requests over the next several hours. Once complete, new requests should begin to trigger as expected.

We will continue to provide relevant updates and confirm once this process is completed.

6:45 PM PST

Users may still be experiencing issues with some custom apps/bots or Slack Connect not working as expected. We have identified the cause of the issue as a symptom of an earlier incident, and are working on a solution with the highest priority.

We will update with more information when available. Thank you for your ongoing patience while we work on this.

6:15 PM PST

We're receiving reports that some custom apps/bots may not be working as expected in Slack. This is related to Events API and is a symptom of an earlier incident about trouble connecting to or loading Slack. A separate symptom related to Slack Connect is also occurring where some users are having issues @-mentioning channel members.

We're on the case and will follow up once we know more. We appreciate your patience.

5:44 PM PST