Monday February 26, 2024

Issue summary:
On February 26, 2024 from 9:00 AM PST to 9:36 AM PST, some users experienced problems sending and loading messages in Slack. Others may have completed these actions successfully but more slowly than expected.

Earlier today, we deployed a routine update to part of our network infrastructure. During this work, we deprovisioned some of our proxy servers and provisioned a new set. Some of our webapp servers took longer than expected to sync with the new list of proxy servers, resulting in a spike in connection errors. This coincided with the regular 9:00 AM PST increase in traffic to our servers, which may have exacerbated impact.

By around 9:36 AM PST, error rates had decreased and user impact had completely subsided, most likely due to our automatic scaling functionality. However, we're continuing to analyze data from this event to fully understand the root cause and prevent similar issues in the future.

5:47 PM PST

Starting at 9:00 AM PST on Monday, February 26 to 9:30 AM PST, a small percentage of users encountered trouble sending messages, loading messages, and accessing Slack.

These errors were caused by an increased load on our servers, and reduced as our servers balanced the load across our system. User should no longer be encountering issues with Slack.

Please send a note to feedback@slack.com if you continue to have any trouble.

9:49 AM PST

Issue summary:
From 12:30 PM PST to 1:05 PM PST on February 26, 2024, users were unable to create workflows.

We determined that the related API method needed updating in order to eliminate inconsistencies that were causing elevated replication lag.

We deployed an update to the API method correcting this issue, and all workflow creation will now work as expected.

6:50 PM PST

We've released a fix and users should no longer encounter any issues creating new workflows. Thank you for bearing with us as we worked to get this resolved.

3:56 PM PST

We have isolated the cause of this issue and are working on getting a fix out as soon as we can. We’ll be back as soon as we have another update, and we appreciate your patience in the meantime.

3:20 PM PST

We are currently investigating an issue where users are not able to create workflows. Running existing workflows is not impacted. We will update as we have more information to share.

1:02 PM PST