Tuesday October 31, 2023

Issue summary:

From 5:01 PM PDT on October 31, 2023 to around 5:54 PM PDT, customers with international data residency in the Paris, France region, experienced issues connecting to Slack and sending messages.

A routine credential rotation caused database sync issues for the Paris, France data residency region. We reverted this code change and restarted the affected databases, resolving the issue for all impacted customers.

Once we had mitigated the immediate impact and restored connectivity for customers in the Paris, France data residency region, we reviewed the credential rotation for all other data residency regions to ensure the same issue would not occur anywhere else.

Please note that the start and end time of the incident have been edited for accuracy.

8:45 PM PST

We have resolved the connectivity issue for all affected customers. Everyone should be able to connect as normal once again.

Thank you for your patience while we addressed this.

7:43 PM PST

We're completing some last checks to ensure that this issue is fully resolved for all customers.

We'll be back with an update as soon as we have confirmation that Slack is up and running for everyone.

7:24 PM PST

We're continuing to monitor the situation and we'll provide an update in 30 minutes or as soon as connectivity is fully restored, whichever is soonest.

6:53 PM PST

We've pushed a potential fix and we're seeing significant improvements, though we're still monitoring the situation closely.

Affected customers may need to refresh Slack to reconnect. Please open Slack and press Cmd/Ctrl + Shift + R. Slack will reload but will not sign you out.

We'll share another update in 30 minutes or as soon as we have more information.

6:22 PM PST

We are investigating some recent code changes that may be causing issues, and we're preparing to roll these back. We are also working to prevent the impact from spreading to additional data regions.

We'll be back with an update in 30 minutes.

5:57 PM PST

Some customers with international data residency in the Paris, France region may be experiencing issues connecting to Slack or sending messages. We're investigating and we'll be back with more information in 30 minutes or as soon as we have news to share.

We apologize for the disruption to your work. Thank you for bearing with us.

5:22 PM PST