Sunday November 3, 2024

From 12:02 AM PDT to 1:19 AM PDT on Sunday, November 3, 2024, some users experienced trouble logging into Slack with SSO, provisioning users with SCIM, received duplicated notifications, and had trouble triggering workflows.

The system responsible for the handling of database replications and routing traffic had unexpected failures due to a time-logic configuration change. As a result, requests involving these database replicas timed out, impacting all SCIM operations, SSO logins, notification handling, and workflow execution.

As an immediate fix, we manually processed these database replications, restoring full functionality to the affected services.

After verifying that requests were successfully serviced, we implemented a backend logic update to allow for future system changes to be handled automatically, reducing the likelihood of this reoccurring.

12:28 PM PST

Our engineering team has resolved the issue. Impacted users should be able to login to SSO. SCIM operations are recovering.
Some users may have experienced repeated notifications during our resolution efforts and issues triggering workflows, these should no longer be occurring.

2:03 AM PST

Our engineering team have identified the issue and are working to resolve impact.
Impacted users should be seeing recovery and are able to login to SSO. SCIM operations are recovering.

1:32 AM PST

We’ve isolated the problem and we’re focused on getting Slack working again as soon as possible. Thank you for your continued patience.

Additionally, Slack is facing issues with scheduled tasks and connectivity.

1:16 AM PST

Some users may experience trouble logging into Slack. In addition, SCIM operations and workflows are also failing.

We’re on the case and will provide another update once we have more details to share. We’re sorry for the disruption.

12:59 AM PST