Sunday April 7, 2024

Incident

Issues managing user accounts with the SCIM API

Issue Summary: From Friday, April 5 at 11:48 AM PDT to Tuesday, April 9 at 3:37 PM PDT, users updated via our SCIM API were deactivated. We traced this to a change that required API calls for this method to require a field not supplied by all identity providers. We rolled back this change and began remediation efforts to reactivate and restore users to their previous state. Though we previously stated some apps may need to be reactivated by Workspace Owners, we've since determined that all apps were reinstated through our remediation efforts. We're analyzing data from this event to understand ways to mitigate similar issues that may occur in the future.

9:06 AM PST

We have restored all impacted users and they should be able to access Slack once more. There are some remaining issues with user group membership and administrative roles. Workspace Owners will need to make some manual adjustments to reinstate these roles and permissions. Additionally, apps installed by impacted users may be deactivated, and Workspace Owners will need to reactivate these We apologize for the disruption to your work. Please get in touch with us if you have any questions about making these changes.

6:46 PM PST

We’re still actively working on this issue, but we don’t have any new information to share at this time. Thanks for sticking with us as we continue working to restore access for impacted users.

12:45 PM PST

We are continuing to work to identify a full list of affected users so that remediation work can continue. Users with urgent need for access can have their access restored by resyncing the user's account via your identity provider. Thank you for your patience while we work to resolve this. We'll share another update once we have more meaningful info to share.

11:13 AM PST

We're still refining and confirming the list of impacted users. To restore access for an affected user urgently, please resync the user account via your identity provider. We'll provide another update as soon as we have new information to share.

9:29 PM PST

We're still narrowing down our lists to make sure we're only looking at user accounts that have been incorrectly deactivated. If you need to restore access for an impacted user urgently, you should be able to do so by resyncing the user account via your identity provider. We'll have another update for you in 30 minutes.

9:00 PM PST

We've established that a very small number of user accounts have been impacted. We're working to confirm this list to make sure we've identified the correct accounts before we begin remediation. We'll be back with an update in 30 minutes.

8:31 PM PST

We have deactivated the toggle that caused the issue. No further user accounts will be impacted. We are now working to identify the affected accounts and plan remediation steps. Thanks for bearing with us. We'll share an update in 30 minutes.

7:56 PM PST

We believe we've identified the cause of the issue and are working to revert the relevant code change. We'll provide another update in 30 minutes.

7:27 PM PST

Some customers may be experiencing issues managing user accounts with the SCIM API, including cases where users are being incorrectly deactivated. We're investigating and we'll be back with an update in 30 minutes.

6:57 PM PST

Features affected

Workspace/Org Administration

Status

Incident