Tuesday October 14, 2014

At 7:48 AM PDT, we deployed a bad permissions change to a single directory across our entire web cluster. This disconnected approximately 13% of Slack's currently active users. This also coincided with our daily peak usage time on Slack, and our infrastructure was unable to handle the heavy reconnect traffic on top of new user connections for the day. We've rolled back the permissions change and added additional capacity to our web cluster to handle heavy reconnect traffic. We've also identified several opportunities to improve performance during the connection process. We deployed some fixes already and will implement the other changes today. We apologize for the disruption — we let you down. This is not the level of service that we aim to provide, nor is it the level of service that our customer community deserves. Please contact us at https://my.slack.com/help/requests/new if you continue to see problems or need any additional help.

11:50 AM PST

All users should be able to connect to Slack. Integrations are catching up and delayed bot messages will start showing up for all teams. There was no data loss.

11:11 AM PST

Service has been restored for many users and we're working quickly to get all systems up and running for everyone.

9:16 AM PST

We've identified the problem and are working on a fix.

8:15 AM PST

slack.com and other pages on the site are currently inaccessible. We're investigating.

8:07 AM PST