Thursday November 14, 2024

Issue summary: From around 7:00 AM PST to 8:30 AM PST on November 14, some users may have experienced minor delays using Slack. Initially, we believed that there may have been widespread user impact and our updates reflected this. During our investigation, however, we confirmed that most user impact was mitigated by our automatic retry mechanisms and load balancers. A scheduled upgrade added unanticipated load to some parts of our infrastructure. This caused a small spike in errors across our systems. As an immediate mitigation step, we rolled back this planned change. We completed the rollback at 8:30AM PST on November 14, which resolved the issue for all affected customers. The databases that had already picked up the change required restarts. We then carefully shepherded the database fleet through rolling restarts, resolving all errors by 1:00 AM PST November 15. Although user impact had already been mitigated, we continued sharing updates until this process was completed.

4:00 PM PST

This issue is now fully resolved. Any trouble sending messages or uploading files, along with loading threads or channels has been fixed. We apologize for any inconvenience and appreciate your patience as we sorted this out.

10:54 AM PST

We are still monitoring the situation closely as we handle the remaining remediation work. There is no user impact at this time, and we'll provide another update when we have new information or when we can confirm that the issue is fully resolved.

9:03 PM PST

We have mitigated the impact to users and they no longer should experience issues with sending messages, uploading files, or loading threads and channels. We're continuing to monitor our health metrics until we've fully resolved the issue. We'll be back with another update once we have news to share.

12:49 PM PST

We’re seeing signs of improvement, but we’re still actively monitoring the situation. We’ll confirm once the issue is fully resolved or we have another update to share. Thank you for your patience in the meantime.

9:48 AM PST

Some users may be having trouble sending messages and files. There may also be some trouble loading threads and channels. We’re investigating and will let you know as soon as we know more. We appreciate your patience in the meantime.

8:38 AM PST