Sunday September 24, 2023

Issue summary:
From 12:54 PM PDT on September 22, 2023 to 09:13 AM PDT on September 25, 2023, custom emoji rendered as text for a small number of Enterprise Grid users.

We made a code change that inadvertently introduced unnecessary logging for routine emoji events. This unexpected logging increased traffic to the custom emoji infrastructure, impacting its performance.

We deployed a fix to remove the unnecessary logging, which reduced traffic to the custom emoji systems and restored normal emoji rendering.

A small fraction of the affected customers may have experienced lingering issues as the fix was gradually rolled out to everyone.

Note: Due to an internal tooling error, we initially reported the times in this summary in UTC, but labeled with PDT. We have corrected this mistake and apologize for any confusion caused.

1:28 PM PST

This issue is now resolved for all users. Thank you again for bearing with us in the meantime.

10:26 AM PST

We've implemented a fix. While this has fully resolved the issue for many users, it's possible that a small number of customers may experience some continued impact until 10:30 AM PDT tomorrow, September 26, 2023.

We'll share an update here at that time. In the meantime, please write in to feedback@slack.com if you're still encountering issues, as our support team may be able to assist.

5:36 PM PST

We’ve shipped a potential fix for this emoji issue, but we’re still monitoring closely. We’ll let you know when functionality is fully restored. Thank you for your patience.

11:29 AM PST

We’ve identified the cause of this issue and we’re working on getting a fix out as soon as possible. We’ll be back once we have another update to share.

9:41 AM PST

We're still working on this, but we don't have any new information to share at this time. We'll provide another update as soon as we have more news.

7:48 PM PST

We're still digging in to the cause of the custom emoji issue. We'll be back with an update in 30 minutes.

7:19 PM PST

Our investigation is still ongoing. Thank you for bearing with us as we look in to this. We'll share another update in 30 minutes.

6:50 PM PST

We've established that this is only impacting a small number of Enterprise Grid organizations at this time.

We're still actively investigating, and will provide an update in 30 minutes.

6:19 PM PST

There's no additional news to share just yet. We'll be back with another update in 30 minutes.

5:48 PM PST

A small number of customers may be experiencing issues with custom emoji not rendering in messages or workflows. We are investigating and will be back with an update in 30 minutes.

We're very sorry for the disruption.

5:17 PM PST