Tuesday October 15, 2024

Summary

Beginning at 1:30 PM PDT until around 1:41 PM PDT on October 15, 2024, some users were unable to load GovSlack, leading to an error page. We traced this back to an increase in API error rates, triggered by a configuration issue in a backend database service responsible for caching data. To mitigate the impact, we temporarily redirected traffic away from the affected service and corrected the configuration. Once the configuration change was made, full functionality to the database service was restored, API error rates returned to normal, and users were able to access GovSlack again. We've worked to enhance our internal monitoring systems to reduce the likelihood of similar issues occurring in the future.

4:40 PM PST

Resolved

Our health metrics have returned to normal and users should see Slack load normally. Thank you for your patience.

3:44 PM PST

Update

We have mitigated the loading issues some users might have been seeing. We are currently monitoring our health metrics as we search for the root cause.

3:13 PM PST

Investigating

Some GovSlack users may be experiencing issues loading the app entirely. We’re currently investigating the issue and will provide an update once we have more information.

1:53 PM PST