Resolved -
In analyzing the impact, we have determined that the only impacted organizations are internal to Buildkite, used only for monitoring purposes.
There was no impact to customers in this case.
Jan 24, 00:24 UTC
Monitoring -
The rollback has completed, which has resolved the error. Any impacted inbound webhooks have now been reprocessed successfully.
Jan 24, 00:09 UTC
Identified -
We have performed a rollback, and are seeing error rates returning to baseline levels.
Jan 23, 23:57 UTC
Investigating -
We are observing elevated error rates on webhook ingestion. We're currently investigating the issue, and will provide an update soon.
Jan 23, 23:38 UTC