On September 16, 2026, Medusa Cloud experienced a partial outage affecting applications running in our EU region during an upgrade of our networking infrastructure.
The upgrade introduced a version of Knative Serving/Kourier with a startup performance regression. In our EU data plane, which manages a large number of application routes, the affected controllers could not finish loading routing state before their health-check deadline. Kubernetes therefore restarted them repeatedly.
Because the controllers were unable to fully initialize the routing layer after the upgrade, some application traffic could not be routed successfully and returned 502 or 503 responses.
The primary customer impact lasted approximately 25 minutes, from 13:30 to 13:55 UTC.
The incident affected applications hosted in the EU region.
The outage was partial rather than complete. Some requests continued to succeed throughout the incident, while a significant portion of requests returned 502 or 503 responses.
Applications in other Medusa Cloud regions were not affected.
All times are UTC.
13:27 — We began upgrading Knative Serving and Kourier in the EU data plane.
13:28 — New versions of the routing controllers began starting.
13:29 — The new controllers began failing their health checks before completing initialization and were restarted by Kubernetes.
13:30 — Customer-facing errors increased sharply, with requests beginning to return 502 and 503 responses.
13:31–13:54 — The affected controllers repeatedly attempted to start, failed to complete initialization within their health-check window, and restarted. We investigated the controller startup behavior and worked to restore the routing layer.
13:54 — We increased the resources available to the affected controllers, allowing them to complete initialization.
~13:55 — Customer applications began recovering and traffic returned to normal.
16:12 — We upgraded Knative to version 1.22, which contains the upstream fix for the startup regression.
The incident was caused by a startup performance regression in the version of Knative/Kourier deployed during the upgrade.
During startup, Kourier's controller reconstructs the routing state for applications in the cluster. A regression in the affected version caused this process to perform inefficiently when initializing a large number of routes. In our EU data plane, initialization consequently took longer than the controller's configured health-check window.
Kubernetes interpreted the delayed startup as an unhealthy controller and restarted it. Because all newly upgraded controller replicas encountered the same condition, they entered a restart loop and could not reliably provide the routing state required by the Kourier gateways.
The regression was subsequently fixed upstream and is included in Knative 1.22.
For technical reference, the upstream issue and fix are documented in the net-kourier issue and corresponding fix.
We first restored service by increasing the resources available to the Kourier controllers, which allowed them to complete initialization and repopulate the required routing state.
After service had recovered, we upgraded Knative to version 1.22, which contains the upstream fix for the regression.
We are taking the following actions to reduce the likelihood and impact of similar incidents:
We apologize for the disruption this incident caused to customers running workloads in our EU region.