# Incident Report – Increased Latency in Charger Data Processing Pipeline
# Description
On the evening of 3 May 2026, our systems experienced increased latency in the internal pipeline that delivers charger data to downstream services. As a result, processing of the data was delayed. This led to:
- Charging sessions starting with delays of up to approximately 20 minutes.
- Our public charger-state API experiencing elevated error rates and timeouts for part of the incident window.
The issue has since been resolved, and systems have returned to normal operation.
# Timeline (Oslo time)
## May 3, 2026
21:00 – Internal monitoring detected that one of our downstream services was falling behind in processing incoming charger data, indicating increased delay in the data pipeline.
21:08 – An alert was triggered and the on-call engineer began investigation.
21:30 – The team identified increased latency in the internal data delivery path responsible for sending charger data to our processing services.
21:58 – A support case was opened with our infrastructure provider for further investigation of the upstream latency.
22:18
As a protective measure to reduce load and avoid excessive state-change events, one of our internal charger state services was temporarily scaled down.
As a side effect, our public charger-state API, which depends on this service, started returning more HTTP 500 errors and timeouts as clients cancelled long-running requests.
22:43 – Internal metrics indicated that the downstream processing delay had returned to normal levels.
22:53 – Latency in the internal data delivery path decreased back to normal.
23:07 – Initial status update communicated internally.
23:15 – The previously scaled-down charger state service was restored to normal capacity.
23:16 – The public charger-state API returned to normal behaviour.
23:30 – Scope of the incident was refined and communicated.
23:39 – The backlog of delayed charger data was fully processed.
## May 4, 2026
7:57– The incident was marked as resolved.
# Root Cause
An internal messaging pipeline responsible for delivering charger data exhibited significantly increased latency.
This caused many chargers to repeatedly switch between offline and online states in our systems, generating a high volume of state-change events for downstream components.
The additional load from these state changes, combined with the bursty delivery of delayed data, led to further processing delays and directly impacted the charger-state API.
# Follow-Up Actions
Adding more targeted monitoring and alerting on the latency within our internal data pipeline, so we can identify and mitigate similar issues more quickly.
Improving our charger state service so it can detect when there are delays in upstream data delivery and temporarily pause or adjust offline/online detection to avoid large numbers of false transitions.
We are continuing to work with our infrastructure provider to better understand the underlying cause of the upstream latency and to reduce the likelihood of recurrence.