On September 24, 2026, we identified a delay in one of our telemetry-processing services. The service was receiving data faster than it could process it, which created a growing backlog.
Core charging functionality and primary communication services continued to operate normally. However, delayed telemetry processing affected the responsiveness of the Solar feature for some APM devices.
The backlog was fully processed by October 1, 2026, and the incident was subsequently resolved.
The incident had a limited impact on customers:
September 24, 13:13 — Monitoring detected a growing difference between the amount of telemetry received and the amount processed. The team began investigating and initiated incident communications.
September 24, 15:06 — An internal incident was declared. The service appeared healthy according to the available monitoring, but its processing capacity was insufficient to keep up with incoming data.
September 24, 15:19 — The team identified oversized telemetry records that could cause groups of messages to be processed repeatedly. A change was deployed to isolate and discard invalid records rather than retrying the entire group.
September 24, 15:22 — The change improved resilience but did not resolve the processing delay. The investigation found that the service's limited monitoring made it difficult to identify the main performance constraint.
September 24, 15:57 — Further analysis indicated that the issue was within the application's processing method rather than the underlying data-storage service.
September 24, 17:12 — An updated version of the service was prepared for testing. Because core services remained operational, the team prioritized careful validation over an immediate production deployment to avoid introducing additional risk.
September 24, 21:41 — The updated service was running successfully in the development environment. Testing continued to confirm that all processing scenarios worked as expected.
September 30, 17:01 — The updated service was deployed to production. Processing performance improved immediately, and the telemetry backlog began decreasing.
October 1, 00:30 — The telemetry backlog was fully processed.
October 1, 08:30 — The incident was declared resolved.
The affected service processed data-storage operations one at a time. As the volume of telemetry increased, this sequential approach could no longer provide sufficient throughput, causing the backlog to grow.
The service was also based on an older application design with limited performance monitoring. This made the underlying bottleneck difficult to identify and extended the investigation.
Oversized telemetry records contributed to unnecessary reprocessing but were not the primary cause of the incident.
We replaced the affected service with an updated version that can safely process multiple storage operations concurrently. This substantially increased processing capacity and allowed the service to clear the backlog.
We also improved the handling of oversized records so that a single invalid record no longer causes a larger group of telemetry messages to be reprocessed.
To reduce the likelihood and duration of similar incidents, we have:
Modernized the affected service and aligned it with our current application standards.
Added improved monitoring for service health, processing performance, and backlog growth.
Introduced controlled parallel processing to support higher telemetry volumes.
Improved the handling of oversized or invalid telemetry records.
Begun reviewing the wider telemetry-processing architecture to simplify data flows and make service dependencies and customer impact easier to identify.
This was the final application using the relevant legacy architecture. Its modernization removes a significant source of technical debt and improves our ability to detect, diagnose, and resolve future performance issues.