Slowness in user interfaces and API
Incident Report for Zaptec
Postmortem

On Thursday 7th September Zaptec APIs and user interfaces (Portal and mobile apps) started experiencing intermittent issues with performance, leading users to experience long periods to interact with our services. Zaptec engineers identified the cause as a cumulative effect of multiple systems dealing with increasing load from users and third parties, leading to cascading delays from system to system.

For the following two days we addressed multiple bottlenecks in our database infrastructure that gradually alleviated the problem, while experiencing further increases in load that again caused intermittent slowdowns. From Saturday 9th our database optimizations stabilized system performance, which we have since been monitoring.

We also continue work in substantial improvements to our data processing architecture, which we expect to put in place in the coming weeks — this will allow Zaptec to scale the capacity to handle increasing load, as our installed base continues to grow.

Posted Sep 13, 2023 - 14:53 CEST

Resolved
This incident has been resolved.
Posted Sep 13, 2023 - 11:49 CEST
Monitoring
A fix has been applied and we're currently monitoring our systems to confirm that performance returns to normal levels.
Posted Sep 08, 2023 - 18:11 CEST
Update
We are currently experiencing sporadic bursts of activity that cause a slowdown to our interfaces (Portal and App) and our API. Some operations can take several seconds to complete, and occasionally time out. If you experience a timeout, please wait some seconds and try the operation again. Our engineers are actively investigating and attempting to mitigate the issue.
Posted Sep 08, 2023 - 12:32 CEST
Update
We are continuing to investigate this issue. Currently, we do not have any new information.
Posted Sep 08, 2023 - 12:07 CEST
Update
We are continuing to investigate this issue.
Posted Sep 07, 2023 - 22:19 CEST
Investigating
We are receiving reports about issues with the portal. We are investigating the issue, and will update soon.
Posted Sep 07, 2023 - 14:28 CEST
This incident affected: Zaptec Cloud Services (Portal, API).