On Thursday 7th September Zaptec APIs and user interfaces (Portal and mobile apps) started experiencing intermittent issues with performance, leading users to experience long periods to interact with our services. Zaptec engineers identified the cause as a cumulative effect of multiple systems dealing with increasing load from users and third parties, leading to cascading delays from system to system.
For the following two days we addressed multiple bottlenecks in our database infrastructure that gradually alleviated the problem, while experiencing further increases in load that again caused intermittent slowdowns. From Saturday 9th our database optimizations stabilized system performance, which we have since been monitoring.
We also continue work in substantial improvements to our data processing architecture, which we expect to put in place in the coming weeks — this will allow Zaptec to scale the capacity to handle increasing load, as our installed base continues to grow.