Postmortem
For the last few weeks we’ve experienced temporary performance degradation in our systems, with the impact gradually growing. The main manifestation of the issue could be seen on delays in replication to one of our databases, leading the App and Portal to reflect outdated states, sometimes off by many minutes.
On October 4th the performance grew so bad that we decided to perform an upgrade of our database infrastructure. Given the effects seen by our users and partners, we decided to do this upgrade as soon as possible, as an unscheduled maintenance. This will be done in two parts, the first one was Thursday October 5th.
The upgrade, Thursday October 5th, was successful, and it was completed 14:00. No user impact was registered during the upgrade.
The same upgrade will be performed on Tuesday, October 10th, for the remaining database infrastructure.