Incident Report - Outages and Performance Issues 02/10/2020

On Friday 02/10/2020 between approximately 12.10pm NZDT and approximately 7.20pm NZDT Invoxy began experiencing database performance and integrity issues. 

The Invoxy team have identified a failing process that was causing data to not be written to the database successfully. Because the data sent was part of an open and sleeping transaction it was returned to the user as though it had been written successfully to the database.

This fault resulted in some users receiving a variety of error messages and inconsistent results. Because the session affected was associated with a single connection pool users may have not experienced any issues. Issues included slow responses, time entries showing as saved but not showing on a later refresh, errors displayed indicating records didn't exist.

The process at fault has been identified and rectified and a system wide review has been undertaken to identify any further potential areas where the same fault could arise. 

Users are advised to review all record updates from these periods to ensure any time records have been correctly updated in the database.