Outage Report - 29/5/2020

Issue
On Friday 29/05/2020 at approximately 5:57pm Invoxy experienced a downtime that lasted until 6:35pm. This was experienced as login issues by Users whereby they were unable to log in to Invoxy. The server reported Connection is closed.

Microsoft Support have indicated that there were no issues with the database during this time and that the issues were due to errors within the application.

The system was made available through a restart of the NodeJS application that was reporting errors.

We have been unable to determine the root cause for this outage. Additional logging and monitoring will be implemented in order to reduce the impact of this type of event and to provide more information about the root case if it recurs.

Remediation Plan

1. TCP Connection re-use was indicated as an issue by Microsoft. Take steps to re-use TCP connections for outbound calls.

2. Reduce contention on the database server by directing PowerBI queries to a separate mirrored database.

3. Add Application Insights to allow deeper tracing of events and processes within the server runtime.

4. Increase monitoring and alerts around application issues and availability.

5. Increase application-level logging to gain deeper insight into points of failure.