Incident Report: Root Cause Analysis (RCA)

Incident: Degraded Performance in Backend API
Date: April 29, 2025
Start Time: 12:01 PM IST
Resolution Time: 12:19 PM IST
Status Page Update: https://status.fyno.io/incident/553591

Affected Services:
Application Backend API (Fyno WebApp at https://app.fyno.io)

Incident Summary:
On April 29, 2025, at approximately 12:01 PM IST, the backend infrastructure experienced intermittent failures, resulting in HTTP 500 errors. During this period, certain API functionalities were impacted; however, sending/receiving notifications, sending/receiving delivery reports, and Fyno Shorty services remained unaffected.

Timeline of Events:
- 12:01 PM IST: Detection of increased error rates (HTTP 500) on backend services.
- 12:06 PM IST: Root cause identified as an unexpected error handling condition within a rate-limiting module.
- 12:09 PM IST: Engineering team initiated deployment of the corrective fix after validation.
- 12:17 PM IST: Deployment completed of the corrective fix after validation.
- 12:19 PM IST: Services fully restored and stabilized.

Root Cause:
The backend services experienced failures due to unhandled exceptions occurring within the rate-limiting functionality. Specifically, certain requests were causing an issue in validation of authentication tokens, causing backend processes to terminate unexpectedly.

Immediate Resolution
During the incident, the engineering team proactively increased backend application server capacity, enabling customers to continue using the web application without significant disruption.

Resolution:
The team implemented comprehensive error handling within the affected module. The updated implementation ensures robust validation of authentication tokens and graceful recovery from unexpected errors, thereby preventing recurrence.

Preventive Measures:
Enhanced validation procedures added to critical authentication modules.
Additional monitoring and alerting configurations implemented to proactively detect similar issues in the future.
Comprehensive code reviews and automated tests reinforced to detect potential points of failure proactively.

Conclusion:
We apologize for any inconvenience this may have caused and reaffirm our commitment to service reliability and quality. Our engineering team has proactively addressed this incident and improved our monitoring and validation processes to avoid similar disruptions in the future.

The backend (Fyno Application service) is back up.

The fix has been deployed.

Team has identified the issue and are deploying a fix shortly.

We are currently seeing 500 in Fyno Backend API calls. Team is working on a fix.
Note: Sending/Receiving of notifications is not affected.