Incidents reported on status page for Fyno https://status.fyno.io/ https://d1lppblt9t2x15.cloudfront.net/logos/33ab66ca3f3c259acf8bc98bcd9a6156.png https://status.fyno.io/ en https://status.fyno.io/incident/553591 Tue, 29 Apr 2025 12:17:00 -0000 https://status.fyno.io/incident/553591#97eeff5e500897f86d1b78b7ea14e6f9634b9046cc39c92a3faf63a16a367aa4 Incident Report: Root Cause Analysis (RCA) Incident: Degraded Performance in Backend API Date: April 29, 2025 Start Time: 12:01 PM IST Resolution Time: 12:19 PM IST Status Page Update: https://status.fyno.io/incident/553591 Affected Services: Application Backend API (Fyno WebApp at https://app.fyno.io) Incident Summary: On April 29, 2025, at approximately 12:01 PM IST, the backend infrastructure experienced intermittent failures, resulting in HTTP 500 errors. During this period, certain API functionalities were impacted; however, sending/receiving notifications, sending/receiving delivery reports, and Fyno Shorty services remained unaffected. Timeline of Events: - 12:01 PM IST: Detection of increased error rates (HTTP 500) on backend services. - 12:06 PM IST: Root cause identified as an unexpected error handling condition within a rate-limiting module. - 12:09 PM IST: Engineering team initiated deployment of the corrective fix after validation. - 12:17 PM IST: Deployment completed of the corrective fix after validation. - 12:19 PM IST: Services fully restored and stabilized. Root Cause: The backend services experienced failures due to unhandled exceptions occurring within the rate-limiting functionality. Specifically, certain requests were causing an issue in validation of authentication tokens, causing backend processes to terminate unexpectedly. Immediate Resolution During the incident, the engineering team proactively increased backend application server capacity, enabling customers to continue using the web application without significant disruption. Resolution: The team implemented comprehensive error handling within the affected module. The updated implementation ensures robust validation of authentication tokens and graceful recovery from unexpected errors, thereby preventing recurrence. Preventive Measures: Enhanced validation procedures added to critical authentication modules. Additional monitoring and alerting configurations implemented to proactively detect similar issues in the future. Comprehensive code reviews and automated tests reinforced to detect potential points of failure proactively. Conclusion: We apologize for any inconvenience this may have caused and reaffirm our commitment to service reliability and quality. Our engineering team has proactively addressed this incident and improved our monitoring and validation processes to avoid similar disruptions in the future. https://status.fyno.io/incident/553591 Tue, 29 Apr 2025 06:49:00 -0000 https://status.fyno.io/incident/553591#f69b72e0ed8cc91889d25809ee4e56f6951a9f0a072900d2a32c77d67a4bfa62 The backend (Fyno Application service) is back up. https://status.fyno.io/incident/553591 Tue, 29 Apr 2025 06:49:00 -0000 https://status.fyno.io/incident/553591#13176342cad75ac7e1f4a8e635cfd8863a8724d6a46a161483964744b761f6ef The fix has been deployed. https://status.fyno.io/incident/553591 Tue, 29 Apr 2025 06:36:00 -0000 https://status.fyno.io/incident/553591#a0c88057b3187d75bfd8df02f4643e5c96c22db103bcdf89611900f9fa52ca88 Team has identified the issue and are deploying a fix shortly. https://status.fyno.io/incident/553591 Tue, 29 Apr 2025 06:31:00 -0000 https://status.fyno.io/incident/553591#2041c367c600fee6aaec1c96a74d45f4316c1ec0c740b917d682eca3e6ddc063 We are currently seeing 500 in Fyno Backend API calls. Team is working on a fix. Note: Sending/Receiving of notifications is not affected. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 08:05:00 -0000 https://status.fyno.io/incident/515999#352a6c56083893190e77c962fbf74533f5b175a1f7904e279d1eebd94c951dc4 We have recovered from the disruption. We apologize for any inconvenience that this may have caused. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 08:05:00 -0000 https://status.fyno.io/incident/515999#352a6c56083893190e77c962fbf74533f5b175a1f7904e279d1eebd94c951dc4 We have recovered from the disruption. We apologize for any inconvenience that this may have caused. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 08:05:00 -0000 https://status.fyno.io/incident/515999#352a6c56083893190e77c962fbf74533f5b175a1f7904e279d1eebd94c951dc4 We have recovered from the disruption. We apologize for any inconvenience that this may have caused. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 08:05:00 -0000 https://status.fyno.io/incident/515999#352a6c56083893190e77c962fbf74533f5b175a1f7904e279d1eebd94c951dc4 We have recovered from the disruption. We apologize for any inconvenience that this may have caused. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 07:52:00 -0000 https://status.fyno.io/incident/515999#fa10378f5e5c560a796c02113f7dfead75e9c841f3f5496eacfda368e317ebd0 Our engineering team has identified the root cause of the disruption. We are currently deploying a fix. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 07:52:00 -0000 https://status.fyno.io/incident/515999#fa10378f5e5c560a796c02113f7dfead75e9c841f3f5496eacfda368e317ebd0 Our engineering team has identified the root cause of the disruption. We are currently deploying a fix. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 07:52:00 -0000 https://status.fyno.io/incident/515999#fa10378f5e5c560a796c02113f7dfead75e9c841f3f5496eacfda368e317ebd0 Our engineering team has identified the root cause of the disruption. We are currently deploying a fix. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 07:52:00 -0000 https://status.fyno.io/incident/515999#fa10378f5e5c560a796c02113f7dfead75e9c841f3f5496eacfda368e317ebd0 Our engineering team has identified the root cause of the disruption. We are currently deploying a fix. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 07:49:00 -0000 https://status.fyno.io/incident/515999#81ef26a76f5aeb854fa5f9e4bff83a6d1c41eee610c26c8b9e50a14545aa5d8f We are currently investigating this issue. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 07:49:00 -0000 https://status.fyno.io/incident/515999#81ef26a76f5aeb854fa5f9e4bff83a6d1c41eee610c26c8b9e50a14545aa5d8f We are currently investigating this issue. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 07:49:00 -0000 https://status.fyno.io/incident/515999#81ef26a76f5aeb854fa5f9e4bff83a6d1c41eee610c26c8b9e50a14545aa5d8f We are currently investigating this issue. https://status.fyno.io/incident/515999 Mon, 17 Jun 2024 07:49:00 -0000 https://status.fyno.io/incident/515999#81ef26a76f5aeb854fa5f9e4bff83a6d1c41eee610c26c8b9e50a14545aa5d8f We are currently investigating this issue. https://status.fyno.io/incident/516009 Mon, 14 Aug 2023 04:13:00 -0000 https://status.fyno.io/incident/516009#2b781d81f345e32a86dec556083f724a4fb8ee65d6644c4911bce23e79856709 We have recovered from the disruption. We apologize for any inconvenience that this may have caused. https://status.fyno.io/incident/516009 Mon, 14 Aug 2023 01:30:00 -0000 https://status.fyno.io/incident/516009#2e9e743f936b8c8ebfc0f8deac185eb9f49190c78e2d10c1bd33cb811d38fce5 Our engineering team has identified the root cause of the disruption. We are currently deploying a fix. https://status.fyno.io/incident/516009 Mon, 14 Aug 2023 00:32:00 -0000 https://status.fyno.io/incident/516009#4d995b6a49690459183a23c798b13e0ae3fc584dcd8c05e3dd0e57eb0dcba2fa Our technical team is investigating the root cause of the issue. https://status.fyno.io/incident/516009 Mon, 14 Aug 2023 00:13:00 -0000 https://status.fyno.io/incident/516009#562a7086fb67443f8a4ebcf0412888da7effd07b9af1e649ba14a6053f3b49cf Our monitoring system has detected a sudden dip in the number of delivery reports and inbound messages received by Fyno Services. https://status.fyno.io/incident/516025 Wed, 31 May 2023 08:12:00 -0000 https://status.fyno.io/incident/516025#0729efa408347b24a122afb534cd5cf4d8121d0ceb83e86eb52e85421d858062 We have recovered from the disruption. We apologize for any inconvenience that this may have caused. https://status.fyno.io/incident/516025 Wed, 31 May 2023 07:35:00 -0000 https://status.fyno.io/incident/516025#4ad059461beb94765f472ce124b5c7b5a0fe3b2708e7152d8d84728a7ad3751f Our engineering team has identified the root cause of the disruption. We are currently deploying a fix. https://status.fyno.io/incident/516025 Wed, 31 May 2023 07:28:00 -0000 https://status.fyno.io/incident/516025#7df981c26e7a6c2ec5ebf077cfcc46ff7c883d73dde4394c39eb9579e60ac9b2 We are seeing a delay in processing delivery reports and inbound messages.