Incident report
Data services and API performance degradation
2. Customer Impact
2.1 Location of incident
No dependency on the location.
2.2 Timing (GMT)
Incident started – degradation of data traffic success rate detected.
Issue escalated to the Development team for investigation.
An emergency bridge call was initiated.
Following the technical assessment, an emergency rollback was approved.
Rollback activities were started.
Service was restored.
Rollback execution and restart of all application nodes were successfully completed. Monitoring ongoing.
Customer communication was sent confirming full-service restoration.
Total duration - 1 hour 15 minutes.
2.3 Services impacted
Data traffic (partially, no charging)
API Performance – approximately 15% of API requests failed, returning errors such as: ‘invalid input parameters’, ‘SIM provisioning failures’, 'endpoint-not-found'.
3. Detailed incident description
The incident was caused by a software defect introduced during the SFT platform maintenance and update performed on 21 July.
As part of this release, a new database update query was introduced in the counter-reset workflow to soft-delete notification delivery records. Subsequent analysis revealed that the newly implemented query had a negative impact on database performance due to a data type mismatch between the query parameter and the target table column.
The issue became critical when a large-scale counter reset was executed at customer level. Because of the data type mismatch and the high volume of records stored in the target table, the queries generated excessive load on the database cluster. This led to increased latency and failures in backend database operations, preventing the application from resolving certain critical entities required for normal service operation.
Consequently, the following services were impacted:
Gx/Gy success rate (data charging requests)
API performance was affected, resulting in an increased error rate.
Immediately upon detection of the issue, a data‑traffic automatic charging bypass procedure was activated to restore data services. This allowed data connectivity to remain operational, although traffic charging was temporarily disabled during the bypass period - services remained available to customers, but usage was not billed.
Immediately after issue detection situation was discussed with developer team and partial rollback of the update was done, with next restart of CBS nodes, what restored services.
4. Actions taken to restore
5. Future actions identified to improve