Data Service Degradation.

Incident Report for BICS - SIM for Things

Postmortem

Incident report

  1. Executive Summary

Data services and API performance degradation

2. Customer Impact

2.1 Location of incident

No dependency on the location.

2.2 Timing (GMT)

  • 21/07/2026 16:00 GMT

Incident started – degradation of data traffic success rate detected.

  • 21/07/2026 16:20 GMT

Issue escalated to the Development team for investigation.

  • 21/07/2026 16:23 GMT

An emergency bridge call was initiated.

Following the technical assessment, an emergency rollback was approved.

  • 21/07/2026 16:53 GMT

Rollback activities were started.

  • 21/07/2026 17:15 GMT

Service was restored.

  • 21/07/2026 17:39 GMT

Rollback execution and restart of all application nodes were successfully completed. Monitoring ongoing.

  • 21/07/2026 17:49 GMT

Customer communication was sent confirming full-service restoration.

Total duration - 1 hour 15 minutes.

2.3 Services impacted

Data traffic (partially, no charging)
API Performance – approximately 15% of API requests failed, returning errors such as: ‘invalid input parameters’, ‘SIM provisioning failures’, 'endpoint-not-found'.

3. Detailed incident description

The incident was caused by a software defect introduced during the SFT platform maintenance and update performed on 21 July.

As part of this release, a new database update query was introduced in the counter-reset workflow to soft-delete notification delivery records. Subsequent analysis revealed that the newly implemented query had a negative impact on database performance due to a data type mismatch between the query parameter and the target table column.

The issue became critical when a large-scale counter reset was executed at customer level. Because of the data type mismatch and the high volume of records stored in the target table, the queries generated excessive load on the database cluster. This led to increased latency and failures in backend database operations, preventing the application from resolving certain critical entities required for normal service operation.

Consequently, the following services were impacted:

Gx/Gy success rate (data charging requests)

API performance was affected, resulting in an increased error rate.

Immediately upon detection of the issue, a data‑traffic automatic charging bypass procedure was activated to restore data services. This allowed data connectivity to remain operational, although traffic charging was temporarily disabled during the bypass period - services remained available to customers, but usage was not billed.

Immediately after issue detection situation was discussed with developer team and partial rollback of the update was done, with next restart of CBS nodes, what restored services.

4. Actions taken to restore

  • Automatic data charging bypass (mitigation to restore data services)
  • Partial rollback of the update.
  • Applications restart

5. Future actions identified to improve

  • Strengthen testing procedures for MySQL query changes before production deployment.
Posted Jul 24, 2026 - 15:05 CEST

Resolved

This incident has been resolved.
Posted Jul 21, 2026 - 20:04 CEST

Monitoring

Dear Partner,

Please note that the issue has been resolved.

We are continuously monitoring the service closely to ensure service availability and stability.

Thank you.
Kind Regards,
BICS Support Team.
Posted Jul 21, 2026 - 19:49 CEST

Investigating

Dear Partner,

We are observing partial degradation on data services.

We are aware of this issue and working on resolving it.

Further information will be shared.

Thank you for your understanding.

Kind Regards,

BICS Support Team.
Posted Jul 21, 2026 - 19:00 CEST
This incident affected: Data Connectivity.