SFT Services Degradation

Incident Report for BICS - SIM for Things

Postmortem

BICS SIM for Things incident on 26/08/2026

  1. Executive Summary

On 26th of August, BICS SFT(Sim For Things) platform experienced a service degradation impacting Data session success rates, API availability, and Portal access. Charging services were also impacted, with a temporary bypass mechanism activated to maintain continuity during the disruption.

Incident was caused by an exceptional processing load on the platform's database layer, triggered by repeated large-scale service profile modification requests submitted in close succession against the same customer account. This drove internal resource consumption beyond configured operating thresholds, ultimately resulting in a database-level service interruption.

Service was restored through a controlled switchover to the resilience site. However, the failover was delayed by a software version mismatch between the primary and resilience environments. This mismatch was an expected consequence of the ongoing staged release rollout, in which the resilience site is upgraded only after stability has been confirmed in the primary environment. The version alignment issue was resolved before the failover could proceed.

‌

2. Customer Impact

2.1 Location of incident

The incident affected the shared SFT platform and was therefore not limited to a specific geographic location.

2.2 Timing (GMT)

‌26/08/2026 11:15 SFT Services degradation started

26/08/2026 11:28 Alarms triggered, issue detected and starting the incident management

26/08/2026 11:51 Customer communication sent

26/08/2026 11:52 Switchover to Marseille resilience site, order given

26/08/2026 12:05 Switchover to Marseille, blockers identified

26/08/2026 13:20 Switchover to Marseille, blockers resolved

26/08/2026 13:25 Switchover to Marseille resilience site initiated

26/08/2026 13:50 Switchover to Marseille resilience site completed

26/08/2026 14:30 API and PORTAL services are restored

26/08/2026 15:15 Data services are restored. Post restoration monitoring confirmed service stability

26/08/2026 18:30 Charging full restoration

2.3 Services impacted

Total duration of service impact during an incident: 4 hours

Total duration of the impact on charging capabilities: Degradation for 7 hours (~50% charging events loss) including full unavailability of 3 h

Impact description and duration per type of service:

  • DATA: Degradation for 4 hours (~40% traffic loss) including full downtime of 1 h
  • API: Degradation (~20% success rate drop) for 3h15 m including full downtime of 1h
  • PORTAL: Degradation (intermittent unavailability) for 3h15 m including full downtime of 1h

‌

3. Detailed incident description

Incident was triggered by a large-scale service-profile modification affecting a significant number of endpoints. Multiple large-scale modification requests – targeting the same set of profiles – generated an exceptional processing workload that exceeded normal operating thresholds and resulted in service degradation. As the workload increased, critical database resources became unavailable, resulting in a temporary loss of database service availability and broader platform impact.

The issue developed in the following stages:

  • The first request locked the service profile, while processing the large update. Subsequent requests were unable to access the same profile and eventually timed out.
  • The database became overloaded due to the high-volume processing activity, resulting in increasing errors and service degradation.
  • Critical database resources were exhausted, causing multiple database nodes to fail and ultimately leading to a database cluster outage.

As a result, customers experienced:

  • Combination of partial degradation and full downtime of API services
  • Portal performance and availability issues
  • Reduced data session success rates including a downtime of one hour. Automated mitigation mechanisms significantly reduced the overall impact during the incident

‌

To resolve the incident, a switchover to the Marseille resilience site was initiated. API and portal services were restored following the migration, with full data traffic subsequently returning to normal operation. Under normal circumstances, such a switchover is completed within approximately 15 minutes. In this case, however, the recovery process took longer due to two factors:

  • Software version misalignment between the primary and resilience environments. This was an expected consequence of the staged software rollout in progress on 26 August, during which the resilience site was scheduled to be upgraded only after the stability of the primary site had been confirmed.
  • Configuration and data synchronization inconsistencies identified during the switchover. These required corrective actions before services could be fully restored on the resilience environment.

‌

4. Actions taken to restore

To restore service, the following actions were performed:

  • Automatic data traffic bypass mechanisms were activated to mitigate the impact on data services. A limited number of sessions remained impacted until full-service restoration was completed.
  • Traffic was switched to the Marseille resilience site. During the switchover, additional issues were identified, which extended the recovery process and delayed full-service restoration.
  • Application routing and connectivity configuration inconsistencies were identified and corrected during the recovery phase.
  • Primary Brussels environment was restored.

‌

5. Future actions identified to improve

Short-term actions (within one month):

  • Temporarily restrict large-scale service profile modification activities, to avoid triggering similar conditions and prevent incident from future happening (DONE)
  • Implement additional safeguards that prevent simultaneous modifications to the same service profile.
  • Review and further enhance the resilience-site switchover procedures to ensure a smooth and reliable transition in the event of a service disruption.
  • Enhance service impact mitigation and bypass procedures, including documentation updates and additional operational training.

Mid-term actions

  • Implement enhanced workload protection mechanisms to prevent abnormal processing activity from affecting platform availability
Posted Sep 03, 2026 - 12:00 CEST

Resolved

Dear Customers,

Following continued observation, we can confirm that all SFT services have remained stable, running via Marseille site. We are therefore closing this incident.

A detailed Root Cause Analysis will follow in the coming days.

Thank you for your patience and understanding throughout this incident.

Kind Regards,
BICS Support Team
Posted Aug 26, 2026 - 21:17 CEST

Update

Please be informed that BICS faced a SIM for Things incident on 26/8/2026, all SFT services went down at 11:50 GMT, following the degradation that had started at 11:15 GMT. Impact was noticed for traffic, provisioning, portal and API services during this time.

The troubleshooting measures involved switchover from site in Brussels to Marseille, that was successful after requiring some additional configurations.
Since 14:30 GMT, the API and portal serviced had been restored, while data/traffic was restored gradually in a controlled way.

All teams are mobilised around the clock to follow up the situation.
The services are currently running via Marseille site, and we are keeping a freeze of activity for Marseille data center, to ensure stability.
Strict monitoring is ongoing.

Detailed Root cause analysis will be conducted and shared in the coming days.
Posted Aug 26, 2026 - 19:02 CEST

Update

Dear Partner,

Portal and API services, which are now operating normally.

Data services are also recovering.

We will provide further updates as more information becomes available.

Thank you for your continued patience and understanding.

Kind Regards,
BICS Support Team
Posted Aug 26, 2026 - 17:49 CEST

Update

Dear Partner,

We are seeing continued recovery of the Portal and API services, which are now returning to normal operation.

Data services, however, are still experiencing some degradation and might take a bit more time to fully recover and our teams continue to monitor the situation closely.

We will provide further updates as more information becomes available.

Thank you for your continued patience and understanding.

Kind Regards,
BICS Support Team
Posted Aug 26, 2026 - 16:46 CEST

Update

Dear Partner,

We are observing that the portal and API services are recovering now.
For data services we are still observing degradation

We keep monitoring closely and will share further updates.
Thank you for your patience and understanding.

Kind Regards,
BICS Support Team,
Posted Aug 26, 2026 - 15:08 CEST

Update

We are continuing to investigate this issue.
Posted Aug 26, 2026 - 13:52 CEST

Investigating

Dear customers,

We are currently experiencing partial service degradation affecting SFT services. Our team is aware of the issue and actively investigating the root cause.
Further updates will be shared as more information becomes available.

We apologize for any inconvenience and thank you for your patience.

BICS Support Team
Posted Aug 26, 2026 - 13:51 CEST
This incident affected: Management Portal, API, Data Connectivity, API data stream, and eSIM for Consumer platform.