Summary of Incident

On September 30 and again on October 2, Molecule experienced a brief interruption with the ICE market data feed. Both of these occurances were resolved within an hour or less.

What happened?

In both instances, connections to ICE stalled, and real-time pricing and trade ingestion were interrupted. The issue was resolved by re-establishing ICE connections, and any missed trades were successfully reprocessed after the service was restored.

Why?

A failure in the ICE FIX listener caused the incident. That is the service that maintains connectivity between Molecule and ICE. The listener encountered a resource exhaustion issue that caused connections to stall without automatically recovering.

Resolution

Our team restored service quickly by restarting ICE connectivity for affected accounts. Once connections were re-established, all missed trades were ingested and processed successfully.

Next Steps

We are implementing improvements to prevent recurrence:

• Infrastructure updates: Increasing system resources and tuning the ICE listener to handle higher load.
• Automatic recovery: Adding self-healing logic to re-establish connections without manual intervention.
• Monitoring improvements:  Enhancing alerts to detect early warning signs and automatically restart services if needed.

We sincerely apologize for any disruption. While the issue has been resolved with no data loss, we are committed to ensuring this does not recur and are strengthening our systems and processes accordingly.

The ICE market data feed was down for multiple accounts due to a stalled connection between customer accounts and the ICE FIX listener. The issue was resolved by manually restarting the service for each affected account.

We are currently experiencing ICE adaptor stability issues. This might cause a delay in trades flowing in from ICE. We are currently instigating this as a priority.