Summary

On January 27, 2026, some customers experienced missing or delayed trade valuations due to a performance issue in our market-data processing pipeline.

What Happened?

A recent database upgrade exposed an inefficiency in our market-data copier, leading to excessive memory usage for certain large ("big-bang") accounts. This slowed downstream systems, causing valuation jobs to stall or fail. The incident was mitigated by 9:35 AM CT the same day.

Root Cause

The issue stemmed from a systemic regression in the market-data copier, where a validation step loaded large datasets into memory. A change in the database query planner post-upgrade exacerbated this behavior, overwhelming system resources and delaying processing pipelines.

Impact

• Some trades did not generate legs or valuation results.
• Affected customers saw blank or missing valuations during the incident window.

Immediate Mitigations

• Declared a SEV-0 and initiated incident response.
• Scaled workers and reduced data ingress to stabilize the system.
• Cleared backlogs and reprocessed affected trades.

Permanent Fixes & Preventive Actions

• Optimized memory-intensive validation logic with a more efficient approach.
• Enhanced retry logic and queue handling for workers and RabbitMQ.
• Added a set of new early-warning alerts that trigger on conditions that may indicate a problem.
• Published internal response playbooks and added auto-scaling for queue backlogs.
• Introduced new load tests and validation specifically for large account scenarios.
• Reprocessed impacted trades and validated key fields. Support will confirm with affected customers directly.

We apologize for the disruption and remain committed to strengthening the resilience of our platform. For any trade-specific questions, please reach out to Support.

Trade processing has returned to normal and all queued jobs are now being processed as expected. We are continuing to closely monitor the platform to ensure stability and prevent a recurrence.

Thank you for your patience.

The underlying issue causing trade processing delays has been identified, and corrective actions are currently underway. Trades will begin processing as the system recovers, and remaining queued work will be completed.
We will share a final update once the issue is fully resolved and normal performance is restored.

We are aware of an issue impacting trade processing times on the Molecule platform. Trades are being received and captured as expected, however some jobs—such as leg building—are experiencing extended queue times.

The team is actively investigating and working to restore normal processing performance. Further updates will be shared as progress is made.