Incident Report: Incomplete Valuation Data - Resolved

Date of Resolution: July 1, 2024

Summary

On Monday morning, July 1, 2024, an issue where certain trades were missing valuation entries was identified and addressed the same day. We apologize for any inconvenience this may have caused and appreciate your understanding.

What happened?

The Molecule application has a feature that limits how far into the future valuation entries will be created when trades are made.

However, there was no mechanism to create entries for skipped as_of dates when they became relevant. These previously excluded entries became relevant for the first time on July 1, and some customers began reporting the issue on Friday, June 28.

Why

• When implementing the as_of_date limits for entry building, a corresponding job to build skipped entries before they became relevant was not available to be run in production.
• The system did not include sufficient checks to identify these issues ahead of time.

Corrective/Preventive Actions 

• A script was deployed immediately to generate missing entries, addressing the issue and correcting the problem for impacted customers as soon as possible.  This script will run daily to ensure all entries are built until we remove the limit for good.
• An internal check has been created to catch missing entries ahead of time.

‌

We apologize for any inconvenience caused and appreciate your understanding as we work to improve our systems and prevent similar issues in the future.

This incident has been resolved. All current-day valuations line up with expectations.
Later today we will run a script to fix any missing place-holder valuations for tomorrow's as_of date, and we will continue monitoring forward-dated valuations

Multiple trades have not valued for today. We are in the process of fixing manually and doing a root cause analysis.
We will reach out to individual risk managers as we clear each customer, and we will update this incident as we get a comprehensive solution.