Last Friday, Molecule experienced a handful of 1- to 4-minute service interruptions between 5:55 PM and 6:59 PM Central Time. We apologize for the interruption, and would like to share the findings from our internal investigation.

What Happened

At 5:55 PM, the Molecule app was unavailable for 4 minutes, because our primary database was suffering excessive load. This happened 3 more times (with up to 4-minute interruptions each) over the next hour.

Why?

The database load was primarily due to a mis-timed backup job running during a high-load time for Molecule. The backup is designed to run during the US night, but due to a time zone configuration change (i.e., US Central to UTC) it was running during the US day. Complicating this was an unusual set of customer activity querying a large amount of old data (which is more resource-intensive to query).

Preventing This Type of Issue

We have, of course, fixed the backup time zone issue. Other items we are working on as well:

• Addressing the issue of high-load queries for old data
• Ensuring that all of our scheduled jobs are set to the same time zone, to prevent confusion

‌

Thank you for your patience, and again--we’re sorry for the interruption.

This incident has been resolved.

We are currently investigating.