Summary

On May 10, 2024, from 08:03 AM to 08:09 AM CST, a brief service interruption affected our US-based users. We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

What happened?

The system became unavailable to US users for approximately 6 minutes due to excessive frontend resource requests that resulted in unresponsive web application nodes.

Why?

The outage was triggered by an unusual surge in resource demand due to excessive frontend resource requests to the Molecule web application. This placed an unexpectedly high load on our systems, leading to unresponsive web application nodes.

Corrective/Preventive Actions 

To enhance our system's reliability and efficiency, we are implementing the following measures:

• Enhanced System Monitoring: We have upgraded our monitoring capabilities to detect and address anomalies more effectively before affecting service availability.
• Scalability Improvements: We are increasing the scalability of our web servers to handle spikes in demand more fluidly.
• Rate Limiting Enhancements: We plan to improve our rate-limiting measures to prevent any single user or service from making excessive requests in a short time frame.
• Software Updates: A specific software issue that allowed for unnecessary resource consumption to help alleviate the issue was discovered and fixed.

Once again, we thank you for your patience and understanding as we strive to ensure the reliability of our services.

We have identified the likely cause of the outage this morning and will continuing to monitor.

This morning, Molecule had an outage but is now operational. We are investigating.