Web App, Windy and MCP Unavailable

Background

Windmill generates Windy reports in the background to prepare content for our users. On October 12, 2026, a large number of these reports started at the same time.

Our services connect to the database through a connection proxy. The proxy usually shares a small number of database connections between many requests. But a problem in how our services use their database connections stopped the proxy from sharing them. Each request held its own database connection, so the large report load opened thousands of connections, and each connection used memory on the database server.

At 10:00 AM ET, the database server that handles reads ran out of memory and restarted two times. We added more read servers and the service recovered at 10:40 AM. At 11:44 AM, when the reports started to finish, the load moved to our primary database server, which handles writes. That server ran out of memory, and our database provider automatically moved the primary role to a standby server three times. Each move interrupted database connections for 1–2 minutes.

During these periods, the Windmill web app, Windy and MCP were unavailable or slow, and many requests failed. Some review chats were sent later than planned.

Timeline

• 10/12 @ 10:00 AM ET - Report generation starts. The read database server runs out of memory.
• 10/12 @ 10:09 AM ET - Web app becomes unavailable. Status page updated at 10:10 AM.
• 10/12 @ 10:10 AM ET - Cause linked to report generation.
• 10/12 @ 10:16 AM ET - Read database server restarts and recovers, but some requests continue to fail.
• 10/12 @ 10:25 AM ET - Read database server runs out of memory a second time.
• 10/12 @ 10:33 AM ET - Additional read servers added.
• 10/12 @ 10:40 AM ET - Error rate drops to 0%. Incident resolved and monitoring continues.
• 10/12 @ 11:25 AM ET - Read servers slow down again under load. More read servers added.
• 10/12 @ 11:44 AM ET - Primary database server runs out of memory. The primary role moves to a standby server.
• 10/12 @ 12:00 PM ET - Reports finish, load moves to write database.
• 10/12 @ 12:04 PM ET - Primary database server runs out of memory a second time. Status page updated at 12:07 PM.
• 10/12 @ 12:42 PM ET - Primary database server runs out of memory a third time.
• 10/12 @ 1:05 PM ET - Query improvements deployed. Database memory recovers and errors stop.
• 10/12 @ 1:28 PM ET - Incident resolved and monitoring continues.

Remediation

We understand the impact this downtime had; our customers rely on Windmill for critical work and we apologize greatly for interrupting it.

Internally, we are implementing changes to prevent similar incidents from occurring in the future. This includes:

• Lowering the maximum number of connections each database server can hold
• Continuing to optimize our database queries for agent-driven workloads
• Upgrading our performance review systems to spread report generation over a longer period of time

More resources now online, this is resolved.

We have identified the problem, working on a solution to deploy more resources.

Windmill is experiencing high-load and elevated error rates. We have determined the root cause and are actively investigating a fix