Summary

As part of the offboarding of an organization, its client database was stopped. The Cloud SQL Auth Proxy was still configured to connect to that database, causing the proxy to fail at startup. The worker starts only after the proxy is available because it requires a database connection.

As a result, the worker did not start and background processing was unavailable. An already-running API backend instance continued serving requests.

During restoration, a database environment variable was removed. This left the remaining database environment variables non-sequential. The API backend started, but the Cloud SQL Auth Proxy did not configure customer databases after the gap, causing ingestion and decisions to fail for approximately half of customers until the configuration was corrected.

Impact

• All customers experienced a worker downtime from 12:10 to 15:03 CEST, causing background processing to be delayed or blocked, primarily case creation and workflows.
• From 15:03 to 15:08, ingestion and decisions failed for approximately half of customers.
• The UI remained available throughout the incident.

Timeline

• 12:10: A client database was stopped as part of organization offboarding.
• 12:10: The Cloud SQL Auth Proxy failed to start because it was still configured to connect to the stopped database. The worker could not start.
• 12:10–15:03: Background processing was unavailable.
• 15:03: A restoration configuration was deployed. The worker backend started successfully.
• 15:03–15:08: Because a removed database environment variable left subsequent variable names non-sequential, the proxy did not configure databases after the gap. Ingestion and decisions failed for approximately half of customers.
• 15:08: Environment variable ordering was corrected. All customer database connections, worker processing, ingestion, and decisions were restored.

Detection

The worker does not have a sufficiently frequent heartbeat, so the outage was not identified immediately.

No Sentry issue was generated because the worker container only starts once the Cloud SQL Auth Proxy is live. Since the proxy failed first, the worker never started and could not report its failure.

Root Cause

The Cloud SQL Auth Proxy sidecar is configured to connect to all customer databases during startup. A single unavailable client database prevented the proxy from starting, which in turn prevented the worker from starting.

The restoration issue was caused by the proxy's reliance on sequential database environment variable names. Removing one variable created a gap, excluding all following customer databases from the proxy configuration.

Follow-ups

• Add a short-interval heartbeat for the worker to detect worker downtime quickly.
• Rework Cloud SQL connectivity by moving proxy management into Go code rather than relying on a sidecar container.
• Ensure an unavailable customer database cannot prevent shared worker infrastructure from starting.
• Remove the dependency on sequential database environment variable names, or validate the configuration before deployment.