Incidents and maintenance reported on status page for marble https://status.checkmarble.com/ en https://status.checkmarble.com/maintenance/1072418 Fri, 09 Oct 2026 21:20:00 -0000 https://status.checkmarble.com/incident/1072418#703de277462f4979310f41d24e9898f88c2a2c8a77603e639304794f1e057c6a Maintenance Maintenance completed https://status.checkmarble.com/maintenance/1072418 Wed, 23 Sep 2026 20:24:35 -0000 https://status.checkmarble.com/incident/1072418#2394e28f89affcfbb2f0f3ba71696a75fb186c2d488acc7ec6693f891a059207 Maintenance Marble will be unavailable as we restart our production databases for a major version increase (postgres 17 => 18). https://status.checkmarble.com/incident/1065670 Wed, 16 Sep 2026 13:57:00 -0000 https://status.checkmarble.com/incident/1065670#b2e5d9484539d8c712f9fc637c5267f1b297c901e39d77aee2656135a328a3dd Incident ## Summary As part of the offboarding of an organization, its client database was stopped. The Cloud SQL Auth Proxy was still configured to connect to that database, causing the proxy to fail at startup. The worker starts only after the proxy is available because it requires a database connection. As a result, the worker did not start and background processing was unavailable. An already-running API backend instance continued serving requests. During restoration, a database environment variable was removed. This left the remaining database environment variables non-sequential. The API backend started, but the Cloud SQL Auth Proxy did not configure customer databases after the gap, causing ingestion and decisions to fail for approximately half of customers until the configuration was corrected. ## Impact - All customers experienced a worker downtime from 12:10 to 15:03 CEST, causing background processing to be delayed or blocked, primarily case creation and workflows. - From 15:03 to 15:08, ingestion and decisions failed for approximately half of customers. - The UI remained available throughout the incident. ## Timeline - 12:10: A client database was stopped as part of organization offboarding. - 12:10: The Cloud SQL Auth Proxy failed to start because it was still configured to connect to the stopped database. The worker could not start. - 12:10–15:03: Background processing was unavailable. - 15:03: A restoration configuration was deployed. The worker backend started successfully. - 15:03–15:08: Because a removed database environment variable left subsequent variable names non-sequential, the proxy did not configure databases after the gap. Ingestion and decisions failed for approximately half of customers. - 15:08: Environment variable ordering was corrected. All customer database connections, worker processing, ingestion, and decisions were restored. ## Detection The worker does not have a sufficiently frequent heartbeat, so the outage was not identified immediately. No Sentry issue was generated because the worker container only starts once the Cloud SQL Auth Proxy is live. Since the proxy failed first, the worker never started and could not report its failure. ## Root Cause The Cloud SQL Auth Proxy sidecar is configured to connect to all customer databases during startup. A single unavailable client database prevented the proxy from starting, which in turn prevented the worker from starting. The restoration issue was caused by the proxy's reliance on sequential database environment variable names. Removing one variable created a gap, excluding all following customer databases from the proxy configuration. ## Follow-ups - Add a short-interval heartbeat for the worker to detect worker downtime quickly. - Rework Cloud SQL connectivity by moving proxy management into Go code rather than relying on a sidecar container. - Ensure an unavailable customer database cannot prevent shared worker infrastructure from starting. - Remove the dependency on sequential database environment variable names, or validate the configuration before deployment. https://status.checkmarble.com/maintenance/1030284 Mon, 24 Aug 2026 07:44:03 -0000 https://status.checkmarble.com/incident/1030284#af8681f1f88a6101fad3a9b864a244bc753f64b48d1e3c9d23260d33580e3ca8 Maintenance All Marble databases will be restarted for a regular minor version upgrade. The downtime should be limited to a few minutes, around 02:00 CET. https://status.checkmarble.com/maintenance/924523 Mon, 15 Jun 2026 14:21:39 -0000 https://status.checkmarble.com/incident/924523#c53e803aeeaff6a5c44b5d13a65c4e6d74ccb2cf119e0b66a5431b4a87abf6dd Maintenance The Marble databases will be briefly restarting on 16/06/2026, 2am, CET. https://status.checkmarble.com/maintenance/734489 Mon, 29 Sep 2025 19:07:18 -0000 https://status.checkmarble.com/incident/734489#fb5b17d4c5e8ab18dd1657d7d3e0029db96264878377138f078e75081d8d524e Maintenance The APIs and the application will be unavailable for customers, for up to one hour but most likely for 10 to 15minutes. https://status.checkmarble.com/incident/602078 Thu, 12 Jun 2025 20:41:00 -0000 https://status.checkmarble.com/incident/602078#7e45aa99e61dfaee1780c614d70fefab683614f7e59d9f8e20cc8ee9c8b2f29d Incident A global GCP incident impacted Marble APIs in production and staging for 1h15. The incident was resolved by the Google infrastructure team. https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW https://status.checkmarble.com/maintenance/599364 Mon, 09 Jun 2025 08:33:54 -0000 https://status.checkmarble.com/incident/599364#935927541fd0a5beca7c86b6dd290006ea18266a737c85a81d90bff4eab1df46 Maintenance The app will be unresponsive for about 15 minutes as we upgrade some of the databases to a new major version.