Harden bulk retries, parallelize guest uploads, and move guest map below gallery

2026-09-11 18:54:46 -04:00
parent 4bc48db656
commit 1e299a9914
25 changed files with 376 additions and 62 deletions
@@ -10,3 +10,40 @@
  unchanged. The final image must pass the Sharp WebP smoke test.
 - If memory pressure remains high, move builds to a separate builder rather than
  increasing concurrency or running more diagnostic builds on the live VM.
+
+## Coolify recovery and build isolation (2026-09-11)
+
+Docker 29.3.0 crashed with SIGSEGV in its embedded BuildKit mount/read-entrypoint
+path during the `4bc48db` deployment. The VM did not reboot, and the application
+containers were not OOM-killed. Restarting the existing Manyangles containers
+restored service. The trace identifies the failing process, not a confirmed
+upstream defect or hardware cause.
+
+- `/etc/docker/daemon.json` now enables `live-restore`. It was validated and
+  applied with `systemctl reload docker`, without restarting running containers.
+  The prior configuration is backed up at
+  `/etc/docker/daemon.json.before-manyangles-recovery-20260911`.
+- Manyangles uses the named `manyangles-isolated` buildx builder with the
+  `docker-container` driver and `moby/buildkit:v0.33.0`, instead of the builder
+  embedded inside dockerd. Other applications' build selection is unchanged.
+- The builder has a 3 GiB RAM limit, 4 GiB combined RAM/swap limit, two-CPU quota,
+  and two BuildKit execution slots. Images automatically load into the local
+  Docker image store. Its cache is stored in a dedicated Docker volume.
+- Coolify's application-level custom Compose build command is:
+
+  ```sh
+  docker compose --parallel 1 --project-name nzuqxqw47tbt117lrpw3f8ch build --builder manyangles-isolated --pull
+  ```
+
+  Coolify injects the project directory, Compose file, and build environment.
+  Builder metadata lives under `/root/.docker/buildx`, which Coolify mounts into
+  its helper. Do not select this builder globally or prune its volume during builds.
+
+Verify using `docker info` (Live Restore Enabled), `docker buildx inspect
+manyangles-isolated` as root, container health, and the public
+`https://ma.hadlock.tech/api/health/ready` endpoint. Live restore mitigates daemon
+outages; it does not provide zero-downtime Compose rollouts or protect against VM
+failure. Do not deliberately crash/restart the shared daemon to test it in production.
+
+References: [live restore](https://docs.docker.com/engine/daemon/live-restore/),
+[containerized builders](https://docs.docker.com/build/builders/drivers/docker-container/).