deadstyle/forgo

Watch

Fork 0

fix: isolate runner job networks and pin cache.host so concurrent jobs stop sharing service container aliases #1

New issue

Open

opened 2026-07-20 20:46:05 -05:00 by deadstyle · 1 comment

deadstyle commented

2026-07-20 20:46:05 -05:00

Owner

Problem

The Forgejo runner at ~/Development/forgejo (separate repo, shared production infra) sets container.network: "forgejo-internal" in runner-config/config.yml — a named, long-lived bridge — together with runner.capacity: 2. Because the network is named rather than empty, the runner attaches every job container and every service container to one shared network instead of creating a per-job network.

Forgejo's own admin documentation calls out this exact combination:

"Changing the network mode to a custom network name will isolate the network of all job containers, but they will not be isolated from each other. In combination with a runner.capacity configured > 1 where multiple workflows will be executing concurrently on the same custom named network, the availability of workflow jobs can be affected as container names can conflict with each other; for example, if two jobs run a service container named pgsql, DNS resolution for the name pgsql will round-robin between the two containers."

Two distinct consequences:

Reliability. Concurrent jobs whose service containers share an alias get round-robined between each other's databases. This is the root cause of kirria#205. That ticket works around it inside kirria/.forgejo/workflows/ci.yml by giving Postgres a run-scoped unique name; this ticket is the platform-level fix, which would benefit every repo on the instance rather than just kirria.

Security. Job containers currently share a network with Forgejo's own production db and mail containers. Forgejo's security documentation flags this as an access-boundary weakness independent of any reliability concern — arbitrary workflow code on this instance sits on the same bridge as the production database.

Plan

Two lines, not one. Clearing container.network alone will likely break the runner's built-in cache server.

runner-config/config.yml — set container.network: "" so the runner auto-creates an isolated per-job network (forgejo-runner's documented default).
runner-config/config.yml — pin cache.host to a host-reachable address. It is currently "" (auto-detect). forgejo-runner's config.example.yaml warns:

"If empty, it will be detected automatically. It may be impossible to figure out the host automatically if the containers or host running the workflows reside on a different network than the Forgejo runner... you can specify which IP or hostname to use to reach the internal cache server."

Per-job networks are precisely the condition that breaks auto-detect. Because docker_host: "automount" means runner and spawned job containers are siblings against the same host daemon, a bridge-network container can always reach the host's IP even when isolated from other bridges — so the host's stable LAN or docker0-gateway IP is the right value. Leave proxy_port: 0 (auto-select).

Runner→Forgejo (http://forgejo:3000) is unaffected. That connection is made by the forgejo-kirria-org-ci-runner container itself for polling, registration, log upload, and status reporting; its network placement is fixed in docker-compose.yml (networks: - forgejo-internal) and is not governed by container.network, which only applies to the ephemeral containers the runner spawns. Job containers never talk to Forgejo over the internal bridge — actions/checkout clones over the public ROOT_URL via normal outbound NAT.

Context

This is inferred from documented behavior, not a live reproduction. Confidence is moderate-high on the reachability analysis, lower on there being no fourth-order surprise.
Canary before trusting broadly: one manual job trigger, watching actions/cache steps specifically. Cache failure is the predicted failure mode if cache.host is wrong.
Affected runners all share runner-config/config.yml via a read-only bind mount: deploy-runner-1, deploy-runner-2, kirria-org-runner, kirria-org-ci-runner. A change here hits all four at once, which is an argument for the canary.
kirria#205 is deliberately independent of this ticket and is not superseded by it. That fix is scoped to one repo, needs no shared-infra sign-off, and stays correct even if cache.host tuning goes sideways here.
Diagnosis and the two-line diff came from a fable-model analysis session; see kirria#205 for the full evidence chain.

Out of scope

Anything inside the kirria repo. That is #205.
Splitting runners onto separate networks per org, or any broader runner topology redesign.

## Problem The Forgejo runner at `~/Development/forgejo` (separate repo, shared production infra) sets `container.network: "forgejo-internal"` in `runner-config/config.yml` — a named, long-lived bridge — together with `runner.capacity: 2`. Because the network is named rather than empty, the runner attaches every job container *and* every service container to one shared network instead of creating a per-job network. Forgejo's own admin documentation calls out this exact combination: > "Changing the network mode to a custom network name will isolate the network of all job containers, but they will not be isolated from each other. In combination with a `runner.capacity` configured `> 1` where multiple workflows will be executing concurrently on the same custom named network, the availability of workflow jobs can be affected as container names can conflict with each other; for example, if two jobs run a service container named `pgsql`, DNS resolution for the name `pgsql` will round-robin between the two containers." Two distinct consequences: **Reliability.** Concurrent jobs whose service containers share an alias get round-robined between each other's databases. This is the root cause of kirria#205. That ticket works around it inside `kirria/.forgejo/workflows/ci.yml` by giving Postgres a run-scoped unique name; this ticket is the platform-level fix, which would benefit every repo on the instance rather than just kirria. **Security.** Job containers currently share a network with Forgejo's own production `db` and mail containers. Forgejo's security documentation flags this as an access-boundary weakness independent of any reliability concern — arbitrary workflow code on this instance sits on the same bridge as the production database. ## Plan Two lines, not one. Clearing `container.network` alone will likely break the runner's built-in cache server. 1. `runner-config/config.yml` — set `container.network: ""` so the runner auto-creates an isolated per-job network (forgejo-runner's documented default). 2. `runner-config/config.yml` — pin `cache.host` to a host-reachable address. It is currently `""` (auto-detect). forgejo-runner's `config.example.yaml` warns: > "If empty, it will be detected automatically. It may be impossible to figure out the host automatically if the containers or host running the workflows reside on a different network than the Forgejo runner... you can specify which IP or hostname to use to reach the internal cache server." Per-job networks are precisely the condition that breaks auto-detect. Because `docker_host: "automount"` means runner and spawned job containers are siblings against the same host daemon, a bridge-network container can always reach the host's IP even when isolated from other bridges — so the host's stable LAN or docker0-gateway IP is the right value. Leave `proxy_port: 0` (auto-select). Runner→Forgejo (`http://forgejo:3000`) is unaffected. That connection is made by the `forgejo-kirria-org-ci-runner` container itself for polling, registration, log upload, and status reporting; its network placement is fixed in `docker-compose.yml` (`networks: - forgejo-internal`) and is not governed by `container.network`, which only applies to the ephemeral containers the runner spawns. Job containers never talk to Forgejo over the internal bridge — `actions/checkout` clones over the public `ROOT_URL` via normal outbound NAT. ## Context - This is inferred from documented behavior, not a live reproduction. Confidence is moderate-high on the reachability analysis, lower on there being no fourth-order surprise. - Canary before trusting broadly: one manual job trigger, watching `actions/cache` steps specifically. Cache failure is the predicted failure mode if `cache.host` is wrong. - Affected runners all share `runner-config/config.yml` via a read-only bind mount: `deploy-runner-1`, `deploy-runner-2`, `kirria-org-runner`, `kirria-org-ci-runner`. A change here hits all four at once, which is an argument for the canary. - kirria#205 is deliberately independent of this ticket and is not superseded by it. That fix is scoped to one repo, needs no shared-infra sign-off, and stays correct even if `cache.host` tuning goes sideways here. - Diagnosis and the two-line diff came from a fable-model analysis session; see kirria#205 for the full evidence chain. ## Out of scope - Anything inside the kirria repo. That is #205. - Splitting runners onto separate networks per org, or any broader runner topology redesign.

deadstyle commented

2026-07-20 20:54:14 -05:00

Author

Owner

History: why the setting exists

Traced the provenance, which changes the risk assessment.

5a29733 "add network" (Dec 14 21:24) flipped network: "" -> "forgejo-internal", five minutes after ddc97ef "rename the runners" (21:19). At that point every runner ran against a separate dind daemon:

- DOCKER_HOST=tcp://docker-dind:2375

78beee3 "take runners out of dind" later moved the runners onto the host socket (docker_host: automount, no DOCKER_HOST). The network setting was never revisited after that move. The upstream explanatory comments were also stripped from config.yml in a later rewrite, which is why nothing in the file explains the choice today.

Implication: the setting most likely solved a dind-era reachability problem that no longer exists. That lowers the expected risk of reverting it, but does not remove the need for a canary — one of the original justifications may still bind.

Correction to the Plan above

The Plan claims job containers never talk to Forgejo over the internal bridge, on the grounds that actions/checkout clones over the public ROOT_URL. That is asserted, not verified, and the whole ticket rests on it.

The runners register with GITEA_INSTANCE_URL=http://forgejo:3000 — an internal-only name. Whether a job container's actions/checkout derives its clone URL from Forgejo's configured public ROOT_URL or from the runner's registration URL decides the outcome:

If ROOT_URL: clearing the network is safe, outbound NAT handles it.
If registration URL: job containers on an auto-created network cannot resolve forgejo, and checkout breaks for every repo on the instance.

Verify before or during the canary. The failure mode is total, not partial — watch the checkout step, not just actions/cache.

Whichever way it resolves, add a comment to config.yml recording the decision. Its absence is why this took a git-archaeology session to reconstruct.

## History: why the setting exists Traced the provenance, which changes the risk assessment. `5a29733 "add network"` (Dec 14 21:24) flipped `network: ""` -> `"forgejo-internal"`, five minutes after `ddc97ef "rename the runners"` (21:19). At that point every runner ran against a **separate dind daemon**: ``` - DOCKER_HOST=tcp://docker-dind:2375 ``` `78beee3 "take runners out of dind"` later moved the runners onto the host socket (`docker_host: automount`, no `DOCKER_HOST`). **The network setting was never revisited after that move.** The upstream explanatory comments were also stripped from `config.yml` in a later rewrite, which is why nothing in the file explains the choice today. Implication: the setting most likely solved a dind-era reachability problem that no longer exists. That lowers the expected risk of reverting it, but does not remove the need for a canary — one of the original justifications may still bind. ## Correction to the Plan above The Plan claims job containers never talk to Forgejo over the internal bridge, on the grounds that `actions/checkout` clones over the public `ROOT_URL`. **That is asserted, not verified, and the whole ticket rests on it.** The runners register with `GITEA_INSTANCE_URL=http://forgejo:3000` — an internal-only name. Whether a job container's `actions/checkout` derives its clone URL from Forgejo's configured public `ROOT_URL` or from the runner's registration URL decides the outcome: - If `ROOT_URL`: clearing the network is safe, outbound NAT handles it. - If registration URL: job containers on an auto-created network cannot resolve `forgejo`, and **checkout breaks for every repo on the instance.** Verify before or during the canary. The failure mode is total, not partial — watch the checkout step, not just `actions/cache`. Whichever way it resolves, add a comment to `config.yml` recording the decision. Its absence is why this took a git-archaeology session to reconstruct.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

deadstyle/forgo#1

No description provided.

Rows
Columns