- Makefile 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
All checks were successful
YAML Lint and Fix / yaml-lint-fix (push) Successful in 17s
Records the root cause behind the "all predefined address pools have been fully subnetted" runner errors on cachy-mira (leaked per-job networks from runner restarts plus no default-address-pools override), the fix already applied there, and the systemd timer now pruning orphaned networks weekly. Also documents the two-machine layout for working with this repo (Forgejo host vs offloaded build-runner host). |
||
| .forgejo/workflows | ||
| runner-config | ||
| runner-images | ||
| .env.example | ||
| .env.runner.example | ||
| .gitignore | ||
| .prettierrc | ||
| .yamllint | ||
| CLAUDE.md | ||
| docker-compose.runner.yml | ||
| docker-compose.yml | ||
| Makefile | ||
| README.md | ||
| RUNNER-SETUP.md | ||
Forgejo Docker Setup
Self-hosted Git service using Forgejo with Traefik reverse proxy and PostgreSQL database.
Prerequisites
- Docker and Docker Compose installed
- Traefik reverse proxy running with external
traefiknetwork - DNS record pointing to your server
- Borg backup configured for Docker volume backups
Quick Start
-
Create environment file:
cp .env.example .env -
Configure environment variables: Edit
.envand set:DB_PASSWORD- Strong database passwordFORGEJO_DOMAIN- Your domain (e.g.,git.example.com)FORGEJO_ROOT_URL- Full URL (e.g.,https://git.example.com)SSH_PORT- SSH port for Git operations (default: 2222)- Leave
GITEA_RUNNER_REGISTRATION_TOKENempty for now (see Runner Setup below)
-
Ensure Traefik network exists:
docker network create traefik -
Start services (without runner initially):
# Start only Forgejo and database first docker compose up -d forgejo db -
Complete initial setup:
- Navigate to your domain and complete the setup wizard
- Create an admin account
- Enable Actions in Site Administration > Actions (if not already enabled)
-
Configure the runner (after initial setup):
- In Forgejo web UI, go to: Admin Settings > Actions > Runners
- Click "Create new Runner" and copy the registration token
- Add the token to
.envasGITEA_RUNNER_REGISTRATION_TOKEN - Start the runner:
docker compose up -d runner
SSH Configuration
Forgejo's SSH service is exposed on port 2222 (configurable via SSH_PORT in .env).
Clone via SSH:
git clone ssh://git@git.example.com:2222/username/repo.git
Configure SSH client:
Add to ~/.ssh/config:
Host git.example.com
Port 2222
User git
Then clone normally:
git clone git@git.example.com:username/repo.git
Initial Setup
- Navigate to your Forgejo domain
- Complete the installation wizard
- Create an admin account
- Configure any additional settings
Management
View logs:
docker compose logs -f forgejo
Restart services:
docker compose restart
Stop services:
docker compose down
Backup:
This service uses Docker named volumes for data persistence. Backups are handled via Borg, which is configured to backup Docker volumes.
The following volumes are backed up:
forgejo_data- Forgejo repositories and configurationpostgres_data- PostgreSQL databaserunner_data- Runner configuration and cache
Manual backup commands (if needed):
# Backup database to SQL file
docker compose exec db pg_dump -U forgejo forgejo > backup.sql
# Backup Forgejo data volume
docker run --rm \
-v forgejo_data:/data \
-v $(pwd):/backup \
alpine tar czf /backup/forgejo-data-$(date +%Y%m%d).tar.gz /data
Forgejo Actions Runner
The runner is included by default for CI/CD workflows. It must be configured after initial Forgejo setup.
Runner Setup Process
-
Complete Forgejo initial setup (see Quick Start above)
-
Generate registration token:
- Login as admin
- Navigate to: Site Administration > Actions > Runners
- Click "Create new Runner"
- Copy the registration token
-
Configure runner:
# Edit .env and add the token nano .env # Set: GITEA_RUNNER_REGISTRATION_TOKEN=your_token_here -
Start the runner:
docker compose up -d runner -
Verify runner is connected:
- Go back to Admin Settings > Actions > Runners
- You should see "main-runner" listed as online
Organization-Scoped Runner
This repo also supports an organization-scoped runner for the Kirria organization without changing workflow labels in that org's repositories.
The kirria-org-runner service is intentionally separate from the existing user-level deployment runners:
- Separate container:
forgejo-kirria-org-runner - Separate persistent volume:
kirria_org_runner_data - Separate token variable:
KIRRIA_ORG_RUNNER_REGISTRATION_TOKEN - Same
deploylabel mapping, so existingruns-on: deployworkflows continue to match
Manual Setup On The Server
-
Generate an organization runner token in Forgejo:
- Go to
Kirriaorganization >Settings>Actions>Runners - Click
Create new Runner - Copy the registration token
- Go to
-
Place the token in the server's
.envfile:KIRRIA_ORG_RUNNER_REGISTRATION_TOKEN=your_kirria_org_runner_token_here -
Start only the org-scoped runner:
docker compose up -d kirria-org-runner -
Verify registration:
- In Forgejo UI:
Kirriaorganization >Settings>Actions>Runners - Confirm
kirria-org-runnerappears as online - Confirm it shows the
deploylabel
- In Forgejo UI:
-
Optional log check:
docker compose logs -f kirria-org-runner
Assumptions
- Compose project name remains the repo default, so the internal network name for service-to-service access is still the Compose-managed network while the service joins the existing
forgejo-internalnetwork declared indocker-compose.yml - The Forgejo service remains reachable from runners at
http://forgejo:3000on that network - The
deploylabel format continues to bedeploy:docker://git.bobparsons.dev/deadstyle/runner-images:deploy, which matchesruns-on: deploy
Testing the Runner
To verify your runner is working correctly, create a test repository and add a simple workflow:
-
Create a new repository in Forgejo (or use an existing one)
-
Create the workflow file
.forgejo/workflows/test.yml:name: Test Runner on: [push] jobs: test: runs-on: ubuntu-latest steps: - name: Hello World run: echo "Runner is working!" - name: Show system info run: | echo "Running on: $(uname -a)" echo "Docker available: $(docker --version)" -
Commit and push the workflow file
-
Check the Actions tab in your repository to see the workflow run
If the workflow executes successfully and shows "Runner is working!", your runner is configured correctly!
Customizing Runner Labels
Edit GITEA_RUNNER_LABELS in .env to support different environments:
# Default: Ubuntu with Node.js 20 and Docker-in-Docker
GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:20-bookworm,docker:docker://docker:dind
# Add more environments:
GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:20-bookworm,node-18:docker://node:18-alpine,docker:docker://docker:dind
Using Actions in Repositories
Create .forgejo/workflows/ or .gitea/workflows/ in your repository with workflow YAML files:
name: CI
on: [push, pull_request]
jobs:
test:
runs-on: deploy # Uses deployment runners on the server
steps:
- uses: actions/checkout@v4
- name: Run tests
run: npm test
Remote Runners
Runners can be deployed on any machine that has network access to your Forgejo instance. This allows you to:
- Offload CI/CD work from your production server
- Use spare/older hardware for builds and tests
- Scale horizontally by adding more runner machines
- Separate CI tasks from deployment tasks using different labels
📦 Quick Setup: For a simplified, reusable runner deployment, see RUNNER-SETUP.md. This guide provides a standalone Docker Compose configuration that can be easily deployed across multiple machines.
Remote Runner Setup (e.g., Desktop/Workstation)
Follow these steps to set up a runner on a remote machine:
1. Install Docker
On your remote machine (Ubuntu/Debian):
# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
# Add your user to docker group (optional - allows non-root docker)
sudo usermod -aG docker $USER
newgrp docker
2. Create Runner Directory
mkdir -p ~/forgejo-runner
cd ~/forgejo-runner
3. Get Registration Token
On your Forgejo server:
- Login as admin
- Go to: Site Administration > Actions > Runners
- Click "Create new Runner"
- Copy the registration token
Note: The same token can be used for multiple runners - each will register as a separate instance.
4. Create Docker Compose File
Create docker-compose.yml on your remote machine:
networks:
runner-network:
driver: bridge
volumes:
runner_data:
docker_certs:
services:
docker-dind:
image: docker:dind
container_name: ci-runner-docker-dind
restart: unless-stopped
privileged: true
networks:
- runner-network
command: ['dockerd', '-H', 'tcp://0.0.0.0:2375', '--tls=false']
volumes:
- docker_certs:/certs
ci-runner:
image: data.forgejo.org/forgejo/runner:11
container_name: ci-runner
restart: unless-stopped
networks:
- runner-network
environment:
# Replace with your Forgejo domain
- GITEA_INSTANCE_URL=https://git.bobparsons.dev
- GITEA_RUNNER_REGISTRATION_TOKEN=YOUR_TOKEN_HERE
- GITEA_RUNNER_NAME=ci-runner
# Use "ci" label to differentiate from deployment runners
- GITEA_RUNNER_LABELS=ci:docker://node:20-bookworm
- DOCKER_HOST=tcp://docker-dind:2375
volumes:
- runner_data:/data
command: >
sh -c "if [ ! -f /data/.runner ]; then forgejo-runner register --no-interactive --instance $${GITEA_INSTANCE_URL} --token $${GITEA_RUNNER_REGISTRATION_TOKEN} --name $${GITEA_RUNNER_NAME} --labels $${GITEA_RUNNER_LABELS}; fi && forgejo-runner daemon"
depends_on:
- docker-dind
5. Configure Runner Capacity (Optional)
For high-capacity machines (like a 14900K with 64GB RAM), increase job capacity:
# Generate config file
docker compose run --rm ci-runner forgejo-runner generate-config > config.yml
# Edit config.yml and change:
# capacity: 1 -> capacity: 6 (or whatever suits your hardware)
# Mount config in docker-compose.yml:
volumes:
- runner_data:/data
- ./config.yml:/data/config.yml:ro
# Update command:
command: >
sh -c "if [ ! -f /data/.runner ]; then forgejo-runner register --no-interactive --instance $${GITEA_INSTANCE_URL} --token $${GITEA_RUNNER_REGISTRATION_TOKEN} --name $${GITEA_RUNNER_NAME} --labels $${GITEA_RUNNER_LABELS}; fi && forgejo-runner daemon -c /data/config.yml"
6. Start the Runner
# Replace YOUR_TOKEN_HERE in docker-compose.yml first!
docker compose up -d
# Check logs
docker compose logs -f ci-runner
7. Verify Registration
In Forgejo UI (Site Administration > Actions > Runners), you should see your remote runner listed as online.
Using Different Runners in Workflows
Use the runs-on field to target specific runners by label:
name: Full CI/CD Pipeline
on: [push]
jobs:
# Runs on remote CI runner (fast desktop/workstation)
lint:
runs-on: ci
steps:
- uses: actions/checkout@v4
- name: Lint code
run: npm run lint
test:
runs-on: ci
needs: lint
steps:
- uses: actions/checkout@v4
- name: Run tests
run: npm test
build:
runs-on: ci
needs: test
steps:
- uses: actions/checkout@v4
- name: Build Docker image
run: docker build -t myapp:${{ github.sha }} .
# Runs on server deployment runners
deploy:
runs-on: deploy
needs: build
steps:
- name: Deploy to production
run: ./deploy.sh
Runner Label Strategy
Recommended setup:
- Remote machines (CI): Use label
cifor linting, testing, building - Server (Deployment): Use label
deployfor production deployments - Specialized: Add labels like
ci-python,build-heavy, etc. as needed
Managing Remote Runners
# View runner logs
docker compose logs -f ci-runner
# Restart runner
docker compose restart ci-runner
# Stop runner
docker compose down
# Update runner
docker compose pull
docker compose up -d
Architecture
- Forgejo: Git service (port 3000 internal)
- PostgreSQL: Database backend
- Forgejo Runner: CI/CD runner with Docker-in-Docker support
- Traefik: Reverse proxy with automatic HTTPS
- Networks:
traefik(external): Routes traffic from Traefikforgejo-internal: Private network for Forgejo ↔ DB ↔ Runner communication
Working With This Repo Across Machines
This repo is checked out in two different places, each with a different Docker daemon and a different role:
| Machine | Runs | Compose file | Docker socket |
|---|---|---|---|
Forgejo host (ssh server@server) |
forgejo, db, and the four on-host runners (deploy-runner-1/2, kirria-org-runner, kirria-org-ci-runner) |
docker-compose.yml |
Production socket — deploy jobs reach host services through it |
Workstation (ssh bobparsons@cachy-mira) |
Offloaded build runners (build-runner-1/2/3, runs-on: ubuntu-latest) |
docker-compose.runner.yml (see RUNNER-SETUP.md) |
Shared with other local dev projects on that box (kirria, jellyfin) |
Don't assume a checkout on some other local machine is the runner host just
because the repo files (docker-compose.runner.yml, .env.runner, the
build-runners-* Makefile targets) are present there — check for actual
running build-runner-* containers (docker ps -a --filter name=build-runner)
before treating a box as "the workstation."
Both .env (on the Forgejo host) and .env.runner (on a workstation) hold
real secrets and are gitignored — on machines set up like this one they're
symlinked in from a separate dotenv store rather than created by hand.
Useful commands from either checkout:
docker compose ps # Forgejo host: service status
make build-runners-status # Workstation: build-runner status
make build-runners-logs # Workstation: follow build-runner-1
Troubleshooting
Check service health:
docker compose ps
Database connection issues:
docker compose logs db
Traefik routing issues:
docker compose logs forgejo
# Check Traefik dashboard at traefik.bobparsons.dev
Job setup fails: "all predefined address pools have been fully subnetted"
Seen on cachy-mira (the offloaded build-runner workstation), not the Forgejo host. Rerunning the job usually succeeds, which is the tell.
Root cause has two parts, and both need fixing or it comes back:
-
Leaked per-job networks.
runner-config/config.remote.ymlsetsnetwork: "", so every job gets a brand-new Docker network, cleaned up when the job finishes ("Cleaning up network for job ..."in the runner log). But if abuild-runner-*container restarts while jobs are in-flight, it loses track of those jobs' cleanup step and their networks are never removed. Confirmed on 2026-07-30: 22 of 23WORKFLOW-*networks had zero containers attached — pure orphans, all dated to the runners' last restart. Check for these with:docker network ls -q --filter name=WORKFLOW | while read -r n; do c=$(docker network inspect "$n" --format '{{len .Containers}}') [ "$c" = "0" ] && docker network inspect "$n" --format '{{.Name}}' donedocker network prune -fis safe to run any time — it refuses to touch a network with containers still attached, so it can't hit a live job. -
No
default-address-poolsoverride. Without one, Docker falls back to its built-in defaults:172.17.0.0/16–172.31.0.0/16(15/16networks), and once those are gone,192.168.0.0/16split into/20s (16 more) — a hard ceiling around 31 networks total. cachy-mira hit exactly 31 before this was fixed, so leaked networks alone were enough to exhaust it. The Forgejo host never hit this because itsdaemon.jsonalready defines a much larger pool of small subnets:{ "default-address-pools": [ { "base": "10.10.0.0/16", "size": 24 }, { "base": "10.11.0.0/16", "size": 24 } ] }That's 512
/24networks instead of ~31. cachy-mira now has the same fix with a non-overlapping range (10.30.0.0/16/10.31.0.0/16), applied 2026-07-30.
Fix, in order: prune orphaned networks first, then add
default-address-pools to the affected host's /etc/docker/daemon.json
(fish doesn't support heredocs — use printf '%s\n' '{' '...' '}' | sudo tee /etc/docker/daemon.json), then sudo systemctl restart docker. Restarting
Docker bounces every container on that host, so time it around anything
in-flight. Existing networks keep their current subnets — only newly created
ones pick up the new pool.
Standing mitigation: since the leak comes back on every runner restart, a
systemd timer on cachy-mira runs docker network prune -f weekly (Sundays
03:00, Persistent=true so a missed run catches up on next boot). It's
system-level, not a user crontab — this host has no cron/crontab at all,
systemd timers are the native mechanism here.
# /etc/systemd/system/docker-network-prune.service
[Unit]
Description=Prune orphaned Docker networks
[Service]
Type=oneshot
ExecStart=/usr/bin/docker network prune -f
# /etc/systemd/system/docker-network-prune.timer
[Unit]
Description=Weekly Docker network prune
[Timer]
OnCalendar=Sun *-*-* 03:00:00
Persistent=true
[Install]
WantedBy=timers.target
Installed with sudo systemctl daemon-reload && sudo systemctl enable --now docker-network-prune.timer. Check status with:
systemctl list-timers docker-network-prune.timer
journalctl -u docker-network-prune.service # last run output
Diagnose on either host with:
docker network ls
for n in $(docker network ls -q); do
docker network inspect "$n" --format '{{.Name}}: {{.IPAM.Config}}'
done