No description
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Bob Parsons e6f56290b2
All checks were successful
YAML Lint and Fix / yaml-lint-fix (push) Successful in 13s
feat(renovate): optional repo input on workflow_dispatch
Reviewed-on: ⁨#18⁩
2026-10-10 12:03:14 -05:00
.forgejo/workflows feat(renovate): optional repo input on workflow_dispatch 2026-10-10 12:03:14 -05:00
renovate Rename deadstyle/forgo to deadstyle/forgejo; document config merge order 2026-09-28 19:31:15 -05:00
runner-config runner: allow /var/cache/headrust-ci volume instead of /tmp 2026-09-12 09:18:53 -05:00
runner-images chore(deps): update dependency node to v24 2026-09-28 01:51:15 +00:00
.env.example feat: sign merge commits with an instance GPG key 2026-07-22 22:09:20 -05:00
.env.runner.example feat: add third build runner (kirria-scoped) 2026-07-22 21:04:11 -05:00
.gitignore add runner and workflow for linting 2025-12-15 21:23:12 -06:00
.prettierrc add prettier 2025-12-15 21:27:19 -06:00
.yamllint add runner and workflow for linting 2025-12-15 21:23:12 -06:00
CLAUDE.md postgres 18: mount volume at /var/lib/postgresql 2026-09-28 19:57:29 -05:00
docker-compose.runner.yml chore(deps): update data.forgejo.org/forgejo/runner docker tag to v13 2026-09-28 01:51:10 +00:00
docker-compose.yml Enable attachments and allow html uploads 2026-10-03 18:21:50 -05:00
Makefile chore: align cache config templates and document proxy port allocation 2026-07-22 21:50:24 -05:00
README.md docs: document all three runner hosts and per-host upgrade steps 2026-09-28 19:57:29 -05:00
renovate.json Rename deadstyle/forgo to deadstyle/forgejo; document config merge order 2026-09-28 19:31:15 -05:00
RENOVATE.md Rename deadstyle/forgo to deadstyle/forgejo; document config merge order 2026-09-28 19:31:15 -05:00
RUNNER-SETUP.md docs: document all three runner hosts and per-host upgrade steps 2026-09-28 19:57:29 -05:00

Forgejo Docker Setup

Self-hosted Git service using Forgejo with Traefik reverse proxy and PostgreSQL database.

Prerequisites

  • Docker and Docker Compose installed
  • Traefik reverse proxy running with external traefik network
  • DNS record pointing to your server
  • Borg backup configured for Docker volume backups

Quick Start

  1. Create environment file:

    cp .env.example .env
    
  2. Configure environment variables: Edit .env and set:

    • DB_PASSWORD - Strong database password
    • FORGEJO_DOMAIN - Your domain (e.g., git.example.com)
    • FORGEJO_ROOT_URL - Full URL (e.g., https://git.example.com)
    • SSH_PORT - SSH port for Git operations (default: 2222)
    • Leave GITEA_RUNNER_REGISTRATION_TOKEN empty for now (see Runner Setup below)
  3. Ensure Traefik network exists:

    docker network create traefik
    
  4. Start services (without runner initially):

    # Start only Forgejo and database first
    docker compose up -d forgejo db
    
  5. Complete initial setup:

    • Navigate to your domain and complete the setup wizard
    • Create an admin account
    • Enable Actions in Site Administration > Actions (if not already enabled)
  6. Configure the runner (after initial setup):

    • In Forgejo web UI, go to: Admin Settings > Actions > Runners
    • Click "Create new Runner" and copy the registration token
    • Add the token to .env as GITEA_RUNNER_REGISTRATION_TOKEN
    • Start the runner: docker compose up -d runner

SSH Configuration

Forgejo's SSH service is exposed on port 2222 (configurable via SSH_PORT in .env).

Clone via SSH:

git clone ssh://git@git.example.com:2222/username/repo.git

Configure SSH client:

Add to ~/.ssh/config:

Host git.example.com
    Port 2222
    User git

Then clone normally:

git clone git@git.example.com:username/repo.git

Initial Setup

  1. Navigate to your Forgejo domain
  2. Complete the installation wizard
  3. Create an admin account
  4. Configure any additional settings

Management

View logs:

docker compose logs -f forgejo

Restart services:

docker compose restart

Stop services:

docker compose down

Backup:

This service uses Docker named volumes for data persistence. Backups are handled via Borg, which is configured to backup Docker volumes.

The following volumes are backed up:

  • forgejo_data - Forgejo repositories and configuration
  • postgres_data - PostgreSQL database
  • runner_data - Runner configuration and cache

Manual backup commands (if needed):

# Backup database to SQL file
docker compose exec db pg_dump -U forgejo forgejo > backup.sql

# Backup Forgejo data volume
docker run --rm \
  -v forgejo_data:/data \
  -v $(pwd):/backup \
  alpine tar czf /backup/forgejo-data-$(date +%Y%m%d).tar.gz /data

Forgejo Actions Runner

The runner is included by default for CI/CD workflows. It must be configured after initial Forgejo setup.

Runner Setup Process

  1. Complete Forgejo initial setup (see Quick Start above)

  2. Generate registration token:

    • Login as admin
    • Navigate to: Site Administration > Actions > Runners
    • Click "Create new Runner"
    • Copy the registration token
  3. Configure runner:

    # Edit .env and add the token
    nano .env
    # Set: GITEA_RUNNER_REGISTRATION_TOKEN=your_token_here
    
  4. Start the runner:

    docker compose up -d runner
    
  5. Verify runner is connected:

    • Go back to Admin Settings > Actions > Runners
    • You should see "main-runner" listed as online

Organization-Scoped Runner

This repo also supports an organization-scoped runner for the Kirria organization without changing workflow labels in that org's repositories.

The kirria-org-runner service is intentionally separate from the existing user-level deployment runners:

  • Separate container: forgejo-kirria-org-runner
  • Separate persistent volume: kirria_org_runner_data
  • Separate token variable: KIRRIA_ORG_RUNNER_REGISTRATION_TOKEN
  • Same deploy label mapping, so existing runs-on: deploy workflows continue to match

Manual Setup On The Server

  1. Generate an organization runner token in Forgejo:

    • Go to Kirria organization > Settings > Actions > Runners
    • Click Create new Runner
    • Copy the registration token
  2. Place the token in the server's .env file:

    KIRRIA_ORG_RUNNER_REGISTRATION_TOKEN=your_kirria_org_runner_token_here
    
  3. Start only the org-scoped runner:

    docker compose up -d kirria-org-runner
    
  4. Verify registration:

    • In Forgejo UI: Kirria organization > Settings > Actions > Runners
    • Confirm kirria-org-runner appears as online
    • Confirm it shows the deploy label
  5. Optional log check:

    docker compose logs -f kirria-org-runner
    

Assumptions

  • Compose project name remains the repo default, so the internal network name for service-to-service access is still the Compose-managed network while the service joins the existing forgejo-internal network declared in docker-compose.yml
  • The Forgejo service remains reachable from runners at http://forgejo:3000 on that network
  • The deploy label format continues to be deploy:docker://git.bobparsons.dev/deadstyle/runner-images:deploy, which matches runs-on: deploy

Testing the Runner

To verify your runner is working correctly, create a test repository and add a simple workflow:

  1. Create a new repository in Forgejo (or use an existing one)

  2. Create the workflow file .forgejo/workflows/test.yml:

    name: Test Runner
    on: [push]
    
    jobs:
      test:
        runs-on: ubuntu-latest
        steps:
          - name: Hello World
            run: echo "Runner is working!"
          - name: Show system info
            run: |
              echo "Running on: $(uname -a)"
              echo "Docker available: $(docker --version)"
    
  3. Commit and push the workflow file

  4. Check the Actions tab in your repository to see the workflow run

If the workflow executes successfully and shows "Runner is working!", your runner is configured correctly!

Customizing Runner Labels

Edit GITEA_RUNNER_LABELS in .env to support different environments:

# Default: Ubuntu with Node.js 20 and Docker-in-Docker
GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:20-bookworm,docker:docker://docker:dind

# Add more environments:
GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:20-bookworm,node-18:docker://node:18-alpine,docker:docker://docker:dind

Using Actions in Repositories

Create .forgejo/workflows/ or .gitea/workflows/ in your repository with workflow YAML files:

name: CI
on: [push, pull_request]

jobs:
  test:
    runs-on: deploy  # Uses deployment runners on the server
    steps:
      - uses: actions/checkout@v4
      - name: Run tests
        run: npm test

Renovate

Dependency updates for every repo on this instance come from one self-hosted Renovate job, .forgejo/workflows/renovate.yml. It runs nightly (08:00 UTC) and on demand (Actions → Renovate → Run workflow) on the dedicated renovate-runner (label renovate), so it never takes a deploy or CI slot.

The policy lives in renovate/default.json:

  • Non-major updates (minor, patch, digest, pin) are grouped into one MR per ecosystem (npm, Go, Docker images, CI tools, Rust, Python, other), opened or refreshed only in the weekly window (Mondays before 06:00 America/Chicago).
  • Each major gets its own MR on any night, with no Dependency Dashboard approval needed.
  • The Dependency Dashboard issue is on, but only as information.
  • Automerge is off everywhere. Every MR is labelled maintenance.

One-time setup

  1. Create a renovate-bot user with email renovate-bot@bobparsons.dev, which matches RENOVATE_GIT_AUTHOR in the workflow.
  2. As that user, create an access token with these scopes: repository read/write, issue read/write, user read, organization read.
  3. Add it to this repo as the Actions secret RENOVATE_TOKEN.
  4. Optional: add a read-only (no scopes) github.com classic token as RENOVATE_GITHUB_COM_TOKEN. Without it, MRs have no upstream release notes and github.com lookups get rate limited.
  5. Start the runner on the server. It registers with GITEA_RUNNER_REGISTRATION_TOKEN, the deadstyle user's token, so it only serves deadstyle's repos, which includes this one: docker compose up -d renovate-runner

Opting a repo in

See RENOVATE.md. It covers the rules (no per-repo workflow or token), the renovate.json to add, labels, bot access, and how to trigger and verify a run.

Changes to the workflow or the preset go live on push. Only the renovate-runner service itself needs the server pull and docker compose up -d step.

Remote Runners

Runners can be deployed on any machine that has network access to your Forgejo instance. This allows you to:

  • Offload CI/CD work from your production server
  • Use spare/older hardware for builds and tests
  • Scale horizontally by adding more runner machines
  • Separate CI tasks from deployment tasks using different labels

📦 Quick Setup: For a simplified, reusable runner deployment, see RUNNER-SETUP.md. This guide provides a standalone Docker Compose configuration that can be easily deployed across multiple machines.

Remote Runner Setup (e.g., Desktop/Workstation)

Follow these steps to set up a runner on a remote machine:

1. Install Docker

On your remote machine (Ubuntu/Debian):

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Add your user to docker group (optional - allows non-root docker)
sudo usermod -aG docker $USER
newgrp docker

2. Create Runner Directory

mkdir -p ~/forgejo-runner
cd ~/forgejo-runner

3. Get Registration Token

On your Forgejo server:

  1. Login as admin
  2. Go to: Site Administration > Actions > Runners
  3. Click "Create new Runner"
  4. Copy the registration token

Note: The same token can be used for multiple runners - each will register as a separate instance.

4. Create Docker Compose File

Create docker-compose.yml on your remote machine:

networks:
  runner-network:
    driver: bridge

volumes:
  runner_data:
  docker_certs:

services:
  docker-dind:
    image: docker:dind
    container_name: ci-runner-docker-dind
    restart: unless-stopped
    privileged: true
    networks:
      - runner-network
    command: ['dockerd', '-H', 'tcp://0.0.0.0:2375', '--tls=false']
    volumes:
      - docker_certs:/certs

  ci-runner:
    image: data.forgejo.org/forgejo/runner:11
    container_name: ci-runner
    restart: unless-stopped
    networks:
      - runner-network
    environment:
      # Replace with your Forgejo domain
      - GITEA_INSTANCE_URL=https://git.bobparsons.dev
      - GITEA_RUNNER_REGISTRATION_TOKEN=YOUR_TOKEN_HERE
      - GITEA_RUNNER_NAME=ci-runner
      # Use "ci" label to differentiate from deployment runners
      - GITEA_RUNNER_LABELS=ci:docker://node:20-bookworm
      - DOCKER_HOST=tcp://docker-dind:2375
    volumes:
      - runner_data:/data
    command: >
      sh -c "if [ ! -f /data/.runner ]; then forgejo-runner register --no-interactive --instance $${GITEA_INSTANCE_URL} --token $${GITEA_RUNNER_REGISTRATION_TOKEN} --name $${GITEA_RUNNER_NAME} --labels $${GITEA_RUNNER_LABELS}; fi && forgejo-runner daemon"
    depends_on:
      - docker-dind

5. Configure Runner Capacity (Optional)

For high-capacity machines (like a 14900K with 64GB RAM), increase job capacity:

# Generate config file
docker compose run --rm ci-runner forgejo-runner generate-config > config.yml

# Edit config.yml and change:
# capacity: 1  ->  capacity: 6  (or whatever suits your hardware)

# Mount config in docker-compose.yml:
volumes:
  - runner_data:/data
  - ./config.yml:/data/config.yml:ro

# Update command:
command: >
  sh -c "if [ ! -f /data/.runner ]; then forgejo-runner register --no-interactive --instance $${GITEA_INSTANCE_URL} --token $${GITEA_RUNNER_REGISTRATION_TOKEN} --name $${GITEA_RUNNER_NAME} --labels $${GITEA_RUNNER_LABELS}; fi && forgejo-runner daemon -c /data/config.yml"

6. Start the Runner

# Replace YOUR_TOKEN_HERE in docker-compose.yml first!
docker compose up -d

# Check logs
docker compose logs -f ci-runner

7. Verify Registration

In Forgejo UI (Site Administration > Actions > Runners), you should see your remote runner listed as online.

Using Different Runners in Workflows

Use the runs-on field to target specific runners by label:

name: Full CI/CD Pipeline
on: [push]

jobs:
  # Runs on remote CI runner (fast desktop/workstation)
  lint:
    runs-on: ci
    steps:
      - uses: actions/checkout@v4
      - name: Lint code
        run: npm run lint

  test:
    runs-on: ci
    needs: lint
    steps:
      - uses: actions/checkout@v4
      - name: Run tests
        run: npm test

  build:
    runs-on: ci
    needs: test
    steps:
      - uses: actions/checkout@v4
      - name: Build Docker image
        run: docker build -t myapp:${{ github.sha }} .

  # Runs on server deployment runners
  deploy:
    runs-on: deploy
    needs: build
    steps:
      - name: Deploy to production
        run: ./deploy.sh

Runner Label Strategy

Recommended setup:

  • Remote machines (CI): Use label ci for linting, testing, building
  • Server (Deployment): Use label deploy for production deployments
  • Specialized: Add labels like ci-python, build-heavy, etc. as needed

Managing Remote Runners

# View runner logs
docker compose logs -f ci-runner

# Restart runner
docker compose restart ci-runner

# Stop runner
docker compose down

# Update runner
docker compose pull
docker compose up -d

Architecture

  • Forgejo: Git service (port 3000 internal)
  • PostgreSQL: Database backend
  • Forgejo Runner: CI/CD runner with Docker-in-Docker support
  • Traefik: Reverse proxy with automatic HTTPS
  • Networks:
    • traefik (external): Routes traffic from Traefik
    • forgejo-internal: Private network for Forgejo ↔ DB ↔ Runner communication

Working With This Repo Across Machines

Runners live on three different machines, each with a different role. The repo is checked out on the first two; the third is not managed from this repo:

Machine Runs Compose file Docker socket
Forgejo host (ssh server@server) forgejo, db, and the five on-host runners (deploy-runner-1/2, kirria-org-runner, kirria-org-ci-runner, renovate-runner) docker-compose.yml Production socket — deploy jobs reach host services through it
Workstation (ssh bobparsons@cachy-mira) Offloaded build runners (build-runner-1/2/3, runs-on: ubuntu-latest) docker-compose.runner.yml (see RUNNER-SETUP.md) Shared with other local dev projects on that box (kirria, jellyfin)
Kirria prod (ssh kirria, ubuntu@kirria-prod) One host-mode runner, label prod, as systemd unit forgejo-runner.service (binary /usr/local/bin/forgejo-runner, config /data/forgejo-runner/config.yml) None — not in this repo Host Docker socket, root-equivalent by design (see the unit file's comments)

Don't assume a checkout on some other local machine is the runner host just because the repo files (docker-compose.runner.yml, .env.runner, the build-runners-* Makefile targets) are present there — check for actual running build-runner-* containers (docker ps -a --filter name=build-runner) before treating a box as "the workstation."

Both .env (on the Forgejo host) and .env.runner (on a workstation) hold real secrets and are gitignored — on machines set up like this one they're symlinked in from a separate dotenv store rather than created by hand.

Upgrading runners (Renovate runner-version PRs)

Merging a runner-version PR updates the repo only. Each machine has to be updated separately:

Machine How Notes
Forgejo host git pull, then docker compose up -d deploy-runner-1 deploy-runner-2 kirria-org-runner kirria-org-ci-runner renovate-runner Do one runner first (e.g. deploy-runner-2) on major bumps and confirm it runs a job
cachy-mira git pull, then make build-runners-up Check for running job containers first; see RUNNER-SETUP.md
Kirria prod Download forgejo-runner-<ver>-linux-amd64 and its .sha256 from the runner release, verify with sha256sum -c, back up the old binary, install -m 755 over /usr/local/bin/forgejo-runner, then sudo systemctl restart forgejo-runner Not from this repo, so Renovate never proposes it and it can drift from the others (updated 13.0.0 → 13.2.0 on 2026-09-28). Host-mode jobs are plain processes, not containers, so check journalctl -u forgejo-runner for running work first. The unit has TimeoutStopSec=600, so a restart waits for running jobs to drain

Compare versions across all three after any runner bump; a host that was skipped keeps running the old version silently.

Useful commands from either checkout:

docker compose ps                          # Forgejo host: service status
make build-runners-status                  # Workstation: build-runner status
make build-runners-logs                    # Workstation: follow build-runner-1

Troubleshooting

Check service health:

docker compose ps

Database connection issues:

docker compose logs db

Traefik routing issues:

docker compose logs forgejo
# Check Traefik dashboard at traefik.bobparsons.dev

Job setup fails: "all predefined address pools have been fully subnetted"

Seen on cachy-mira (the offloaded build-runner workstation), not the Forgejo host. Rerunning the job usually succeeds, which is the tell.

Root cause has two parts, and both need fixing or it comes back:

  1. Leaked per-job networks. runner-config/config.remote.yml sets network: "", so every job gets a brand-new Docker network, cleaned up when the job finishes ("Cleaning up network for job ..." in the runner log). But if a build-runner-* container restarts while jobs are in-flight, it loses track of those jobs' cleanup step and their networks are never removed. Confirmed on 2026-07-30: 22 of 23 WORKFLOW-* networks had zero containers attached — pure orphans, all dated to the runners' last restart. Check for these with:

    docker network ls -q --filter name=WORKFLOW | while read -r n; do
      c=$(docker network inspect "$n" --format '{{len .Containers}}')
      [ "$c" = "0" ] && docker network inspect "$n" --format '{{.Name}}'
    done
    

    docker network prune -f is safe to run any time — it refuses to touch a network with containers still attached, so it can't hit a live job.

  2. No default-address-pools override. Without one, Docker falls back to its built-in defaults: 172.17.0.0/16–172.31.0.0/16 (15 /16 networks), and once those are gone, 192.168.0.0/16 split into /20s (16 more) — a hard ceiling around 31 networks total. cachy-mira hit exactly 31 before this was fixed, so leaked networks alone were enough to exhaust it. The Forgejo host never hit this because its daemon.json already defines a much larger pool of small subnets:

    {
      "default-address-pools": [
        { "base": "10.10.0.0/16", "size": 24 },
        { "base": "10.11.0.0/16", "size": 24 }
      ]
    }
    

    That's 512 /24 networks instead of ~31. cachy-mira now has the same fix with a non-overlapping range (10.30.0.0/16 / 10.31.0.0/16), applied 2026-07-30.

Fix, in order: prune orphaned networks first, then add default-address-pools to the affected host's /etc/docker/daemon.json (fish doesn't support heredocs — use printf '%s\n' '{' '...' '}' | sudo tee /etc/docker/daemon.json), then sudo systemctl restart docker. Restarting Docker bounces every container on that host, so time it around anything in-flight. Existing networks keep their current subnets — only newly created ones pick up the new pool.

Standing mitigation: since the leak comes back on every runner restart, a systemd timer on cachy-mira runs docker network prune -f weekly (Sundays 03:00, Persistent=true so a missed run catches up on next boot). It's system-level, not a user crontab — this host has no cron/crontab at all, systemd timers are the native mechanism here.

# /etc/systemd/system/docker-network-prune.service
[Unit]
Description=Prune orphaned Docker networks

[Service]
Type=oneshot
ExecStart=/usr/bin/docker network prune -f

# /etc/systemd/system/docker-network-prune.timer
[Unit]
Description=Weekly Docker network prune

[Timer]
OnCalendar=Sun *-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

Installed with sudo systemctl daemon-reload && sudo systemctl enable --now docker-network-prune.timer. Check status with:

systemctl list-timers docker-network-prune.timer
journalctl -u docker-network-prune.service   # last run output

Diagnose on either host with:

docker network ls
for n in $(docker network ls -q); do
  docker network inspect "$n" --format '{{.Name}}: {{.IPAM.Config}}'
done

Runner job containers leave anonymous volumes and stopped containers behind

Seen on both the Forgejo host and cachy-mira (the offloaded build-runner workstation). Rust/cargo caching in job containers (e.g. headrust-ci workflows) creates anonymous volumes that never get attached to anything persistent, and job containers that exit aren't removed either. Neither accumulates fast enough to page anyone, but nothing was pruning them, so they built up for months: confirmed 2026-09-12 at 431 of 438 volumes on the Forgejo host and 1,040 of 1,074 on cachy-mira were dangling, reclaiming 138GB locally and dropping the host's reclaimable volume space from 55.71GB to 2.07GB once pruned.

Check for the leak with:

docker volume ls -f dangling=true -q | wc -l
docker system df

Standing mitigation: a systemd timer on both hosts runs docker volume prune -f && docker container prune -f weekly (Sundays 03:15, 15 minutes after the network-prune timer above, Persistent=true). Same system-level pattern as docker-network-prune.timer, not a user crontab.

# /etc/systemd/system/docker-cleanup-prune.service
[Unit]
Description=Prune dangling Docker volumes and stopped containers

[Service]
Type=oneshot
ExecStart=/usr/bin/docker volume prune -f
ExecStart=/usr/bin/docker container prune -f

# /etc/systemd/system/docker-cleanup-prune.timer
[Unit]
Description=Weekly Docker volume/container prune

[Timer]
OnCalendar=Sun *-*-* 03:15:00
Persistent=true

[Install]
WantedBy=timers.target

Installed with sudo systemctl daemon-reload && sudo systemctl enable --now docker-cleanup-prune.timer. Check status with:

systemctl list-timers docker-cleanup-prune.timer
journalctl -u docker-cleanup-prune.service   # last run output

References