Optimize CI/CD pipeline setup, checks, and release builds #64

New issue

deadstyle · 2026-10-08T15:30:17-05:00

deadstyle commented

2026-10-08 15:30:17 -05:00

Problem

Reduce CI feedback time and release pipeline cost without weakening checks, merge gates, or publishing safeguards.

Recent Forgejo CI tasks provide an initial reference: #695 took 1m42s, #693 1m43s, #699 2m24s, and #700 2m35s. These are total task durations, not a measured breakdown or proof of a particular bottleneck. Establish a repeatable baseline before choosing optimizations.

Current behavior and opportunities

.forgejo/workflows/ci.yml and release.yml each install golangci-lint from source with go install and download pinned ShellCheck on every run. Inspect effective cache hits and setup costs; there is no explicit workflow cache for these tool binaries.
Both use actions/setup-go and make check. Audit the existing setup-go cache before adding another cache for Go modules or build output.
make check runs format, lint, race tests, vulnerability scanning, build, and installer checks serially. Measure stage times and evaluate parallel jobs against runner capacity, repeated setup costs, and cache contention.
scripts/test-install.sh runs make release, cross-compiling all four OS/architecture targets. The tag workflow subsequently runs make release again with the real tag version. Remove avoidable duplication while retaining coverage of archive names, platform contents, checksums, corruption handling, and install behavior. Do not publish test-version artifacts as real releases.
CI runs for PRs and pushes to develop/main, as well as manual dispatch for imported GitHub branches. Evaluate cancellation of superseded CI runs without cancelling required branch checks, release publication, or unrelated refs.
GitHub import/sync workflows fetch full history and poll every ten minutes. Measure this overhead separately; optimize only where evidence supports it, preserving import and commit-status behavior.

Scope and implementation approach

Record setup, queue, and individual check durations for representative warm-cache and cold-cache runs; identify the actual dominant costs.
Implement the highest-value measured optimizations: reusable pinned tool installations, effective caches, avoiding duplicate artifact builds, and appropriate parallelism/concurrency. Choose the implementation after verifying support on this Forgejo instance and its runners.
Keep local make check and CI requirements consistent. Preserve the required ci / check status or an equivalent aggregate gate that succeeds only when all required jobs pass; preserve dispatched-run status copying in github-pull.yml.
Document the resulting workflow, cache keys/invalidation, failure behavior, and before/after measurements.

Acceptance criteria

Publish before/after timings for equivalent workloads, separating cold/warm cache and queue time from execution time. Show a measurable improvement in CI wall time or runner work, with an explanation of tradeoffs; do not claim speedups from unlike runs.
Format checks, lint, full race tests, vulnerability scanning, build verification, ShellCheck, and installer behavior coverage remain enforced. A failed required job cannot yield a successful aggregate merge status.
Cache/tool reuse remains reproducible, pinned, and invalidates when the relevant toolchain, tool versions, dependencies, or configuration change. Cache misses still succeed and untrusted PRs cannot poison trusted tool/release artifacts.
Any reduction in cross-compilation preserves verification of all four supported platforms and checksums. Published archives carry the requested release version and correspond to the validated source commit.
CI supersession, if added, is scoped to the correct ref/PR. Manual-dispatch CI and imported-branch status propagation still work, including failure/cancellation reporting.
Release publication retains the main-ancestry gate and the draft-until-all-assets-uploaded safeguard. Validate failure paths without publishing an unsolicited live release.
Demonstrate at least one passing PR run and an injected failing required check; validate the release build/publish flow with a dry run or fixture. Update docs and any scripts/tests affected.

Out of scope

Application feature changes, dropping quality checks to gain speed, changing release policy, or expanding the GitHub synchronization behavior. Runner/infrastructure changes that require separate operational work should be documented or split into a follow-up ticket.

## Problem Reduce CI feedback time and release pipeline cost without weakening checks, merge gates, or publishing safeguards. Recent Forgejo CI tasks provide an initial reference: #695 took 1m42s, #693 1m43s, #699 2m24s, and #700 2m35s. These are total task durations, not a measured breakdown or proof of a particular bottleneck. Establish a repeatable baseline before choosing optimizations. ## Current behavior and opportunities - `.forgejo/workflows/ci.yml` and `release.yml` each install golangci-lint from source with `go install` and download pinned ShellCheck on every run. Inspect effective cache hits and setup costs; there is no explicit workflow cache for these tool binaries. - Both use `actions/setup-go` and `make check`. Audit the existing setup-go cache before adding another cache for Go modules or build output. - `make check` runs format, lint, race tests, vulnerability scanning, build, and installer checks serially. Measure stage times and evaluate parallel jobs against runner capacity, repeated setup costs, and cache contention. - `scripts/test-install.sh` runs `make release`, cross-compiling all four OS/architecture targets. The tag workflow subsequently runs `make release` again with the real tag version. Remove avoidable duplication while retaining coverage of archive names, platform contents, checksums, corruption handling, and install behavior. Do not publish test-version artifacts as real releases. - CI runs for PRs and pushes to develop/main, as well as manual dispatch for imported GitHub branches. Evaluate cancellation of superseded CI runs without cancelling required branch checks, release publication, or unrelated refs. - GitHub import/sync workflows fetch full history and poll every ten minutes. Measure this overhead separately; optimize only where evidence supports it, preserving import and commit-status behavior. ## Scope and implementation approach 1. Record setup, queue, and individual check durations for representative warm-cache and cold-cache runs; identify the actual dominant costs. 2. Implement the highest-value measured optimizations: reusable pinned tool installations, effective caches, avoiding duplicate artifact builds, and appropriate parallelism/concurrency. Choose the implementation after verifying support on this Forgejo instance and its runners. 3. Keep local `make check` and CI requirements consistent. Preserve the required `ci / check` status or an equivalent aggregate gate that succeeds only when all required jobs pass; preserve dispatched-run status copying in `github-pull.yml`. 4. Document the resulting workflow, cache keys/invalidation, failure behavior, and before/after measurements. ## Acceptance criteria - Publish before/after timings for equivalent workloads, separating cold/warm cache and queue time from execution time. Show a measurable improvement in CI wall time or runner work, with an explanation of tradeoffs; do not claim speedups from unlike runs. - Format checks, lint, full race tests, vulnerability scanning, build verification, ShellCheck, and installer behavior coverage remain enforced. A failed required job cannot yield a successful aggregate merge status. - Cache/tool reuse remains reproducible, pinned, and invalidates when the relevant toolchain, tool versions, dependencies, or configuration change. Cache misses still succeed and untrusted PRs cannot poison trusted tool/release artifacts. - Any reduction in cross-compilation preserves verification of all four supported platforms and checksums. Published archives carry the requested release version and correspond to the validated source commit. - CI supersession, if added, is scoped to the correct ref/PR. Manual-dispatch CI and imported-branch status propagation still work, including failure/cancellation reporting. - Release publication retains the main-ancestry gate and the draft-until-all-assets-uploaded safeguard. Validate failure paths without publishing an unsolicited live release. - Demonstrate at least one passing PR run and an injected failing required check; validate the release build/publish flow with a dry run or fixture. Update docs and any scripts/tests affected. ## Out of scope Application feature changes, dropping quality checks to gain speed, changing release policy, or expanding the GitHub synchronization behavior. Runner/infrastructure changes that require separate operational work should be documented or split into a follow-up ticket.

deadstyle added the

opus

tech-debt

labels

2026-10-08 15:30:29 -05:00

deadstyle referenced this issue from a commit

2026-10-08 21:54:30 -05:00

chore(ci): cache Go and pinned tools, stop re-saving the build cache

deadstyle referenced this issue from a commit

2026-10-08 21:54:30 -05:00

test(ci): TEMP let dispatched runs save caches for measurement

deadstyle referenced this issue from a commit