mirror of
https://gitea.com/gitea/runner.git
synced 2026-08-23 20:37:46 +00:00
silverwind
bc8161c673
Jobs occasionally go silent ([example](https://gitea.com/gitea/runner/actions/runs/805045/jobs/1055123)) mid-run and Gitea reaped them after `ZOMBIE_TASK_TIMEOUT`, with no error in the log. This contains a number of related fixes, all with full test coverage: 1. Bound every RPC to Gitea with a timeout, a stalled report otherwise parked logs and heartbeats for the whole job. 2. Cap `runner.fetch_timeout` at that ceiling. 3. Let only the daemon loop close its own channel, the race panicked the process. 4. Stop the job on any terminal server result, not just `RESULT_CANCELLED`. 5. Report that result instead of relabelling it as cancelled. 6. Log reporting failures once at each end of an outage instead of discarding them. 7. Clamp the acknowledged log index, a too-large ack panicked on a slice bound. 8. Stop reading server health from a `FetchTask` deadline, it marked the runner healthy and reset the error backoff on a timeout. 9. Return an error from the Docker version probe instead of a `logrus` panic. 10. Pass the context to go-git's fetch and pull. 11. Fail the clone when a refresh dies on a cancelled context. 12. Set `terminationGracePeriodSeconds` in the Kubernetes examples. Also contains a deprecation fix for goreleaser. Reviewed-on: https://gitea.com/gitea/runner/pulls/1174 Reviewed-by: bircni <bircni@icloud.com> Co-authored-by: silverwind <me@silverwind.io>
26 lines
2.5 KiB
Markdown
26 lines
2.5 KiB
Markdown
## Kubernetes Docker in Docker Deployment with `gitea-runner`
|
|
|
|
NOTE: Docker in Docker (dind) requires elevated privileges on Kubernetes. The current way to achieve this is to set the pod `SecurityContext` to `privileged`. Keep in mind that this is a potential security issue that has the potential for a malicious application to break out of the container context.
|
|
|
|
NOTE: `dind-docker.yaml` uses the native sidecar pattern (init container with `restartPolicy: Always`), which requires Kubernetes 1.29+ (or 1.28 with the `SidecarContainers` feature gate).
|
|
|
|
NOTE: A helm chart for `gitea-runner` also exists for easier deployments https://gitea.com/gitea/helm-actions
|
|
|
|
Each example persists **two** things, and it is worth knowing which is which:
|
|
|
|
- `/data` is the runner's working directory. It holds the `.runner` registration file and, optionally, the config file — so the runner re-attaches to the server instead of registering again.
|
|
- The Docker daemon's data root holds the images pulled for jobs (`/var/lib/docker` for the dind sidecar, `/home/rootless/.local/share/docker` for `dind-rootless`). It is *not* under `/data`. If you drop this volume, the examples still work, but the image cache is discarded whenever the pod is recreated and every job re-pulls its images.
|
|
|
|
- Kubernetes SIGKILLs a pod 30s after SIGTERM by default, long before a job finishes and reports its result, which leaves tasks the server can only reap as zombies. The manifests raise `terminationGracePeriodSeconds` to three hours, matching the systemd example and the `runner.timeout` job ceiling; set `runner.shutdown_timeout` below that so the runner drains jobs within the window rather than being killed mid-cleanup.
|
|
|
|
Files in this directory:
|
|
|
|
- [`dind-docker.yaml`](dind-docker.yaml)
|
|
How to create a Deployment and Persistent Volume for Kubernetes to act as a runner. The Docker credentials are re-generated each time the pod connects and does not need to be persisted.
|
|
|
|
- [`rootless-docker.yaml`](rootless-docker.yaml)
|
|
How to create a rootless Deployment and Persistent Volume for Kubernetes to act as a runner. The Docker credentials are re-generated each time the pod connects and does not need to be persisted.
|
|
|
|
- [`statefulset-dind.yaml`](statefulset-dind.yaml)
|
|
StatefulSet variant of the dind example. Each replica gets a stable identity and its own persistent volume via `volumeClaimTemplates`, so the runner keeps its `.runner` registration across restarts and reschedules instead of trying to register again.
|