mirror of
https://gitea.com/gitea/runner.git
synced 2026-08-26 05:47:45 +00:00
fix: bound blocking calls and stop failing silently (#1174)
Jobs occasionally go silent ([example](https://gitea.com/gitea/runner/actions/runs/805045/jobs/1055123)) mid-run and Gitea reaped them after `ZOMBIE_TASK_TIMEOUT`, with no error in the log. This contains a number of related fixes, all with full test coverage: 1. Bound every RPC to Gitea with a timeout, a stalled report otherwise parked logs and heartbeats for the whole job. 2. Cap `runner.fetch_timeout` at that ceiling. 3. Let only the daemon loop close its own channel, the race panicked the process. 4. Stop the job on any terminal server result, not just `RESULT_CANCELLED`. 5. Report that result instead of relabelling it as cancelled. 6. Log reporting failures once at each end of an outage instead of discarding them. 7. Clamp the acknowledged log index, a too-large ack panicked on a slice bound. 8. Stop reading server health from a `FetchTask` deadline, it marked the runner healthy and reset the error backoff on a timeout. 9. Return an error from the Docker version probe instead of a `logrus` panic. 10. Pass the context to go-git's fetch and pull. 11. Fail the clone when a refresh dies on a cancelled context. 12. Set `terminationGracePeriodSeconds` in the Kubernetes examples. Also contains a deprecation fix for goreleaser. Reviewed-on: https://gitea.com/gitea/runner/pulls/1174 Reviewed-by: bircni <bircni@icloud.com> Co-authored-by: silverwind <me@silverwind.io>
This commit is contained in:
@@ -11,6 +11,8 @@ Each example persists **two** things, and it is worth knowing which is which:
|
||||
- `/data` is the runner's working directory. It holds the `.runner` registration file and, optionally, the config file — so the runner re-attaches to the server instead of registering again.
|
||||
- The Docker daemon's data root holds the images pulled for jobs (`/var/lib/docker` for the dind sidecar, `/home/rootless/.local/share/docker` for `dind-rootless`). It is *not* under `/data`. If you drop this volume, the examples still work, but the image cache is discarded whenever the pod is recreated and every job re-pulls its images.
|
||||
|
||||
- Kubernetes SIGKILLs a pod 30s after SIGTERM by default, long before a job finishes and reports its result, which leaves tasks the server can only reap as zombies. The manifests raise `terminationGracePeriodSeconds` to three hours, matching the systemd example and the `runner.timeout` job ceiling; set `runner.shutdown_timeout` below that so the runner drains jobs within the window rather than being killed mid-cleanup.
|
||||
|
||||
Files in this directory:
|
||||
|
||||
- [`dind-docker.yaml`](dind-docker.yaml)
|
||||
|
||||
@@ -56,6 +56,7 @@ spec:
|
||||
app: runner
|
||||
spec:
|
||||
restartPolicy: Always
|
||||
terminationGracePeriodSeconds: 10800 # keep above runner.shutdown_timeout, see README
|
||||
volumes:
|
||||
- name: docker-socket
|
||||
emptyDir: {}
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
// Copyright 2026 The Gitea Authors. All rights reserved.
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
package kubernetes_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
var gracePeriod = regexp.MustCompile(`terminationGracePeriodSeconds: (\d+)`)
|
||||
|
||||
// Without it Kubernetes SIGKILLs the pod 30s after SIGTERM, mid-job.
|
||||
func TestManifestsSetTerminationGracePeriod(t *testing.T) {
|
||||
files, err := filepath.Glob("*.yaml")
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, files)
|
||||
|
||||
for _, file := range files {
|
||||
content, err := os.ReadFile(file)
|
||||
require.NoError(t, err)
|
||||
if !strings.Contains(string(content), "containers:") {
|
||||
continue
|
||||
}
|
||||
match := gracePeriod.FindStringSubmatch(string(content))
|
||||
require.NotNil(t, match, file)
|
||||
seconds, err := strconv.Atoi(match[1])
|
||||
require.NoError(t, err)
|
||||
assert.GreaterOrEqual(t, seconds, 3600, file)
|
||||
}
|
||||
}
|
||||
@@ -56,6 +56,7 @@ spec:
|
||||
app: runner
|
||||
spec:
|
||||
restartPolicy: Always
|
||||
terminationGracePeriodSeconds: 10800 # keep above runner.shutdown_timeout, see README
|
||||
volumes:
|
||||
- name: runner-data
|
||||
persistentVolumeClaim:
|
||||
|
||||
@@ -33,6 +33,7 @@ spec:
|
||||
app: runner
|
||||
spec:
|
||||
restartPolicy: Always
|
||||
terminationGracePeriodSeconds: 10800 # keep above runner.shutdown_timeout, see README
|
||||
volumes:
|
||||
- name: docker-socket
|
||||
emptyDir: {}
|
||||
|
||||
Reference in New Issue
Block a user