Compare commits

..

3 Commits

Author SHA1 Message Date
bircni a0c4de79f7 feat: add log.job.dir (#1165)
`log.job.dir` makes the runner write a copy of every task's log to that directory, as `<start time>-task-<id>.log`: the rows exactly as Gitea received them, with the same masking and the job's result on the last line. Off by default, and what Gitea shows does not change.

`log.job.retention` (default `168h`) and `log.job.max_size` (default `1GB`) bound the directory. Documented in the README.

---------

Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1165
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
2026-08-21 07:25:11 +00:00
silverwind dfe979e1d0 fix: stop a failed step disabling the toolkit patch (#1177)
A step that failed, for any reason, made the runner restore the action's stock bundle and mark it never to be patched again. Every later `actions/upload-artifact` run then failed with `GHESNotSupportedError`, and nothing in the log said why.

The edit is now made as the action is copied into the job container, under the lock that guards the copy, and nothing reverts it. That also closes the race where another job's checkout reset the bundle mid-job.

`cache.v2` no longer decides whether the edit is made, it only withdraws the v2 advertisement, so artifacts work whatever the cache is set to.

Also added a new `runner.patch_actions` option to turn the edit off if it ever breaks an action.

Fixes https://gitea.com/gitea/runner/issues/1176

Reviewed-on: https://gitea.com/gitea/runner/pulls/1177
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-08-20 15:46:33 +00:00
bircni 11ac12efa4 enhance: add runner.default_image for jobs matching no label (#1164)
A job whose `runs-on` matches none of the runner's labels, which includes every job that sets no `runs-on` at all, runs in `runner.default_image`. It defaults to `docker.gitea.com/runner-images:ubuntu-latest` as before, so a mirror can be pointed at instead.

A runner with no reachable docker daemon now runs such a job on the host, rather than failing on an image it cannot pull. Runners that use docker are unaffected and never probe for one.

This matters most to a host-mode runner, one whose labels are all `host`. Such a runner has no daemon to pull an image with, so a job matching none of its labels used to fail at container start. It now runs on the host, where that runner runs everything else anyway, and it takes no configuration to get there. A host-mode runner that does have a daemon within reach keeps using the image, unchanged.

Supersedes https://gitea.com/gitea/runner/pulls/642

---------

Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1164
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
2026-08-19 18:36:00 +00:00
19 changed files with 522 additions and 363 deletions
+8 -2
View File
@@ -228,7 +228,7 @@ a workflow with `runs-on: ubuntu-latest` is executed in the `runner-images:ubunt
Names may themselves contain a colon (for example `pool:e57e18d4-10d4-406f-93bf-60f127221bdd`); only `host` and `docker` are treated as schemas. Names may themselves contain a colon (for example `pool:e57e18d4-10d4-406f-93bf-60f127221bdd`); only `host` and `docker` are treated as schemas.
If a job's `runs-on` matches none of the runner's labels, the job still runs, in the default `docker.gitea.com/runner-images:ubuntu-latest` image. Images maintained for this purpose are listed at [gitea/runner-images](https://gitea.com/gitea/runner-images). If a job's `runs-on` matches none of the runner's labels, or sets no `runs-on` at all, it still runs: in `runner.default_image` where docker is available, on the host where it is not. Images maintained for this purpose are listed at [gitea/runner-images](https://gitea.com/gitea/runner-images).
Labels are chosen at registration time (`--labels`, or the interactive prompt) and can be changed afterwards by editing `runner.labels` in the config file, or in the Gitea UI under the runner's settings. Labels are chosen at registration time (`--labels`, or the interactive prompt) and can be changed afterwards by editing `runner.labels` in the config file, or in the Gitea UI under the runner's settings.
@@ -314,7 +314,7 @@ cache:
v2: false v2: false
``` ```
Those actions refuse any host they do not take for GitHub. Rather than misreport the server URL, the runner edits that check out of the action's own bundle and keeps the untouched copy beside it; a bundle it does not recognise is left alone and keeps to v1. The same edit lets the stock `actions/upload-artifact` and `actions/download-artifact` work from `v4.4.0` on, without the `gitea-upload-artifact` fork. Those actions refuse any host they do not take for GitHub. Rather than misreport the server URL, the runner edits that check out of the action's own bundle on its way into the job, undone whenever the action is downloaded again. A bundle it does not recognise is left alone and keeps to v1. The same edit lets the stock `actions/upload-artifact` and `actions/download-artifact` work from `v4.4.0` on, without the `gitea-upload-artifact` fork, so it is made whatever `v2` says: that setting only governs the API the runner advertises. Set `runner.patch_actions: false` to leave every bundle exactly as shipped, an escape hatch for an action the edit breaks. The artifact actions then refuse again and the cache client keeps to v1.
**Shared cache across multiple runners** **Shared cache across multiple runners**
@@ -386,6 +386,12 @@ Both hooks are synchronous and block the job while they run. Either one exiting
See **[docs/job-hooks.md](docs/job-hooks.md)** for the execution order, environment, and platform notes. See **[docs/job-hooks.md](docs/job-hooks.md)** for the execution order, environment, and platform notes.
#### Local job logs (`log.job.dir`)
Set `log.job.dir` to a path and the runner writes a copy of every task's log there as `<start time>-task-<id>.log`: the rows exactly as Gitea received them, with the same secrets masked and the job's result on the last line. Off by default, and what Gitea shows does not change.
`log.job.retention` (default `168h`) is how long a log is kept, expired ones being deleted as new tasks start, and `log.job.max_size` (default `1GB`) caps one log. Keep `retention` above `runner.timeout` so a long job cannot outlive its own log, and prefer local disk, the file is written while the job runs. Only the runner's own user can read it.
### Example Deployments ### Example Deployments
Check out the [examples](examples) directory for sample deployment types. Check out the [examples](examples) directory for sample deployment types.
+5
View File
@@ -167,6 +167,11 @@ func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actio
defer git.AcquireCloneLock(actionDir)() defer git.AcquireCloneLock(actionDir)()
if !rc.Config.NoActionPatch {
// A concurrent job's prepare resets this directory, so patch under the copy's lock.
patchActions(ctx, actionScriptPaths(filepath.Join(actionDir, actionPath), step.getActionModel()))
}
if err := removeGitIgnore(ctx, actionDir); err != nil { if err := removeGitIgnore(ctx, actionDir); err != nil {
return err return err
} }
@@ -12,7 +12,6 @@ import (
"strings" "strings"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git"
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
) )
@@ -41,8 +40,7 @@ const (
cacheURLEnv = "ACTIONS_CACHE_URL" cacheURLEnv = "ACTIONS_CACHE_URL"
resultsURLEnv = "ACTIONS_RESULTS_URL" resultsURLEnv = "ACTIONS_RESULTS_URL"
// localhostHost is the suffix isGhes accepts; emptying the test is what opens the gate, // localhostHost is the suffix isGhes accepts.
// because every hostname ends with the empty string.
localhostHost = ".LOCALHOST" localhostHost = ".LOCALHOST"
// artifactRefusal is the only thing the gate guards in @actions/artifact, which is what makes // artifactRefusal is the only thing the gate guards in @actions/artifact, which is what makes
@@ -50,13 +48,6 @@ const (
// runner has not looked at, and is left alone. // runner has not looked at, and is left alone.
artifactRefusal = "GHESNotSupportedError" artifactRefusal = "GHESNotSupportedError"
// sidecarSuffix names the directory of untouched copies, a sibling of the action directory
// because that directory is copied wholesale into job containers.
sidecarSuffix = ".toolkit-patch"
// skipMarker in the sidecar means a patched bundle already failed once here.
skipMarker = "skip"
maxBundleSize = 64 << 20 maxBundleSize = 64 << 20
) )
@@ -101,163 +92,74 @@ func actionScriptPaths(dir string, action *model.Action) []string {
} }
var paths []string var paths []string
for _, script := range []string{action.Runs.Pre, action.Runs.Main, action.Runs.Post} { for _, script := range []string{action.Runs.Pre, action.Runs.Main, action.Runs.Post} {
if script != "" { if script == "" {
paths = append(paths, filepath.Join(dir, script)) continue
} }
path := filepath.Join(dir, script)
// `runs` is the action's own yaml, and a key pointing outside its directory is not ours.
if rel, err := filepath.Rel(dir, path); err != nil || strings.HasPrefix(rel, "..") {
continue
}
paths = append(paths, path)
} }
return paths return paths
} }
// patchToolkit edits the toolkit in an action's bundles, keeping each original beside them. Every // patchActions edits the toolkit in an action's bundles. The caller holds the action directory's
// failure is silent and leaves the bundle as it was, which costs the cache client the v2 API and // clone lock, which is what keeps another job's checkout from resetting them before the copy.
// an artifact action nothing at all. func patchActions(ctx context.Context, scripts []string) {
func patchToolkit(ctx context.Context, actionDir string, scripts []string) {
if len(scripts) == 0 {
return
}
if _, err := os.Stat(filepath.Join(sidecarDir(actionDir), skipMarker)); err == nil {
return
}
defer git.AcquireCloneLock(actionDir)()
for _, script := range scripts { for _, script := range scripts {
if err := patchBundle(script, originalFor(actionDir, script)); err != nil { switch patched, err := patchBundle(script); {
common.Logger(ctx).Debugf("actions toolkit: %s left unpatched: %v", filepath.Base(script), err) case err != nil:
common.Logger(ctx).Warnf("actions toolkit: %s left unpatched: %v", script, err)
case patched:
common.Logger(ctx).Debugf("actions toolkit: patched %s", script)
} }
} }
} }
// revertToolkit puts the originals back and stops this action being patched again, so the next job func patchBundle(script string) (bool, error) {
// runs it exactly as shipped. Called when a step failed with a patched bundle; it does not re-run
// the step, because a step's outputs and env-file writes are already recorded by then.
func revertToolkit(ctx context.Context, actionDir string, scripts []string) {
if len(scripts) == 0 {
return
}
if _, err := os.Stat(sidecarDir(actionDir)); err != nil {
return
}
defer git.AcquireCloneLock(actionDir)()
reverted := false
for _, script := range scripts {
original := originalFor(actionDir, script)
if !isPatchOf(original, script) {
continue
}
if err := os.Rename(original, script); err == nil {
reverted = true
}
}
if reverted {
_ = os.WriteFile(filepath.Join(sidecarDir(actionDir), skipMarker), nil, 0o600)
common.Logger(ctx).Warnf("actions toolkit: restored the original %s, it will not be patched again", filepath.Base(actionDir))
}
}
// sidecarDir holds an action's untouched bundles, and the marker that stops it being patched.
func sidecarDir(actionDir string) string {
return actionDir + sidecarSuffix
}
// originalFor is where a script's untouched copy lives, or "" for a script the action's own
// `runs` keys placed outside its directory, which is not this runner's to rewrite.
func originalFor(actionDir, script string) string {
rel, err := filepath.Rel(actionDir, script)
if err != nil || strings.HasPrefix(rel, "..") {
return ""
}
return filepath.Join(sidecarDir(actionDir), rel)
}
// patchBundle rewrites one entrypoint in place. The untouched copy kept beside it is what marks
// the bundle as already patched.
func patchBundle(script, original string) error {
if original == "" {
return nil
}
if _, err := os.Stat(original); err == nil {
if isPatchOf(original, script) {
return nil
}
// The action's ref moved and git checked the new bundle out over the patched one, so
// the pair no longer belongs together. Patch afresh rather than keep an original that
// would restore an older version of the action.
if err := os.Remove(original); err != nil {
return err
}
}
info, err := os.Stat(script) info, err := os.Stat(script)
if err != nil { if err != nil {
return err return false, err
} }
if info.Size() > maxBundleSize { if info.Size() > maxBundleSize {
return nil return false, nil
} }
data, err := os.ReadFile(script) data, err := os.ReadFile(script)
if err != nil { if err != nil {
return err return false, err
} }
patched, ok := patchedBundle(data) patched, ok := patchedBundle(data)
if !ok { if !ok {
return nil return false, nil
} }
if err := os.MkdirAll(filepath.Dir(original), 0o755); err != nil { // No atomic write needed: every prepare checks the action out and hard resets it.
return err return true, os.WriteFile(script, patched, info.Mode().Perm())
}
// The copy is taken before the bundle is replaced, so a write that fails part way can put the
// action back as it was. A crash needs no handling: the clone executor checks the action out
// and hard resets it on every prepare, so a half-written bundle never outlives the job.
if err := os.WriteFile(original, data, info.Mode().Perm()); err != nil {
return err
}
if err := os.WriteFile(script, patched, info.Mode().Perm()); err != nil {
_ = os.Rename(original, script)
return err
}
return nil
}
// isPatchOf reports whether script is exactly what patching original produced. It is what proves
// the two still belong together: an action whose ref moved is checked out over the patched bundle,
// leaving an original that would restore the version before the move.
func isPatchOf(original, script string) bool {
data, err := os.ReadFile(original)
if err != nil {
return false
}
current, err := os.ReadFile(script)
if err != nil {
return false
}
patched, ok := patchedBundle(data)
return ok && bytes.Equal(patched, current)
} }
// patchedBundle opens the GHES gate, and where the cache toolkit is present, points the cache // patchedBundle opens the GHES gate, and where the cache toolkit is present, points the cache
// service at the cache server. A bundle this runner cannot account for comes back untouched. // service at the cache server. A bundle this runner cannot account for comes back untouched.
func patchedBundle(data []byte) ([]byte, bool) { func patchedBundle(data []byte) ([]byte, bool) {
// Literals before regex: most bundles carry neither toolkit and stop here. The artifact gate
// guards a refusal with no URL to move, so it opens alone; the cache gate opens only with its
// service URL, since a bundle whose getter this cannot find is better left on v1.
artifact := bytes.Contains(data, []byte(artifactRefusal))
cache := bytes.Contains(data, []byte(CacheServiceV2Env)) && serviceURLBranches.Match(data)
if !artifact && !cache {
return data, false
}
if !localhostTest.Match(data) { if !localhostTest.Match(data) {
return data, false return data, false
} }
switch {
case bytes.Contains(data, []byte(CacheServiceV2Env)):
// The cache toolkit: both edits or neither, because choosing v2 without redirecting the
// URL would send the client to a results URL that serves no cache service.
if !serviceURLBranches.Match(data) {
return data, false
}
case bytes.Contains(data, []byte(artifactRefusal)):
// The artifact toolkit, where the gate is a plain refusal and there is no URL to move:
// artifacts already go to Gitea, which implements that service.
default:
return data, false
}
opened := localhostTest.ReplaceAllFunc(data, func(test []byte) []byte { opened := localhostTest.ReplaceAllFunc(data, func(test []byte) []byte {
// Drop the hostname from the test rather than rewriting the call, so the bundle's own // Drop the hostname from the test rather than rewriting the call, so the bundle's own
// quoting survives and the result stays valid even inside a string literal. // quoting survives and the result stays valid even inside a string literal.
return bytes.Replace(test, []byte(localhostHost), nil, 1) return bytes.Replace(test, []byte(localhostHost), nil, 1)
}) })
return serviceURLBranches.ReplaceAll(opened, cacheURLFirst), true if cache {
opened = serviceURLBranches.ReplaceAll(opened, cacheURLFirst)
}
return opened, true
} }
@@ -108,17 +108,15 @@ func runActionEntrypoint(t *testing.T, script string, env jobEnv, inputs map[str
return string(out) return string(out)
} }
// patchedAction downloads one entrypoint and patches it exactly as a downloaded action would be, // patchedAction downloads one entrypoint and patches it exactly as a downloaded action would be.
// keeping the untouched original in the sidecar beside it.
func patchedAction(t *testing.T, repo, ref, entrypoint string) string { func patchedAction(t *testing.T, repo, ref, entrypoint string) string {
t.Helper() t.Helper()
body, err := os.ReadFile(bundleFromGitHub(t, repo, ref, entrypoint)) body, err := os.ReadFile(bundleFromGitHub(t, repo, ref, entrypoint))
require.NoError(t, err) require.NoError(t, err)
dir := tempDirPath(t) script := filepath.Join(tempDirPath(t), filepath.Base(entrypoint))
script := filepath.Join(dir, filepath.Base(entrypoint))
require.NoError(t, os.WriteFile(script, body, 0o600)) require.NoError(t, os.WriteFile(script, body, 0o600))
patchToolkit(t.Context(), dir, []string{script}) patchActions(t.Context(), []string{script})
return script return script
} }
@@ -184,7 +182,7 @@ func TestCacheServiceV2EndToEnd(t *testing.T) {
// cache server on its own address. That is what a runner without a results service of its own // cache server on its own address. That is what a runner without a results service of its own
// leaves its jobs with, so it has to round trip too. // leaves its jobs with, so it has to round trip too.
env.workspace = tempDirPath(t) env.workspace = tempDirPath(t)
v1 := runActionEntrypoint(t, filepath.Join(sidecarDir(filepath.Dir(restore)), "index.js"), env, inputs) v1 := runActionEntrypoint(t, bundleFromGitHub(t, "actions/cache", actionsCacheRef, "dist/restore/index.js"), env, inputs)
require.Contains(t, v1, "Cache service version: v1") require.Contains(t, v1, "Cache service version: v1")
require.Contains(t, v1, "Cache restored from key: "+key) require.Contains(t, v1, "Cache restored from key: "+key)
} }
@@ -194,7 +192,7 @@ func TestCacheServiceV2EndToEnd(t *testing.T) {
// anchored on that distance. One entrypoint from each of the families that bundle the cache // anchored on that distance. One entrypoint from each of the families that bundle the cache
// toolkit, patched but not run, is what keeps a future release from quietly matching only one of // toolkit, patched but not run, is what keeps a future release from quietly matching only one of
// the two shapes and leaving every cache on v1. // the two shapes and leaving every cache on v1.
func TestToolkitPatchAcrossActions(t *testing.T) { func TestPatchedBundleAcrossActions(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
repo, ref, path string repo, ref, path string
wantPatched bool wantPatched bool
@@ -5,7 +5,6 @@ package runner
import ( import (
"context" "context"
"errors"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
@@ -14,6 +13,7 @@ import (
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@@ -172,51 +172,42 @@ func TestPatchedBundleSurvivesInsideAStringLiteral(t *testing.T) {
require.NoError(t, err, "%s", checked) require.NoError(t, err, "%s", checked)
} }
func TestPatchBundleKeepsTheOriginal(t *testing.T) { // An action with a pre step is copied, and so patched, twice.
dir, script := bundleFile(t, gateTSC) func TestPatchBundleIsIdempotent(t *testing.T) {
original := originalFor(dir, script) script := bundleFile(t, gateTSC)
require.NoError(t, patchBundle(script, original)) done, err := patchBundle(script)
require.NoError(t, err)
require.True(t, done)
patched, err := os.ReadFile(script) patched, err := os.ReadFile(script)
require.NoError(t, err) require.NoError(t, err)
assert.True(t, gateOpened(string(patched))) require.True(t, gateOpened(string(patched)))
kept, err := os.ReadFile(original) done, err = patchBundle(script)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, gateTSC, string(kept), "the untouched bundle is kept outside the action tree") assert.False(t, done, "a patched bundle is not patched again")
assert.NotContains(t, original, dir+string(filepath.Separator), "originals must not ship into job containers")
// Patching again must not stack, and must not overwrite the kept original.
require.NoError(t, patchBundle(script, original))
again, err := os.ReadFile(script) again, err := os.ReadFile(script)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, string(patched), string(again)) assert.Equal(t, string(patched), string(again))
kept, err = os.ReadFile(original)
require.NoError(t, err)
assert.Equal(t, gateTSC, string(kept))
} }
// A bundle with nothing to patch is left exactly as it was, with no original kept beside it.
func TestPatchBundleLeavesOtherActionsAlone(t *testing.T) { func TestPatchBundleLeavesOtherActionsAlone(t *testing.T) {
dir, script := bundleFile(t, `console.log("checkout")`) script := bundleFile(t, `console.log("checkout")`)
original := originalFor(dir, script)
require.NoError(t, patchBundle(script, original)) done, err := patchBundle(script)
require.NoError(t, err)
assert.False(t, done)
body, err := os.ReadFile(script) body, err := os.ReadFile(script)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, `console.log("checkout")`, string(body)) assert.Equal(t, `console.log("checkout")`, string(body))
_, err = os.Stat(original)
assert.True(t, os.IsNotExist(err), "no original is kept for a bundle that was not patched")
} }
// bundleFile writes one entrypoint into a fresh action directory. func bundleFile(t *testing.T, body string) string {
func bundleFile(t *testing.T, body string) (dir, script string) {
t.Helper() t.Helper()
dir = t.TempDir() script := filepath.Join(t.TempDir(), "index.js")
script = filepath.Join(dir, "index.js")
require.NoError(t, os.WriteFile(script, []byte(body), 0o600)) require.NoError(t, os.WriteFile(script, []byte(body), 0o600))
return dir, script return script
} }
func TestActionScriptPaths(t *testing.T) { func TestActionScriptPaths(t *testing.T) {
@@ -226,101 +217,50 @@ func TestActionScriptPaths(t *testing.T) {
// Only a node action has a bundle to patch. // Only a node action has a bundle to patch.
assert.Nil(t, actionScriptPaths("/a", &model.Action{Runs: model.ActionRuns{Using: "docker", Image: "alpine"}})) assert.Nil(t, actionScriptPaths("/a", &model.Action{Runs: model.ActionRuns{Using: "docker", Image: "alpine"}}))
assert.Nil(t, actionScriptPaths("/a", nil)) assert.Nil(t, actionScriptPaths("/a", nil))
// An action naming a file outside its own directory does not get it rewritten.
escaping := &model.Action{Runs: model.ActionRuns{Using: "node20", Main: "../../elsewhere/index.js"}}
assert.Nil(t, actionScriptPaths("/a", escaping))
} }
// A step that fails with a patched bundle gets the untouched bundle back, and the action is not // The bundle has to be patched whatever state the shared action directory is in, because a
// patched again, so later jobs run it exactly as its author shipped it. // concurrent job's prepare checks the action out again and resets it.
func TestRevertToolkit(t *testing.T) { func TestPatchActionsAtTheContainerCopy(t *testing.T) {
dir, script := bundleFile(t, gateTSC) copiedBundle := func(t *testing.T, noPatch bool) string {
scripts := []string{script}
patchToolkit(t.Context(), dir, scripts)
body, err := os.ReadFile(script)
require.NoError(t, err)
require.True(t, gateOpened(string(body)), "precondition: the bundle is patched")
revertToolkit(t.Context(), dir, scripts)
body, err = os.ReadFile(script)
require.NoError(t, err)
assert.Equal(t, gateTSC, string(body), "the original bundle is back")
// The skip marker survives, so the action stays unpatched from now on.
patchToolkit(t.Context(), dir, scripts)
body, err = os.ReadFile(script)
require.NoError(t, err)
assert.Equal(t, gateTSC, string(body), "a reverted action stays unpatched")
}
// An action whose ref moves is checked out over the patched bundle. The kept original then
// belongs to the version before the move, and must not be restored over the new one.
func TestPatchBundleAfterTheActionMoved(t *testing.T) {
dir, script := bundleFile(t, gateTSC)
original := originalFor(dir, script)
scripts := []string{script}
patchToolkit(t.Context(), dir, scripts)
require.NoError(t, os.WriteFile(script, []byte(gateWebpack), 0o600)) // the new version lands
// Reverting must not roll the action back to the version the original came from.
revertToolkit(t.Context(), dir, scripts)
body, err := os.ReadFile(script)
require.NoError(t, err)
assert.Equal(t, gateWebpack, string(body))
// Nothing was reverted, so the action is not marked off either: the new version is patched
// in its own right, and keeps its own original.
require.NoFileExists(t, filepath.Join(sidecarDir(dir), skipMarker))
require.NoError(t, patchBundle(script, original))
body, err = os.ReadFile(script)
require.NoError(t, err)
assert.True(t, gateOpened(string(body)))
kept, err := os.ReadFile(original)
require.NoError(t, err)
assert.Equal(t, gateWebpack, string(kept))
}
// The wiring: a step patches its own bundles only when the runner serves the v2 API, and a step
// that fails gets them back. The action's path inside its repository is part of where they live.
func TestStepActionRemoteToolkitPatch(t *testing.T) {
newStep := func(t *testing.T, patch bool) (*stepActionRemote, string) {
t.Helper() t.Helper()
cm := &containerMock{}
sar := &stepActionRemote{ sar := &stepActionRemote{
Step: &model.Step{Uses: "owner/repo/sub@v1"}, Step: &model.Step{Uses: "owner/repo/sub@v1"},
remoteAction: &remoteAction{Org: "owner", Repo: "repo", Path: "sub", Ref: "v1"}, remoteAction: &remoteAction{Org: "owner", Repo: "repo", Path: "sub", Ref: "v1"},
action: &model.Action{Runs: model.ActionRuns{Using: "node20", Main: "index.js"}}, action: &model.Action{Runs: model.ActionRuns{Using: "node20", Main: "index.js"}},
RunContext: &RunContext{ RunContext: &RunContext{
Config: &Config{ActionCacheDir: t.TempDir(), PatchToolkit: patch}, Config: &Config{ActionCacheDir: t.TempDir(), NoActionPatch: noPatch},
JobContainer: cm,
}, },
} }
script := filepath.Join(sar.actionDir(), "sub", "index.js") script := filepath.Join(sar.actionDir(), "sub", "index.js")
require.NoError(t, os.MkdirAll(filepath.Dir(script), 0o755)) require.NoError(t, os.MkdirAll(filepath.Dir(script), 0o755))
require.NoError(t, os.WriteFile(script, []byte(gateTSC), 0o600)) require.NoError(t, os.WriteFile(script, []byte(gateTSC), 0o600))
return sar, script
var copied string
cm.On("CopyDir", mock.Anything, mock.Anything, mock.Anything).Return(func(context.Context) error {
body, err := os.ReadFile(script)
require.NoError(t, err)
copied = string(body)
return nil
})
require.NoError(t, maybeCopyToActionDir(t.Context(), sar, sar.actionDir(), "sub", "/var/run/act/actions/repo/sub"))
return copied
} }
t.Run("left alone when the runner does not patch", func(t *testing.T) { t.Run("patched on its way in", func(t *testing.T) {
sar, script := newStep(t, false) assert.True(t, gateOpened(copiedBundle(t, false)))
require.NoError(t, sar.patchActionToolkit(t.Context()))
body, err := os.ReadFile(script)
require.NoError(t, err)
assert.Equal(t, gateTSC, string(body))
}) })
t.Run("patched, and put back when the step fails", func(t *testing.T) { // The escape hatch, for an action the edit breaks: the artifact actions refuse again, and the
sar, script := newStep(t, true) // cache client keeps to v1.
require.NoError(t, sar.patchActionToolkit(t.Context())) t.Run("as shipped when the runner is told not to patch", func(t *testing.T) {
assert.Equal(t, gateTSC, copiedBundle(t, true))
body, err := os.ReadFile(script)
require.NoError(t, err)
require.True(t, gateOpened(string(body)))
failed := errors.New("the step failed")
require.ErrorIs(t, sar.revertToolkitOnFailure(func(context.Context) error { return failed })(t.Context()), failed)
body, err = os.ReadFile(script)
require.NoError(t, err)
assert.Equal(t, gateTSC, string(body))
}) })
} }
+1 -1
View File
@@ -74,7 +74,7 @@ type Config struct {
ContainerNetworkCreateOptions container.NewDockerNetworkCreateExecutorInput // the default network create options ContainerNetworkCreateOptions container.NewDockerNetworkCreateExecutorInput // the default network create options
ActionCache ActionCache // Use a custom ActionCache Implementation ActionCache ActionCache // Use a custom ActionCache Implementation
ProxyEnv map[string]string // the proxy variables the job runs with, also given to service containers and image builds ProxyEnv map[string]string // the proxy variables the job runs with, also given to service containers and image builds
PatchToolkit bool // edit the @actions toolkit bundled into an action so it works against Gitea, see toolkit_patch.go NoActionPatch bool // run actions exactly as published, applying no compatibility patches, see patch_actions.go
PresetGitHubContext *model.GithubContext // the preset github context, overrides some fields like DefaultBranch, Env, Secrets etc. PresetGitHubContext *model.GithubContext // the preset github context, overrides some fields like DefaultBranch, Env, Secrets etc.
EventJSON string // the content of JSON file to use for event.json in containers, overrides EventPath EventJSON string // the content of JSON file to use for event.json in containers, overrides EventPath
+3 -40
View File
@@ -180,9 +180,6 @@ func (sar *stepActionRemote) prepareActionExecutor() common.Executor {
sar.action = actionModel sar.action = actionModel
return err return err
}, },
// A stage of its own: it takes the same clone lock, and it has to land before
// runAction copies the action into the job container.
sar.patchActionToolkit,
)(ctx) )(ctx)
} }
} }
@@ -201,7 +198,7 @@ func (sar *stepActionRemote) pre() common.Executor {
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
sar.prepareActionExecutor(), sar.prepareActionExecutor(),
runStepExecutor(sar, stepStagePre, sar.revertToolkitOnFailure(runPreStep(sar))).If(hasPreStep(sar)).If(shouldRunPreStep(sar))) runStepExecutor(sar, stepStagePre, runPreStep(sar)).If(hasPreStep(sar)).If(shouldRunPreStep(sar)))
} }
func (sar *stepActionRemote) main() common.Executor { func (sar *stepActionRemote) main() common.Executor {
@@ -223,47 +220,13 @@ func (sar *stepActionRemote) main() common.Executor {
return sar.RunContext.JobContainer.CopyDir(copyToPath, sar.RunContext.Config.Workdir+string(filepath.Separator)+".", sar.RunContext.Config.UseGitIgnore)(ctx) return sar.RunContext.JobContainer.CopyDir(copyToPath, sar.RunContext.Config.Workdir+string(filepath.Separator)+".", sar.RunContext.Config.UseGitIgnore)(ctx)
} }
actionDir := sar.actionDir() return sar.runAction(sar, sar.actionDir(), sar.remoteAction)(ctx)
return sar.revertToolkitOnFailure(sar.runAction(sar, actionDir, sar.remoteAction))(ctx)
}), }),
) )
} }
func (sar *stepActionRemote) post() common.Executor { func (sar *stepActionRemote) post() common.Executor {
return runStepExecutor(sar, stepStagePost, sar.revertToolkitOnFailure(runPostStep(sar))).If(hasPostStep(sar)).If(shouldRunPostStep(sar)) return runStepExecutor(sar, stepStagePost, runPostStep(sar)).If(hasPostStep(sar)).If(shouldRunPostStep(sar))
}
// toolkitBundles is the action directory and the entrypoints the toolkit may live in.
func (sar *stepActionRemote) toolkitBundles() (string, []string) {
if sar.remoteAction == nil {
return "", nil
}
dir := sar.actionDir()
return dir, actionScriptPaths(filepath.Join(dir, sar.remoteAction.Path), sar.action)
}
// patchActionToolkit edits the bundled toolkit so it works against Gitea: the artifact actions
// stop refusing, and the cache client keeps to the cache server whichever API version it picks.
func (sar *stepActionRemote) patchActionToolkit(ctx context.Context) error {
if sar.RunContext.Config.PatchToolkit {
dir, scripts := sar.toolkitBundles()
patchToolkit(ctx, dir, scripts)
}
return nil
}
// revertToolkitOnFailure restores the untouched bundles when the action fails, so a later job
// runs it as shipped rather than repeating a failure the patch may have caused.
func (sar *stepActionRemote) revertToolkitOnFailure(exec common.Executor) common.Executor {
return func(ctx context.Context) error {
err := exec(ctx)
if err != nil {
dir, scripts := sar.toolkitBundles()
revertToolkit(ctx, dir, scripts)
}
return err
}
} }
func (sar *stepActionRemote) actionDir() string { func (sar *stepActionRemote) actionDir() string {
+1 -2
View File
@@ -446,7 +446,6 @@ func runExec(ctx context.Context, execArgs *executeArgs) func(cmd *cobra.Command
config := &runner.Config{ config := &runner.Config{
Workdir: execArgs.Workdir(), Workdir: execArgs.Workdir(),
BindWorkdir: false, BindWorkdir: false,
PatchToolkit: true, // the cache server started above is what the patch points at
ReuseContainers: false, ReuseContainers: false,
ForcePull: execArgs.forcePull, ForcePull: execArgs.forcePull,
ForceRebuild: execArgs.forceRebuild, ForceRebuild: execArgs.forceRebuild,
@@ -558,7 +557,7 @@ func loadExecCmd(ctx context.Context) *cobra.Command {
execCmd.PersistentFlags().BoolVarP(&execArg.noSkipCheckout, "no-skip-checkout", "", false, "Do not skip actions/checkout") execCmd.PersistentFlags().BoolVarP(&execArg.noSkipCheckout, "no-skip-checkout", "", false, "Do not skip actions/checkout")
execCmd.PersistentFlags().BoolVarP(&execArg.debug, "debug", "d", false, "enable debug log") execCmd.PersistentFlags().BoolVarP(&execArg.debug, "debug", "d", false, "enable debug log")
execCmd.PersistentFlags().BoolVarP(&execArg.dryrun, "dryrun", "n", false, "dryrun mode") execCmd.PersistentFlags().BoolVarP(&execArg.dryrun, "dryrun", "n", false, "dryrun mode")
execCmd.PersistentFlags().StringVarP(&execArg.image, "image", "i", "docker.gitea.com/runner-images:ubuntu-latest", "Docker image to use. Use \"-self-hosted\" to run directly on the host.") execCmd.PersistentFlags().StringVarP(&execArg.image, "image", "i", config.DefaultImage, "Docker image to use. Use \"-self-hosted\" to run directly on the host.")
execCmd.PersistentFlags().StringVarP(&execArg.toolCacheMode, "tool-cache-mode", "", config.ToolCacheModeNone, "What to mount at RUNNER_TOOL_CACHE: none, or shared to reuse one tool cache across runs") execCmd.PersistentFlags().StringVarP(&execArg.toolCacheMode, "tool-cache-mode", "", config.ToolCacheModeNone, "What to mount at RUNNER_TOOL_CACHE: none, or shared to reuse one tool cache across runs")
execCmd.PersistentFlags().StringVarP(&execArg.network, "network", "", "", "Specify the network to which the container will connect") execCmd.PersistentFlags().StringVarP(&execArg.network, "network", "", "", "Specify the network to which the container will connect")
execCmd.PersistentFlags().StringVarP(&execArg.githubInstance, "gitea-instance", "", "", "Gitea instance to use.") execCmd.PersistentFlags().StringVarP(&execArg.githubInstance, "gitea-instance", "", "", "Gitea instance to use.")
+44 -10
View File
@@ -28,6 +28,7 @@ import (
"gitea.com/gitea/runner/internal/pkg/client" "gitea.com/gitea/runner/internal/pkg/client"
"gitea.com/gitea/runner/internal/pkg/config" "gitea.com/gitea/runner/internal/pkg/config"
"gitea.com/gitea/runner/internal/pkg/disk" "gitea.com/gitea/runner/internal/pkg/disk"
"gitea.com/gitea/runner/internal/pkg/envcheck"
"gitea.com/gitea/runner/internal/pkg/labels" "gitea.com/gitea/runner/internal/pkg/labels"
"gitea.com/gitea/runner/internal/pkg/metrics" "gitea.com/gitea/runner/internal/pkg/metrics"
"gitea.com/gitea/runner/internal/pkg/report" "gitea.com/gitea/runner/internal/pkg/report"
@@ -172,7 +173,7 @@ func (r *Runner) OnIdle(ctx context.Context) {
// directories above, a task beginning during the pass is safe because the cutoff keeps a // directories above, a task beginning during the pass is safe because the cutoff keeps a
// network it has created but not yet attached a container to out of scope. // network it has created but not yet attached a container to out of scope.
func (r *Runner) cleanupOrphanNetworks(ctx context.Context) { func (r *Runner) cleanupOrphanNetworks(ctx context.Context) {
if r.uuid == "" || !r.labels.RequireDocker() && !r.cfg.Container.RequireDocker { if r.uuid == "" || !r.requiresDocker() {
return return
} }
cutoff := r.now().Add(-r.cfg.Runner.WorkdirCleanupAge) cutoff := r.now().Add(-r.cfg.Runner.WorkdirCleanupAge)
@@ -347,6 +348,27 @@ func (r *Runner) isSelfHostedActionsURL(task *runnerv1.Task) bool {
return giteaDefaultActionsURL != "" && giteaDefaultActionsURL != "https://github.com" return giteaDefaultActionsURL != "" && giteaDefaultActionsURL != "https://github.com"
} }
// dockerReachable is a variable so tests can substitute one that needs no Docker daemon. It
// probes the environment act connects through, not container.docker_host, which act ignores.
var dockerReachable = func(ctx context.Context) bool {
ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()
return envcheck.CheckIfDockerRunning(ctx, "") == nil
}
func (r *Runner) requiresDocker() bool {
return r.labels.RequireDocker() || r.cfg.Container.RequireDocker
}
// fallbackPlatform is where a job runs whose runs-on matches no label, as any job without a
// runs-on does, since Gitea sends those to every runner.
func (r *Runner) fallbackPlatform(ctx context.Context) string {
if r.requiresDocker() || dockerReachable(ctx) {
return r.cfg.Runner.DefaultImage
}
return labels.SelfHostedPlatform
}
func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.Reporter) (err error) { func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.Reporter) (err error) {
defer func() { defer func() {
if r := recover(); r != nil { if r := recover(); r != nil {
@@ -439,10 +461,13 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
// is that server's responsibility to authenticate requests. // is that server's responsibility to authenticate requests.
revokeCache, resultsURL := r.registerCacheForTask(giteaRuntimeToken, preset.Repository, reporter) revokeCache, resultsURL := r.registerCacheForTask(giteaRuntimeToken, preset.Repository, reporter)
defer revokeCache() defer revokeCache()
// A cache server that agreed to forward the artifact half is the whole results service, so // A cache server that agreed to forward the artifact half is the whole results service, so the
// the job is pointed at it and the v2 variable is finally true. // job is pointed at it.
if resultsURL != "" { if resultsURL != "" {
envs["ACTIONS_RESULTS_URL"], envs[runner.CacheServiceV2Env] = resultsURL, "true" envs["ACTIONS_RESULTS_URL"] = resultsURL
if r.cacheServiceV2() {
envs[runner.CacheServiceV2Env] = "true"
}
} }
eventJSON, err := json.Marshal(preset.Event) eventJSON, err := json.Marshal(preset.Event)
@@ -475,6 +500,15 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
// Without bind_workdir, the workspace path omits the task id; concurrent host-mode jobs // Without bind_workdir, the workspace path omits the task id; concurrent host-mode jobs
// for the same repository would share this directory and can race with per-job cleanup. // for the same repository would share this directory and can race with per-job cleanup.
// act asks for the platform once per step, so resolve the fallback at most once per task.
fallbackPlatform := sync.OnceValue(func() string { return r.fallbackPlatform(ctx) })
platformPicker := func(runsOn []string) string {
if platform := r.labels.PickPlatform(runsOn); platform != "" {
return platform
}
return fallbackPlatform()
}
runnerConfig := &runner.Config{ runnerConfig := &runner.Config{
// On Linux, Workdir will be like "/<parent_directory>/<owner>/<repo>" // On Linux, Workdir will be like "/<parent_directory>/<owner>/<repo>"
// On Windows, Workdir will be like "\<parent_directory>\<owner>\<repo>" // On Windows, Workdir will be like "\<parent_directory>\<owner>\<repo>"
@@ -484,7 +518,7 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
AllocatePTY: r.cfg.Runner.AllocatePTY, AllocatePTY: r.cfg.Runner.AllocatePTY,
ActionOfflineMode: r.cfg.Cache.OfflineMode, ActionOfflineMode: r.cfg.Cache.OfflineMode,
ActionCloneDepth: actionCloneDepth, ActionCloneDepth: actionCloneDepth,
PatchToolkit: r.patchToolkit(), NoActionPatch: r.cfg.Runner.PatchActions != nil && !*r.cfg.Runner.PatchActions,
ReuseContainers: false, ReuseContainers: false,
ForcePull: r.cfg.Container.ForcePull, ForcePull: r.cfg.Container.ForcePull,
@@ -517,7 +551,7 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
Privileged: r.cfg.Container.Privileged, Privileged: r.cfg.Container.Privileged,
DefaultActionInstance: r.getDefaultActionsURL(task), DefaultActionInstance: r.getDefaultActionsURL(task),
DefaultActionInstanceIsSelfHosted: r.isSelfHostedActionsURL(task), DefaultActionInstanceIsSelfHosted: r.isSelfHostedActionsURL(task),
PlatformPicker: r.labels.PickPlatform, PlatformPicker: platformPicker,
JobStartedHook: r.cfg.Runner.Hooks.JobStarted, JobStartedHook: r.cfg.Runner.Hooks.JobStarted,
JobCompletedHook: r.cfg.Runner.Hooks.JobCompleted, JobCompletedHook: r.cfg.Runner.Hooks.JobCompleted,
Vars: task.Vars, Vars: task.Vars,
@@ -559,10 +593,10 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
return execErr return execErr
} }
// patchToolkit reports whether act should edit the toolkit bundled into an action. It follows the // cacheServiceV2 reports whether jobs are told the cache service speaks v2. It is all cache.v2
// cache URL, because that is what the edits point the client at; see act/runner/toolkit_patch.go. // turns off: the bundle edit that reaches it is what the artifact actions need too.
func (r *Runner) patchToolkit() bool { func (r *Runner) cacheServiceV2() bool {
return r.envs["ACTIONS_CACHE_URL"] != "" && (r.cfg.Cache.V2 == nil || *r.cfg.Cache.V2) return r.cfg.Cache.V2 == nil || *r.cfg.Cache.V2
} }
// registerCacheForTask tells the cache server to accept requests authenticated // registerCacheForTask tells the cache server to accept requests authenticated
+36 -4
View File
@@ -12,6 +12,7 @@ import (
"gitea.com/gitea/runner/act/runner" "gitea.com/gitea/runner/act/runner"
clientmocks "gitea.com/gitea/runner/internal/pkg/client/mocks" clientmocks "gitea.com/gitea/runner/internal/pkg/client/mocks"
"gitea.com/gitea/runner/internal/pkg/config" "gitea.com/gitea/runner/internal/pkg/config"
"gitea.com/gitea/runner/internal/pkg/labels"
"gitea.com/gitea/runner/internal/pkg/ver" "gitea.com/gitea/runner/internal/pkg/ver"
"connectrpc.com/connect" "connectrpc.com/connect"
@@ -98,6 +99,34 @@ func TestNewRunnerInitializesLabelsAndEnvironment(t *testing.T) {
require.Empty(t, r.envs[runner.CacheServiceV2Env], "no cache server, nothing to serve v2 from") require.Empty(t, r.envs[runner.CacheServiceV2Env], "no cache server, nothing to serve v2 from")
} }
func TestRunnerFallbackPlatform(t *testing.T) {
tests := []struct {
name string
label string
dockerRunning bool
want string
}{
{"a docker label needs no daemon probe", "ubuntu:docker://node:18", false, "mirror.example/ci:noble"},
{"host labels keep the image where docker runs", "ubuntu:host", true, "mirror.example/ci:noble"},
{"host labels without docker run on the host", "ubuntu:host", false, labels.SelfHostedPlatform},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
reachable := dockerReachable
dockerReachable = func(context.Context) bool { return tt.dockerRunning }
t.Cleanup(func() { dockerReachable = reachable })
label, err := labels.Parse(tt.label)
require.NoError(t, err)
cfg := &config.Config{}
cfg.Runner.DefaultImage = "mirror.example/ci:noble"
r := &Runner{cfg: cfg, labels: labels.Labels{label}}
require.Equal(t, tt.want, r.fallbackPlatform(t.Context()))
})
}
}
// Proxy variables are assembled per task, because a job's service containers have to be // Proxy variables are assembled per task, because a job's service containers have to be
// reached directly and they are only known once the workflow is parsed. // reached directly and they are only known once the workflow is parsed.
func TestNewRunnerLeavesProxyToTheTask(t *testing.T) { func TestNewRunnerLeavesProxyToTheTask(t *testing.T) {
@@ -140,7 +169,6 @@ func TestNewRunnerCacheServiceV2(t *testing.T) {
assert.Equal(t, "https://gitea.example", r.envs["ACTIONS_RESULTS_URL"]) assert.Equal(t, "https://gitea.example", r.envs["ACTIONS_RESULTS_URL"])
assert.Empty(t, r.envs[runner.CacheServiceV2Env], "a promise the runner has not made yet") assert.Empty(t, r.envs[runner.CacheServiceV2Env], "a promise the runner has not made yet")
assert.True(t, r.patchToolkit())
// The registration is what makes it true: the cache server takes the results service over, // The registration is what makes it true: the cache server takes the results service over,
// having been told which instance to forward the artifact half to. // having been told which instance to forward the artifact half to.
@@ -161,6 +189,11 @@ func TestNewRunnerCacheServiceV2(t *testing.T) {
defer resp.Body.Close() defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode, "the advertised results service serves no cache service") assert.Equal(t, http.StatusOK, resp.StatusCode, "the advertised results service serves no cache service")
// Turning v2 off withdraws the advertisement and nothing else.
assert.True(t, r.cacheServiceV2())
cfg.Cache.V2 = new(bool)
assert.False(t, r.cacheServiceV2())
} }
// The v1 cache client appends its path to ACTIONS_CACHE_URL without a separator, so a configured // The v1 cache client appends its path to ACTIONS_CACHE_URL without a separator, so a configured
@@ -174,9 +207,8 @@ func TestNewRunnerNormalizesTheExternalCacheServer(t *testing.T) {
r := NewRunner(cfg, &config.Registration{Name: "runner"}, cli) r := NewRunner(cfg, &config.Registration{Name: "runner"}, cli)
assert.Equal(t, "http://cache.local:8088/", r.envs["ACTIONS_CACHE_URL"]) assert.Equal(t, "http://cache.local:8088/", r.envs["ACTIONS_CACHE_URL"])
// Nothing to front the results service with, so the variable stays unset, but the bundles are // Nothing to front the results service with, so the variable stays unset and the client keeps
// still patched: artifacts v4 need that, and the patch keeps the cache client on the cache URL. // to v1, which reads the cache URL first.
assert.Equal(t, "https://gitea.example", r.envs["ACTIONS_RESULTS_URL"]) assert.Equal(t, "https://gitea.example", r.envs["ACTIONS_RESULTS_URL"])
assert.Empty(t, r.envs[runner.CacheServiceV2Env]) assert.Empty(t, r.envs[runner.CacheServiceV2Env])
assert.True(t, r.patchToolkit())
} }
+21 -7
View File
@@ -1,13 +1,20 @@
# Every option with its default value, all commented out. Read this file, do not copy it. # Every option with its default value, all commented out. Read this file, do not copy it.
# `./gitea-runner config init` writes a config file to copy the lines you change into. # `./gitea-runner config init` writes a config file to copy the lines you change into.
# Logging for the runner process itself (messages printed to stderr). # Logging for the runner process itself (messages printed to stderr), plus the copy of
# This does not control how workflow step output is streamed to the Gitea UI; # each task's log kept under log.job. Neither controls how workflow step output is streamed
# tune that with runner.log_report_* below. # to the Gitea UI; tune that with runner.log_report_* below.
log: log:
# logrus severity: trace, debug, info, warn, error, fatal, panic. # logrus severity: trace, debug, info, warn, error, fatal, panic.
# trace and debug turn on caller/file:line in log lines. Default if omitted: info. # trace and debug turn on caller/file:line in log lines. Default if omitted: info.
#level: info #level: info
# Write a copy of each task's log to dir as <start time>-task-<id>.log, so a job's output
# survives a failure to send it to Gitea. A path turns them on, empty turns them off.
# retention is how long a log is kept (0s keeps all), max_size caps one log (0 is no limit).
#job:
# dir: ""
# retention: 168h
# max_size: 1GB
runner: runner:
# Where to store the registration result. # Where to store the registration result.
@@ -80,6 +87,10 @@ runner:
# When true (the default), inject the ACT=true environment variable into jobs. # When true (the default), inject the ACT=true environment variable into jobs.
# Set to false so workflows gated on `if: ${{ !env.ACT }}` behave like they do on GitHub. # Set to false so workflows gated on `if: ${{ !env.ACT }}` behave like they do on GitHub.
#set_act_env: true #set_act_env: true
# When true (the default), apply compatibility patches to the actions a job runs, so actions
# written for GitHub work against this instance. Set to false to run them exactly as published,
# at the price of the stock artifact actions refusing and the cache client keeping to v1.
#patch_actions: true
# The labels of a runner are used to determine which jobs the runner can run, and how to run them. # The labels of a runner are used to determine which jobs the runner can run, and how to run them.
# Like: "macos-arm64:host" or "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest" # Like: "macos-arm64:host" or "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest"
# Find more images provided by Gitea at https://gitea.com/gitea/runner-images . # Find more images provided by Gitea at https://gitea.com/gitea/runner-images .
@@ -94,6 +105,9 @@ runner:
# terminal; tools like `docker build` emit redrawing progress frames into the captured log # terminal; tools like `docker build` emit redrawing progress frames into the captured log
# when a TTY is present. # when a TTY is present.
#allocate_pty: false #allocate_pty: false
# Image for a job whose runs-on matches none of the labels above. A runner without a
# docker daemon runs such a job on the host instead.
#default_image: "docker.gitea.com/runner-images:ubuntu-latest"
# What to mount at RUNNER_TOOL_CACHE (/opt/hostedtoolcache), where setup actions install tools: # What to mount at RUNNER_TOOL_CACHE (/opt/hostedtoolcache), where setup actions install tools:
# none: nothing. A docker job sees what its image ships there, a host job an empty dir, and # none: nothing. A docker job sees what its image ships there, a host job an empty dir, and
# either way what it installs is gone when the job ends. # either way what it installs is gone when the job ends.
@@ -161,10 +175,10 @@ cache:
# A moved tag (e.g. a re-tagged "v6") or an updated branch stays at the cached commit # A moved tag (e.g. a re-tagged "v6") or an updated branch stays at the cached commit
# until its cache entry expires or is manually removed. # until its cache entry expires or is manually removed.
#offline_mode: false #offline_mode: false
# Serve the actions cache service v2 API, used by actions/cache@v4.2 and later. Those actions # Serve the actions cache service v2 API. The actions that use it refuse any host they do not
# refuse any host they do not take for GitHub, so reaching it means editing that check out of # take for GitHub, so reaching it means editing that check out of their own bundle, undone
# the action's own bundle, keeping the untouched copy beside it. The same edit lets the stock # whenever it is downloaded again. That edit is made either way, this only governs the API
# upload-artifact and download-artifact work here. A bundle that does not match is left alone. # advertised. A bundle that does not match is left alone.
#v2: true #v2: true
# How the cache server discards entries, ignored when external_server is set since that # How the cache server discards entries, ignored when external_server is set since that
# server applies its own. Leave a setting out for its default; 0s or 0 turns the three # server applies its own. Leave a setting out for its default; 0s or 0 turns the three
+20 -3
View File
@@ -23,6 +23,9 @@ import (
// RequestTimeout bounds every RPC to Gitea, and with it runner.fetch_timeout. // RequestTimeout bounds every RPC to Gitea, and with it runner.fetch_timeout.
const RequestTimeout = 60 * time.Second const RequestTimeout = 60 * time.Second
// DefaultImage is the image jobs run in unless a label or runner.default_image names another.
const DefaultImage = "docker.gitea.com/runner-images:ubuntu-latest"
// DefaultPostTaskScriptTimeout is the fallback cap on how long the post-task // DefaultPostTaskScriptTimeout is the fallback cap on how long the post-task
// script may run when post_task_script is set without an explicit timeout. It is // script may run when post_task_script is set without an explicit timeout. It is
// applied both at config load (for a configured script) and at the point of use // applied both at config load (for a configured script) and at the point of use
@@ -35,9 +38,17 @@ const Minimal = `# Minimal config file. Every option it does not set keeps its d
# "gitea-runner config generate" prints all options, "config set <key> <value>" sets one here. # "gitea-runner config generate" prints all options, "config set <key> <value>" sets one here.
` `
// Log represents the configuration for logging. // Log represents the runner process's own logging, plus the copy of each task's log it keeps.
type Log struct { type Log struct {
Level string `yaml:"level"` // Level indicates the logging level. Level string `yaml:"level"` // Level indicates the logging level.
Job LogJob `yaml:"job"` // Job configures the copy of each task's log kept on the runner host.
}
// LogJob represents the configuration for the copy of each task's log kept on the runner host.
type LogJob struct {
Dir string `yaml:"dir"` // Dir is the directory the runner writes each task's log to. Empty, the default, writes none.
Retention time.Duration `yaml:"retention"` // Retention deletes a task's log directory once it is older than this. Default 168h, 0s keeps them regardless of age.
MaxSize Size `yaml:"max_size"` // MaxSize caps one task's log file. Default 1GB, 0 is no limit.
} }
// Runner represents the configuration for the runner. // Runner represents the configuration for the runner.
@@ -63,7 +74,9 @@ type Runner struct {
GithubMirror string `yaml:"github_mirror"` // GithubMirror defines what mirrors should be used when using github GithubMirror string `yaml:"github_mirror"` // GithubMirror defines what mirrors should be used when using github
ActionShallowClone *bool `yaml:"action_shallow_clone"` // ActionShallowClone fetches only the requested ref of an action repository at depth 1 instead of cloning every branch's full history. It is a pointer to distinguish between false and not set; if not set, it defaults to true. ActionShallowClone *bool `yaml:"action_shallow_clone"` // ActionShallowClone fetches only the requested ref of an action repository at depth 1 instead of cloning every branch's full history. It is a pointer to distinguish between false and not set; if not set, it defaults to true.
SetActEnv *bool `yaml:"set_act_env"` // SetActEnv controls whether the ACT=true environment variable is injected into jobs. It is a pointer to distinguish between false and not set; if not set, it defaults to true. Set it to false so workflows gated on `if: ${{ !env.ACT }}` behave like on GitHub. SetActEnv *bool `yaml:"set_act_env"` // SetActEnv controls whether the ACT=true environment variable is injected into jobs. It is a pointer to distinguish between false and not set; if not set, it defaults to true. Set it to false so workflows gated on `if: ${{ !env.ACT }}` behave like on GitHub.
PatchActions *bool `yaml:"patch_actions"` // PatchActions applies compatibility patches to the actions a job runs, so actions written for GitHub work against Gitea, see act/runner/patch_actions.go. It is a pointer to distinguish between false and not set; if not set, it defaults to true. Set it to false to run every action exactly as published, at the price of the artifact actions refusing.
AllocatePTY bool `yaml:"allocate_pty"` // AllocatePTY allocates a pseudo-TTY for each step's process. Default is false, matching GitHub's actions/runner. Enable only for jobs that need an interactive terminal; tools like docker build emit redrawing progress frames into the captured log when a TTY is present. Applies to both host and docker backends. AllocatePTY bool `yaml:"allocate_pty"` // AllocatePTY allocates a pseudo-TTY for each step's process. Default is false, matching GitHub's actions/runner. Enable only for jobs that need an interactive terminal; tools like docker build emit redrawing progress frames into the captured log when a TTY is present. Applies to both host and docker backends.
DefaultImage string `yaml:"default_image"` // DefaultImage is the image a job runs in when its runs-on matches none of the runner's labels. A runner without docker runs such a job on the host instead.
ToolCacheMode string `yaml:"tool_cache_mode"` // ToolCacheMode is what the runner mounts at RUNNER_TOOL_CACHE on both backends: ToolCacheModeNone or ToolCacheModeShared. ToolCacheMode string `yaml:"tool_cache_mode"` // ToolCacheMode is what the runner mounts at RUNNER_TOOL_CACHE on both backends: ToolCacheModeNone or ToolCacheModeShared.
PostTaskScript string `yaml:"post_task_script"` // PostTaskScript is the path to an executable script run on the host after each task's cleanup completes. Empty disables the hook. On Windows use .exe/.bat/.cmd; PowerShell (.ps1) is not supported yet as the configured path. PostTaskScript string `yaml:"post_task_script"` // PostTaskScript is the path to an executable script run on the host after each task's cleanup completes. Empty disables the hook. On Windows use .exe/.bat/.cmd; PowerShell (.ps1) is not supported yet as the configured path.
PostTaskScriptTimeout time.Duration `yaml:"post_task_script_timeout"` // PostTaskScriptTimeout caps how long the post-task script may run. Default is 5m when post_task_script is set. PostTaskScriptTimeout time.Duration `yaml:"post_task_script_timeout"` // PostTaskScriptTimeout caps how long the post-task script may run. Default is 5m when post_task_script is set.
@@ -86,7 +99,7 @@ type Cache struct {
ExternalSecret string `yaml:"external_secret"` // ExternalSecret is a shared secret between this runner and an external gitea-runner cache-server, enabling per-job ACTIONS_RUNTIME_TOKEN authentication and repo scoping over the network. Required whenever ExternalServer is set; ExternalSecretFile is the alternative way to provide it. ExternalSecret string `yaml:"external_secret"` // ExternalSecret is a shared secret between this runner and an external gitea-runner cache-server, enabling per-job ACTIONS_RUNTIME_TOKEN authentication and repo scoping over the network. Required whenever ExternalServer is set; ExternalSecretFile is the alternative way to provide it.
ExternalSecretFile string `yaml:"external_secret_file"` // ExternalSecretFile is the path to a file holding the ExternalSecret value, so the secret can be mounted instead of stored in the config file. LoadDefault reads it into ExternalSecret; setting both is an error. ExternalSecretFile string `yaml:"external_secret_file"` // ExternalSecretFile is the path to a file holding the ExternalSecret value, so the secret can be mounted instead of stored in the config file. LoadDefault reads it into ExternalSecret; setting both is an error.
OfflineMode bool `yaml:"offline_mode"` // OfflineMode reuses a cached action without fetching from the remote; a moved tag or branch stays at the cached commit until the cache entry is removed. OfflineMode bool `yaml:"offline_mode"` // OfflineMode reuses a cached action without fetching from the remote; a moved tag or branch stays at the cached commit until the cache entry is removed.
V2 *bool `yaml:"v2"` // V2 serves the actions cache service v2 API to jobs, used by actions/cache@v4.2 and later, and edits the action bundles that would otherwise refuse it. Unset means enabled. V2 *bool `yaml:"v2"` // V2 advertises the actions cache service v2 API to jobs. The bundle edit that reaches it is made either way, the artifact actions need it too. Unset means enabled.
// Eviction settings, ignored when ExternalServer is set since that server applies its own. // Eviction settings, ignored when ExternalServer is set since that server applies its own.
Retention time.Duration `yaml:"retention"` // Retention removes entries nothing has read or written within this window. Default 168h, 0 keeps them regardless of age. Retention time.Duration `yaml:"retention"` // Retention removes entries nothing has read or written within this window. Default 168h, 0 keeps them regardless of age.
@@ -197,7 +210,8 @@ type Config struct {
// LoadDefault returns the default configuration. // LoadDefault returns the default configuration.
// If file is not empty, it will be used to load the configuration. // If file is not empty, it will be used to load the configuration.
func LoadDefault(file string) (*Config, error) { func LoadDefault(file string) (*Config, error) {
cfg := &Config{Cache: DefaultCache()} // Seeded before the file is read, so a written 0 can mean off.
cfg := &Config{Cache: DefaultCache(), Log: Log{Job: LogJob{Retention: 7 * 24 * time.Hour, MaxSize: 1024 * 1024 * 1024}}}
definedRunnerKeys := map[string]bool{} definedRunnerKeys := map[string]bool{}
if file != "" { if file != "" {
content, err := os.ReadFile(file) content, err := os.ReadFile(file)
@@ -270,6 +284,9 @@ func LoadDefault(file string) (*Config, error) {
if cfg.Container.WorkdirParent == "" { if cfg.Container.WorkdirParent == "" {
cfg.Container.WorkdirParent = "workspace" cfg.Container.WorkdirParent = "workspace"
} }
if cfg.Runner.DefaultImage == "" {
cfg.Runner.DefaultImage = DefaultImage
}
if cfg.Runner.ToolCacheMode == "" { if cfg.Runner.ToolCacheMode == "" {
cfg.Runner.ToolCacheMode = ToolCacheModeNone cfg.Runner.ToolCacheMode = ToolCacheModeNone
} }
+12 -3
View File
@@ -64,6 +64,7 @@ func TestLoadDefault_DefaultsWorkdirCleanupAge(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, 24*time.Hour, cfg.Runner.WorkdirCleanupAge) assert.Equal(t, 24*time.Hour, cfg.Runner.WorkdirCleanupAge)
assert.Equal(t, 10*time.Minute, cfg.Runner.IdleCleanupInterval) assert.Equal(t, 10*time.Minute, cfg.Runner.IdleCleanupInterval)
assert.Equal(t, DefaultImage, cfg.Runner.DefaultImage)
} }
func TestLoadDefault_HealthChecksAreOptIn(t *testing.T) { func TestLoadDefault_HealthChecksAreOptIn(t *testing.T) {
@@ -185,14 +186,14 @@ runner:
assert.Equal(t, 5*time.Minute, cfg.Runner.PostTaskScriptTimeout) assert.Equal(t, 5*time.Minute, cfg.Runner.PostTaskScriptTimeout)
} }
func TestLoadDefault_LoadsCacheEviction(t *testing.T) { func write(t *testing.T, body string) string {
write := func(t *testing.T, body string) string {
t.Helper() t.Helper()
path := filepath.Join(t.TempDir(), "config.yaml") path := filepath.Join(t.TempDir(), "config.yaml")
require.NoError(t, os.WriteFile(path, []byte(body), 0o600)) require.NoError(t, os.WriteFile(path, []byte(body), 0o600))
return path return path
} }
func TestLoadDefault_LoadsCacheEviction(t *testing.T) {
t.Run("sizes accept any spelling of the unit", func(t *testing.T) { t.Run("sizes accept any spelling of the unit", func(t *testing.T) {
cfg, err := LoadDefault(write(t, "cache:\n retention: 336h\n repo_size_limit: 50gb\n size_limit: 1TiB\n sweep_interval: 15m\n")) cfg, err := LoadDefault(write(t, "cache:\n retention: 336h\n repo_size_limit: 50gb\n size_limit: 1TiB\n sweep_interval: 15m\n"))
require.NoError(t, err) require.NoError(t, err)
@@ -216,6 +217,14 @@ func TestLoadDefault_LoadsCacheEviction(t *testing.T) {
}) })
} }
func TestLoadDefault_LoadsJobLogs(t *testing.T) {
cfg, err := LoadDefault(write(t, "log:\n job:\n dir: /var/log/jobs\n retention: 0s\n max_size: 100MB\n"))
require.NoError(t, err)
assert.Equal(t, "/var/log/jobs", cfg.Log.Job.Dir)
assert.Zero(t, cfg.Log.Job.Retention, "zero keeps every task directory")
assert.Equal(t, Size(100*1024*1024), cfg.Log.Job.MaxSize)
}
func TestLoadDefault_LoadsJobHooks(t *testing.T) { func TestLoadDefault_LoadsJobHooks(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
path := filepath.Join(dir, "config.yaml") path := filepath.Join(dir, "config.yaml")
+6 -13
View File
@@ -11,6 +11,9 @@ import (
const ( const (
SchemeHost = "host" SchemeHost = "host"
SchemeDocker = "docker" SchemeDocker = "docker"
// SelfHostedPlatform is the platform marker act treats as "run on the host".
SelfHostedPlatform = "-self-hosted"
) )
type Label struct { type Label struct {
@@ -61,6 +64,7 @@ func (l Labels) RequireDocker() bool {
return false return false
} }
// PickPlatform returns the platform of the first runs-on entry this runner has a label for, or "".
func (l Labels) PickPlatform(runsOn []string) string { func (l Labels) PickPlatform(runsOn []string) string {
platforms := make(map[string]string, len(l)) platforms := make(map[string]string, len(l))
for _, label := range l { for _, label := range l {
@@ -69,7 +73,7 @@ func (l Labels) PickPlatform(runsOn []string) string {
// "//" will be ignored // "//" will be ignored
platforms[label.Name] = strings.TrimPrefix(label.Arg, "//") platforms[label.Name] = strings.TrimPrefix(label.Arg, "//")
case SchemeHost: case SchemeHost:
platforms[label.Name] = "-self-hosted" platforms[label.Name] = SelfHostedPlatform
default: default:
// unreachable: Parse only produces host or docker schemas // unreachable: Parse only produces host or docker schemas
continue continue
@@ -80,18 +84,7 @@ func (l Labels) PickPlatform(runsOn []string) string {
return v return v
} }
} }
return ""
// TODO: support multiple labels
// like:
// ["ubuntu-22.04"] => "ubuntu:22.04"
// ["with-gpu"] => "linux:with-gpu"
// ["ubuntu-22.04", "with-gpu"] => "ubuntu:22.04_with-gpu"
// return default.
// So the runner receives a task with a label that the runner doesn't have,
// it happens when the user have edited the label of the runner in the web UI.
// TODO: it may be not correct, what if the runner is used as host mode only?
return "docker.gitea.com/runner-images:ubuntu-latest"
} }
func (l Labels) Names() []string { func (l Labels) Names() []string {
+5 -5
View File
@@ -123,10 +123,10 @@ func TestPickPlatform(t *testing.T) {
want string want string
}{ }{
{"docker strips leading slashes", []string{"ubuntu"}, "node:18"}, {"docker strips leading slashes", []string{"ubuntu"}, "node:18"},
{"host maps to self-hosted marker", []string{"self-hosted"}, "-self-hosted"}, {"host maps to self-hosted marker", []string{"self-hosted"}, SelfHostedPlatform},
{"first match wins", []string{"self-hosted", "ubuntu"}, "-self-hosted"}, {"first match wins", []string{"self-hosted", "ubuntu"}, SelfHostedPlatform},
{"unknown falls back to default", []string{"windows"}, "docker.gitea.com/runner-images:ubuntu-latest"}, {"unknown label picks nothing", []string{"windows"}, ""},
{"no runsOn falls back to default", nil, "docker.gitea.com/runner-images:ubuntu-latest"}, {"no runsOn picks nothing", nil, ""},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
@@ -167,5 +167,5 @@ func TestOpaqueLabelRoundTrip(t *testing.T) {
require.Equal(t, ls, again) require.Equal(t, ls, again)
require.Equal(t, []string{raw}, again.Names()) require.Equal(t, []string{raw}, again.Names())
require.False(t, again.RequireDocker()) require.False(t, again.RequireDocker())
require.Equal(t, "-self-hosted", again.PickPlatform([]string{raw})) require.Equal(t, SelfHostedPlatform, again.PickPlatform([]string{raw}))
} }
+120
View File
@@ -0,0 +1,120 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package report
import (
"fmt"
"os"
"path/filepath"
"strings"
"time"
"gitea.com/gitea/runner/internal/pkg/config"
log "github.com/sirupsen/logrus"
)
const (
jobLogNameLayout = "20060102-150405"
jobLogTimestamp = "2006-01-02T15:04:05.000Z"
)
// jobLog is this task's copy of the rows sent to Gitea. A nil *jobLog is a no-op, and every
// caller holds Reporter.stateMu, so it needs no lock.
type jobLog struct {
file *os.File
size int64
max int64
stopped bool // the cap was reached or a write failed, only the trailer still follows
closed bool
}
// openJobLog returns nil when the logs are off or cannot be created: a copy must never fail a job.
func openJobLog(cfg config.LogJob, taskID int64, started time.Time) *jobLog {
if cfg.Dir == "" {
return nil
}
if err := os.MkdirAll(cfg.Dir, 0o700); err != nil { // repository output, readable by this user only
log.Warnf("cannot create job log directory %s: %v", cfg.Dir, err)
return nil
}
pruneJobLogs(cfg.Dir, cfg.Retention, started)
name := fmt.Sprintf("%s-task-%d.log", started.UTC().Format(jobLogNameLayout), taskID)
file, err := os.OpenFile(filepath.Join(cfg.Dir, name), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
log.Warnf("cannot create job log: %v", err)
return nil
}
log.Infof("writing the log of task %d to %s", taskID, file.Name())
return &jobLog{file: file, max: int64(cfg.MaxSize)}
}
func (j *jobLog) write(t time.Time, content string) {
if j == nil || j.stopped || j.closed {
return
}
line := t.UTC().Format(jobLogTimestamp) + " " + content
if j.max > 0 && j.size+int64(len(line))+1 > j.max {
j.stopped = true
j.line(runnerLine(fmt.Sprintf("truncated: log.job.max_size of %d bytes reached", j.max)))
return
}
j.line(line)
}
func (j *jobLog) close(trailer string) {
if j == nil || j.closed {
return
}
j.closed = true
j.line(runnerLine(trailer)) // past the cap on purpose: no trailer means the runner died mid-job
if err := j.file.Close(); err != nil {
log.Warnf("cannot close %s: %v", j.file.Name(), err)
}
}
// line writes unbuffered, so a killed runner keeps what it had written. Only the runner's own
// lines can carry a newline, a row reaching Gitea cannot (see DEVELOPMENT.md).
func (j *jobLog) line(content string) {
n, err := j.file.WriteString(strings.ReplaceAll(content, "\n", `\n`) + "\n")
j.size += int64(n)
if err != nil {
j.stopped = true // reported once, a failing write is a full disk and retrying floods the log
log.Warnf("cannot write %s: %v", j.file.Name(), err)
}
}
func runnerLine(content string) string {
return time.Now().UTC().Format(jobLogTimestamp) + " [runner] " + content
}
// pruneJobLogs removes the logs older than retention. The age comes from the name, not the
// mtime, which a reader or a backup tool can move.
func pruneJobLogs(root string, retention time.Duration, now time.Time) {
if retention <= 0 {
return
}
entries, err := os.ReadDir(root)
if err != nil {
log.Warnf("cannot list job log directory %s: %v", root, err)
return
}
cutoff := now.Add(-retention)
for _, entry := range entries {
stamp, _, isTaskLog := strings.Cut(entry.Name(), "-task-")
if !isTaskLog || entry.IsDir() || !strings.HasSuffix(entry.Name(), ".log") {
continue
}
if started, err := time.Parse(jobLogNameLayout, stamp); err != nil || !started.Before(cutoff) {
continue
}
name := filepath.Join(root, entry.Name())
if err := os.Remove(name); err != nil {
log.Warnf("cannot remove expired job log %s: %v", name, err)
}
}
}
+110
View File
@@ -0,0 +1,110 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package report
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.com/gitea/runner/internal/pkg/client/mocks"
"gitea.com/gitea/runner/internal/pkg/config"
connect_go "connectrpc.com/connect"
runnerv1 "gitea.dev/actionslib/runner/v1"
log "github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/types/known/structpb"
)
var testStart = time.Date(2026, 8, 14, 9, 12, 3, 0, time.UTC)
func readJobLog(t *testing.T, joblog *jobLog) string {
t.Helper()
content, err := os.ReadFile(joblog.file.Name())
require.NoError(t, err)
return string(content)
}
func TestJobLog_MirrorsUploadedRows(t *testing.T) {
client := mocks.NewClient(t)
client.On("UpdateLog", mock.Anything, mock.Anything).Return(func(_ context.Context, req *connect_go.Request[runnerv1.UpdateLogRequest]) (*connect_go.Response[runnerv1.UpdateLogResponse], error) {
return connect_go.NewResponse(&runnerv1.UpdateLogResponse{AckIndex: req.Msg.Index + int64(len(req.Msg.Rows))}), nil
})
client.On("UpdateTask", mock.Anything, mock.Anything).Return(connect_go.NewResponse(&runnerv1.UpdateTaskResponse{}), nil)
cfg, err := config.LoadDefault("")
require.NoError(t, err)
cfg.Log.Job.Dir = t.TempDir()
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
task := &runnerv1.Task{Id: 41, Context: &structpb.Struct{}, Secrets: map[string]string{"TOKEN": "s3cret-value"}}
reporter := NewReporter(ctx, cancel, client, task, cfg)
require.NotNil(t, reporter.jobLog)
reporter.RunDaemon()
reporter.ResetSteps(1)
fire := func(message string) {
require.NoError(t, reporter.Fire(&log.Entry{
Message: message,
Level: log.InfoLevel,
Data: log.Fields{"stage": "Main", "stepNumber": 0, "raw_output": true},
}))
}
fire("the token is s3cret-value")
fire("::add-mask::dyn4mic-value")
fire("and dyn4mic-value too")
fire("::debug::suppressed unless ACTIONS_STEP_DEBUG")
require.NoError(t, reporter.Close(""))
job := readJobLog(t, reporter.jobLog)
assert.Contains(t, job, "the token is ***")
assert.Contains(t, job, "and *** too")
assert.NotContains(t, job, "s3cret-value")
assert.NotContains(t, job, "dyn4mic-value")
assert.NotContains(t, job, "add-mask", "the row carrying the secret never reaches the log")
assert.NotContains(t, job, "suppressed unless", "the file holds only what was sent")
assert.NotRegexp(t, `(?m)^\S+Z ?$`, job, "the empty row Gitea needs is not job output")
assert.Contains(t, job, "[runner] task 41 finished: failure")
}
func TestJobLog_MaxSize(t *testing.T) {
joblog := openJobLog(config.LogJob{Dir: t.TempDir(), MaxSize: 200}, 1, testStart)
require.NotNil(t, joblog)
for range 10 {
joblog.write(testStart, strings.Repeat("x", 40))
}
joblog.close("task 1 finished: success")
joblog.write(testStart, "after the close") // a container goroutine can outlive the step
job := readJobLog(t, joblog)
assert.Equal(t, 1, strings.Count(job, "log.job.max_size"), "the cap is reported once")
assert.NotContains(t, job, "after the close")
assert.Contains(t, job, "[runner] task 1 finished: success", "the trailer is written past the cap")
}
func TestPruneJobLogs(t *testing.T) {
root := t.TempDir()
expired := filepath.Join(root, "20200101-000000-task-1.log")
fresh := filepath.Join(root, testStart.Format(jobLogNameLayout)+"-task-2.log")
unrelated := filepath.Join(root, "20200101-000000-task-3.txt")
for _, name := range []string{expired, fresh, unrelated} {
require.NoError(t, os.WriteFile(name, []byte("log"), 0o600))
}
pruneJobLogs(root, 0, testStart)
assert.FileExists(t, expired, "retention 0 keeps every log")
pruneJobLogs(root, 24*time.Hour, testStart)
assert.NoFileExists(t, expired)
assert.FileExists(t, fresh)
assert.FileExists(t, unrelated)
}
+35 -13
View File
@@ -94,6 +94,8 @@ type Reporter struct {
debugOutputEnabled bool debugOutputEnabled bool
stopCommandEndToken string stopCommandEndToken string
jobLog *jobLog // this task's rows on the runner's own disk, nil when log.job.dir is unset
} }
// extraMasks are values known before the job starts that are not among its secrets, such as // extraMasks are values known before the job starts that are not among its secrets, such as
@@ -132,6 +134,7 @@ func NewReporter(ctx context.Context, cancel context.CancelFunc, client client.C
reportFailing: map[string]bool{}, reportFailing: map[string]bool{},
daemon: make(chan struct{}), daemon: make(chan struct{}),
heartbeatStop: make(chan struct{}), heartbeatStop: make(chan struct{}),
jobLog: openJobLog(cfg.Log.Job, task.Id, time.Now()),
} }
rv.daemonWait = 6 * rv.effectiveCloseTimeout() rv.daemonWait = 6 * rv.effectiveCloseTimeout()
@@ -164,11 +167,14 @@ func (r *Reporter) Levels() []log.Level {
return log.AllLevels return log.AllLevels
} }
func appendIfNotNil[T any](s []*T, v *T) []*T { // appendLogRow buffers a row for the uploader and mirrors it into job.log. A nil row is one
if v != nil { // the command handler dropped, such as ::add-mask::. Caller holds stateMu.
return append(s, v) func (r *Reporter) appendLogRow(row *runnerv1.LogRow) {
if row == nil {
return
} }
return s r.logRows = append(r.logRows, row)
r.jobLog.write(row.Time.AsTime(), row.Content)
} }
// isJobStepEntry is used to not report composite step results incorrectly as step result // isJobStepEntry is used to not report composite step results incorrectly as step result
@@ -245,7 +251,7 @@ func (r *Reporter) Fire(entry *log.Entry) error {
} }
} }
if r.shouldAppendLogRow(entry) { if r.shouldAppendLogRow(entry) {
r.logRows = appendIfNotNil(r.logRows, r.parseLogRow(entry)) r.appendLogRow(r.parseLogRow(entry))
} }
r.unlockAndNotify(urgentState) r.unlockAndNotify(urgentState)
return nil return nil
@@ -259,7 +265,7 @@ func (r *Reporter) Fire(entry *log.Entry) error {
} }
if step == nil { if step == nil {
if r.shouldAppendLogRow(entry) { if r.shouldAppendLogRow(entry) {
r.logRows = appendIfNotNil(r.logRows, r.parseLogRow(entry)) r.appendLogRow(r.parseLogRow(entry))
} }
r.unlockAndNotify(false) r.unlockAndNotify(false)
return nil return nil
@@ -285,11 +291,11 @@ func (r *Reporter) Fire(entry *log.Entry) error {
step.LogIndex = int64(r.logOffset + len(r.logRows)) step.LogIndex = int64(r.logOffset + len(r.logRows))
} }
step.LogLength++ step.LogLength++
r.logRows = append(r.logRows, row) r.appendLogRow(row)
} }
} }
} else if r.shouldAppendLogRow(entry) { } else if r.shouldAppendLogRow(entry) {
r.logRows = appendIfNotNil(r.logRows, r.parseLogRow(entry)) r.appendLogRow(r.parseLogRow(entry))
} }
if v, ok := entry.Data["stepResult"]; ok && isJobStepEntry(entry) { if v, ok := entry.Data["stepResult"]; ok && isJobStepEntry(entry) {
if stepResult, ok := r.parseResult(v); ok { if stepResult, ok := r.parseResult(v); ok {
@@ -428,7 +434,7 @@ func (r *Reporter) logf(format string, a ...any) {
if !r.duringSteps() { if !r.duringSteps() {
// Masked like any other row: these bypass parseLogRow, but a caller can still // Masked like any other row: these bypass parseLogRow, but a caller can still
// interpolate a secret, such as a configured URL carrying credentials. // interpolate a secret, such as a configured URL carrying credentials.
r.logRows = append(r.logRows, r.newLogRow(timestamppb.Now(), fmt.Sprintf(format, a...))) r.appendLogRow(r.newLogRow(timestamppb.Now(), fmt.Sprintf(format, a...)))
} }
} }
@@ -479,13 +485,13 @@ func (r *Reporter) Close(lastWords string) error {
} }
} }
r.state.Result = result r.state.Result = result
r.logRows = append(r.logRows, &runnerv1.LogRow{ r.appendLogRow(&runnerv1.LogRow{
Time: timestamppb.Now(), Time: timestamppb.Now(),
Content: lastWords, Content: lastWords,
}) })
r.state.StoppedAt = timestamppb.Now() r.state.StoppedAt = timestamppb.Now()
} else if lastWords != "" { } else if lastWords != "" {
r.logRows = append(r.logRows, &runnerv1.LogRow{ r.appendLogRow(&runnerv1.LogRow{
Time: timestamppb.Now(), Time: timestamppb.Now(),
Content: lastWords, Content: lastWords,
}) })
@@ -510,6 +516,7 @@ func (r *Reporter) Close(lastWords string) error {
// supported branches, e.g. v1.28+. // supported branches, e.g. v1.28+.
r.stateMu.Lock() r.stateMu.Lock()
if len(r.logRows) == 0 { if len(r.logRows) == 0 {
// Not appendLogRow: the sentinel is not job output and has no place in job.log.
r.logRows = append(r.logRows, &runnerv1.LogRow{ r.logRows = append(r.logRows, &runnerv1.LogRow{
Time: timestamppb.Now(), Time: timestamppb.Now(),
Content: "", Content: "",
@@ -519,10 +526,21 @@ func (r *Reporter) Close(lastWords string) error {
// Separate budgets so a slow ReportLog can't starve the ReportState that // Separate budgets so a slow ReportLog can't starve the ReportState that
// carries the cancel acknowledgement. // carries the cancel acknowledgement.
return errors.Join( err := errors.Join(
r.flushFinal(func() error { return r.ReportLog(true) }), r.flushFinal(func() error { return r.ReportLog(true) }),
r.flushFinal(func() error { return r.ReportState(true) }), r.flushFinal(func() error { return r.ReportState(true) }),
) )
// After the flush so a failed handover is in the file too, under stateMu so a late entry cannot race.
r.stateMu.Lock()
trailer := fmt.Sprintf("task %d finished: %s", r.state.Id, metrics.ResultToStatusLabel(r.state.Result))
if err != nil {
trailer += fmt.Sprintf(", the final flush to Gitea failed: %v", err)
}
r.jobLog.close(r.mask(trailer))
r.stateMu.Unlock()
return err
} }
// flushFinal retries fn on a detached, bounded context so a cancelled r.ctx // flushFinal retries fn on a detached, bounded context so a cancelled r.ctx
@@ -851,10 +869,14 @@ func (r *Reporter) parseLogRow(entry *log.Entry) *runnerv1.LogRow {
func (r *Reporter) newLogRow(t *timestamppb.Timestamp, content string) *runnerv1.LogRow { func (r *Reporter) newLogRow(t *timestamppb.Timestamp, content string) *runnerv1.LogRow {
return &runnerv1.LogRow{ return &runnerv1.LogRow{
Time: t, Time: t,
Content: strings.ToValidUTF8(r.logReplacer.Replace(content), "?"), Content: r.mask(content),
} }
} }
func (r *Reporter) mask(content string) string {
return strings.ToValidUTF8(r.logReplacer.Replace(content), "?")
}
func (r *Reporter) addMask(msg string) { func (r *Reporter) addMask(msg string) {
r.oldnew = runner.AppendSecretMasker(r.oldnew, msg) r.oldnew = runner.AppendSecretMasker(r.oldnew, msg)
r.logReplacer = strings.NewReplacer(r.oldnew...) r.logReplacer = strings.NewReplacer(r.oldnew...)
+7 -12
View File
@@ -1041,18 +1041,13 @@ func TestReporter_StopHeartbeats(t *testing.T) {
"Close() must still send a final UpdateTask after StopHeartbeats") "Close() must still send a final UpdateTask after StopHeartbeats")
} }
func TestAppendIfNotNil(t *testing.T) { func TestAppendLogRow(t *testing.T) {
var s []*int r := &Reporter{}
s = appendIfNotNil(s, nil) row := &runnerv1.LogRow{Time: timestamppb.Now(), Content: "hello"}
assert.Empty(t, s) r.appendLogRow(nil)
r.appendLogRow(row)
v := 7 r.appendLogRow(nil)
s = appendIfNotNil(s, &v) assert.Equal(t, []*runnerv1.LogRow{row}, r.logRows)
require.Len(t, s, 1)
assert.Equal(t, &v, s[0])
s = appendIfNotNil(s, nil)
require.Len(t, s, 1)
} }
func TestReporter_Levels(t *testing.T) { func TestReporter_Levels(t *testing.T) {