diff --git a/.agents/shared/test-loop.md b/.agents/shared/test-loop.md index fa8d427610..735d21e527 100644 --- a/.agents/shared/test-loop.md +++ b/.agents/shared/test-loop.md @@ -194,6 +194,15 @@ writing any overlay: a reason). Do not stop at one or two. 4. **Write the checks into `/test.md` BEFORE running** (format under "Test report"), so the design is explicit and Actual/Result can be filled in per check afterward. +5. **Run economy — plan ONE run.** A test run costs a build, an app launch, and an assessment pass, + so compress the whole design into a single programmed execution: one scenario that steps through + every check on the event loop, emitting per-check markers and saving every log value, measurement, + and tight screenshot needed to judge all of them afterwards. Order steps so earlier ones do not + destroy later fixtures. Plan a second run only when two checks genuinely cannot share one process + lifetime (mutually exclusive fixtures or settings, state one check needs fresh that another + necessarily contaminates) — never for scenario simplicity. Unplanned re-runs stay what the state + machine allows: a TEST_FLAW re-run or the next attempt after an IMPL_BUG fix — and a TEST_FLAW + re-author fixes every flaw observed in that run in one pass, not one flaw per relaunch. ## Visual contract (layout tasks) @@ -250,9 +259,10 @@ How TEST verifies it (numbers over eyes): The overlay is ad-hoc, authored fresh against the CURRENT implementation, injected at the highest level that still exercises the change (often a direct data-layer call like -`item->applyEdition(...)` rather than a faked MTP response). It is also a complete runtime driver -when external desktop control is unavailable. Drive the whole task-specific flow inside the Debug -binary by invoking application actions or posting Qt input events on the event loop, waiting for +`item->applyEdition(...)` rather than a faked MTP response). It is also the complete runtime driver +of first resort: prefer programmatically triggering every required action and judging the saved logs +and captures afterwards over any external desktop driver, whether or not one is available. Drive the +whole task-specific flow inside the Debug binary by invoking application actions or posting Qt input events on the event loop, waiting for observable state, logging assertions, capturing the rendered target in-process, and quitting. A locked macOS session does not reduce required coverage and is never a testing blocker. The overlay must: diff --git a/.agents/skills/perform-task/references/computer-use-testing.md b/.agents/skills/perform-task/references/computer-use-testing.md index a0e27274b6..68ddadaf91 100644 --- a/.agents/skills/perform-task/references/computer-use-testing.md +++ b/.agents/skills/perform-task/references/computer-use-testing.md @@ -20,8 +20,11 @@ authoritative. Resolve one policy from the user's request and pass it to the task-runner: -- `auto` (default) — let the test-author choose hybrid driving only where real pointer, keyboard, - focus, scrolling, dragging, menus, windowing, or native UI materially improves coverage. +- `auto` (default) — overlay-only unless a check's tested subject IS the physical interaction. + Choose hybrid only when the acceptance criteria name behavior that solely real pointer, keyboard, + focus, scrolling, dragging, menus, windowing, or native-UI mechanics can exercise; "improves + coverage" or "more end-to-end" never qualifies. Expect most tasks, including most UI tasks, to + verify fully overlay-only. - `overlay-only` — never use Computer Use. - `required` — use hybrid driving for the named flow; if it cannot run safely, return `BLOCKED(test)` with the exact missing interaction rather than weakening the oracle, except for @@ -29,8 +32,9 @@ Resolve one policy from the user's request and pass it to the task-runner: For each check, select `Driver: overlay` or `Driver: hybrid`. Keep overlay-only for internal state, data, exact text, and geometry that the in-app harness can exercise deterministically. Select hybrid -when the user-input path itself matters. Do not approve a runnable code task from a Computer Use -narrative or an uninstrumented click-through. Even a hybrid test retains the overlay for fixture +only when the user-input path is itself the tested aspect, and fold every hybrid step into the same +planned run as the overlay checks; hybrid never justifies extra runs. Do not approve a runnable code +task from a Computer Use narrative or an uninstrumented click-through. Even a hybrid test retains the overlay for fixture setup, semantic assertions or geometry capture, watchdog, terminal markers, and tight screenshots. Have the test-author add these fields to every hybrid check in `test.md` before the run: diff --git a/.agents/skills/perform-task/references/pipeline.md b/.agents/skills/perform-task/references/pipeline.md index 52653ad166..e4f03dc82c 100644 --- a/.agents/skills/perform-task/references/pipeline.md +++ b/.agents/skills/perform-task/references/pipeline.md @@ -304,6 +304,10 @@ rules, with these external-task safety adaptations: - Enforce both the in-app watchdog and an external wall-clock deadline. Count test runs independently from implementation attempts and stop at `MAX_TEST_RUNS`. +- Plan the fewest possible runs: one complete programmed scenario per attempt + that proves every check in a single execution, splitting only for checks + that cannot share one process lifetime. `MAX_TEST_RUNS` is a safety cap, + never a budget to spend. - Delete the overlay-bearing Debug executable on every terminal test exit so the user cannot launch it accidentally. @@ -313,7 +317,8 @@ contiguous run this is the `BASE_REF..GREEN_REF` diff; for a resumed older task, combine the exact task commits and inspect their current code at `RUN_REF` without treating intervening tasks as this task's changes. It writes checks before running, covers every acceptance surface, declares a falsifiable oracle -for each, and never reuses a generic navigate-and-screenshot scenario. Missing +for each, compresses all checks into the fewest possible runs — normally +exactly one — and never reuses a generic navigate-and-screenshot scenario. Missing or ambiguous evidence is `TEST_FLAW`; no expected task delta is `IMPL_BUG`. Two identical consecutive failure signatures block early. A known implementation bug at the attempt cap is implementation-blocked, not a successful retained