[ai] Attempt to make testing faster.

This commit is contained in:
John Preston
2026-07-20 17:53:45 +04:00
parent bd67b2feb9
commit 4e4d79ffb1
3 changed files with 27 additions and 8 deletions
+13 -3
View File
@@ -194,6 +194,15 @@ writing any overlay:
a reason). Do not stop at one or two.
4. **Write the checks into `<WORK_DIR>/test.md` BEFORE running** (format under "Test report"), so the
design is explicit and Actual/Result can be filled in per check afterward.
5. **Run economy — plan ONE run.** A test run costs a build, an app launch, and an assessment pass,
so compress the whole design into a single programmed execution: one scenario that steps through
every check on the event loop, emitting per-check markers and saving every log value, measurement,
and tight screenshot needed to judge all of them afterwards. Order steps so earlier ones do not
destroy later fixtures. Plan a second run only when two checks genuinely cannot share one process
lifetime (mutually exclusive fixtures or settings, state one check needs fresh that another
necessarily contaminates) — never for scenario simplicity. Unplanned re-runs stay what the state
machine allows: a TEST_FLAW re-run or the next attempt after an IMPL_BUG fix — and a TEST_FLAW
re-author fixes every flaw observed in that run in one pass, not one flaw per relaunch.
## Visual contract (layout tasks)
@@ -250,9 +259,10 @@ How TEST verifies it (numbers over eyes):
The overlay is ad-hoc, authored fresh against the CURRENT implementation, injected at the
highest level that still exercises the change (often a direct data-layer call like
`item->applyEdition(...)` rather than a faked MTP response). It is also a complete runtime driver
when external desktop control is unavailable. Drive the whole task-specific flow inside the Debug
binary by invoking application actions or posting Qt input events on the event loop, waiting for
`item->applyEdition(...)` rather than a faked MTP response). It is also the complete runtime driver
of first resort: prefer programmatically triggering every required action and judging the saved logs
and captures afterwards over any external desktop driver, whether or not one is available. Drive the
whole task-specific flow inside the Debug binary by invoking application actions or posting Qt input events on the event loop, waiting for
observable state, logging assertions, capturing the rendered target in-process, and quitting. A
locked macOS session does not reduce required coverage and is never a testing blocker. The overlay
must:
@@ -20,8 +20,11 @@ authoritative.
Resolve one policy from the user's request and pass it to the task-runner:
- `auto` (default) — let the test-author choose hybrid driving only where real pointer, keyboard,
focus, scrolling, dragging, menus, windowing, or native UI materially improves coverage.
- `auto` (default) — overlay-only unless a check's tested subject IS the physical interaction.
Choose hybrid only when the acceptance criteria name behavior that solely real pointer, keyboard,
focus, scrolling, dragging, menus, windowing, or native-UI mechanics can exercise; "improves
coverage" or "more end-to-end" never qualifies. Expect most tasks, including most UI tasks, to
verify fully overlay-only.
- `overlay-only` — never use Computer Use.
- `required` — use hybrid driving for the named flow; if it cannot run safely, return
`BLOCKED(test)` with the exact missing interaction rather than weakening the oracle, except for
@@ -29,8 +32,9 @@ Resolve one policy from the user's request and pass it to the task-runner:
For each check, select `Driver: overlay` or `Driver: hybrid`. Keep overlay-only for internal state,
data, exact text, and geometry that the in-app harness can exercise deterministically. Select hybrid
when the user-input path itself matters. Do not approve a runnable code task from a Computer Use
narrative or an uninstrumented click-through. Even a hybrid test retains the overlay for fixture
only when the user-input path is itself the tested aspect, and fold every hybrid step into the same
planned run as the overlay checks; hybrid never justifies extra runs. Do not approve a runnable code
task from a Computer Use narrative or an uninstrumented click-through. Even a hybrid test retains the overlay for fixture
setup, semantic assertions or geometry capture, watchdog, terminal markers, and tight screenshots.
Have the test-author add these fields to every hybrid check in `test.md` before the run:
@@ -304,6 +304,10 @@ rules, with these external-task safety adaptations:
- Enforce both the in-app watchdog and an external wall-clock deadline. Count
test runs independently from implementation attempts and stop at
`MAX_TEST_RUNS`.
- Plan the fewest possible runs: one complete programmed scenario per attempt
that proves every check in a single execution, splitting only for checks
that cannot share one process lifetime. `MAX_TEST_RUNS` is a safety cap,
never a budget to spend.
- Delete the overlay-bearing Debug executable on every terminal test exit so
the user cannot launch it accidentally.
@@ -313,7 +317,8 @@ contiguous run this is the `BASE_REF..GREEN_REF` diff; for a resumed older task,
combine the exact task commits and inspect their current code at `RUN_REF`
without treating intervening tasks as this task's changes. It writes checks
before running, covers every acceptance surface, declares a falsifiable oracle
for each, and never reuses a generic navigate-and-screenshot scenario. Missing
for each, compresses all checks into the fewest possible runs — normally
exactly one — and never reuses a generic navigate-and-screenshot scenario. Missing
or ambiguous evidence is `TEST_FLAW`; no expected task delta is `IMPL_BUG`. Two
identical consecutive failure signatures block early. A known implementation
bug at the attempt cap is implementation-blocked, not a successful retained