mirror of
https://github.com/AyuGram/AyuGramDesktop.git
synced 2026-08-13 15:03:36 +00:00
[ai] Attempt to make testing faster.
This commit is contained in:
@@ -194,6 +194,15 @@ writing any overlay:
|
||||
a reason). Do not stop at one or two.
|
||||
4. **Write the checks into `<WORK_DIR>/test.md` BEFORE running** (format under "Test report"), so the
|
||||
design is explicit and Actual/Result can be filled in per check afterward.
|
||||
5. **Run economy — plan ONE run.** A test run costs a build, an app launch, and an assessment pass,
|
||||
so compress the whole design into a single programmed execution: one scenario that steps through
|
||||
every check on the event loop, emitting per-check markers and saving every log value, measurement,
|
||||
and tight screenshot needed to judge all of them afterwards. Order steps so earlier ones do not
|
||||
destroy later fixtures. Plan a second run only when two checks genuinely cannot share one process
|
||||
lifetime (mutually exclusive fixtures or settings, state one check needs fresh that another
|
||||
necessarily contaminates) — never for scenario simplicity. Unplanned re-runs stay what the state
|
||||
machine allows: a TEST_FLAW re-run or the next attempt after an IMPL_BUG fix — and a TEST_FLAW
|
||||
re-author fixes every flaw observed in that run in one pass, not one flaw per relaunch.
|
||||
|
||||
## Visual contract (layout tasks)
|
||||
|
||||
@@ -250,9 +259,10 @@ How TEST verifies it (numbers over eyes):
|
||||
|
||||
The overlay is ad-hoc, authored fresh against the CURRENT implementation, injected at the
|
||||
highest level that still exercises the change (often a direct data-layer call like
|
||||
`item->applyEdition(...)` rather than a faked MTP response). It is also a complete runtime driver
|
||||
when external desktop control is unavailable. Drive the whole task-specific flow inside the Debug
|
||||
binary by invoking application actions or posting Qt input events on the event loop, waiting for
|
||||
`item->applyEdition(...)` rather than a faked MTP response). It is also the complete runtime driver
|
||||
of first resort: prefer programmatically triggering every required action and judging the saved logs
|
||||
and captures afterwards over any external desktop driver, whether or not one is available. Drive the
|
||||
whole task-specific flow inside the Debug binary by invoking application actions or posting Qt input events on the event loop, waiting for
|
||||
observable state, logging assertions, capturing the rendered target in-process, and quitting. A
|
||||
locked macOS session does not reduce required coverage and is never a testing blocker. The overlay
|
||||
must:
|
||||
|
||||
@@ -20,8 +20,11 @@ authoritative.
|
||||
|
||||
Resolve one policy from the user's request and pass it to the task-runner:
|
||||
|
||||
- `auto` (default) — let the test-author choose hybrid driving only where real pointer, keyboard,
|
||||
focus, scrolling, dragging, menus, windowing, or native UI materially improves coverage.
|
||||
- `auto` (default) — overlay-only unless a check's tested subject IS the physical interaction.
|
||||
Choose hybrid only when the acceptance criteria name behavior that solely real pointer, keyboard,
|
||||
focus, scrolling, dragging, menus, windowing, or native-UI mechanics can exercise; "improves
|
||||
coverage" or "more end-to-end" never qualifies. Expect most tasks, including most UI tasks, to
|
||||
verify fully overlay-only.
|
||||
- `overlay-only` — never use Computer Use.
|
||||
- `required` — use hybrid driving for the named flow; if it cannot run safely, return
|
||||
`BLOCKED(test)` with the exact missing interaction rather than weakening the oracle, except for
|
||||
@@ -29,8 +32,9 @@ Resolve one policy from the user's request and pass it to the task-runner:
|
||||
|
||||
For each check, select `Driver: overlay` or `Driver: hybrid`. Keep overlay-only for internal state,
|
||||
data, exact text, and geometry that the in-app harness can exercise deterministically. Select hybrid
|
||||
when the user-input path itself matters. Do not approve a runnable code task from a Computer Use
|
||||
narrative or an uninstrumented click-through. Even a hybrid test retains the overlay for fixture
|
||||
only when the user-input path is itself the tested aspect, and fold every hybrid step into the same
|
||||
planned run as the overlay checks; hybrid never justifies extra runs. Do not approve a runnable code
|
||||
task from a Computer Use narrative or an uninstrumented click-through. Even a hybrid test retains the overlay for fixture
|
||||
setup, semantic assertions or geometry capture, watchdog, terminal markers, and tight screenshots.
|
||||
|
||||
Have the test-author add these fields to every hybrid check in `test.md` before the run:
|
||||
|
||||
@@ -304,6 +304,10 @@ rules, with these external-task safety adaptations:
|
||||
- Enforce both the in-app watchdog and an external wall-clock deadline. Count
|
||||
test runs independently from implementation attempts and stop at
|
||||
`MAX_TEST_RUNS`.
|
||||
- Plan the fewest possible runs: one complete programmed scenario per attempt
|
||||
that proves every check in a single execution, splitting only for checks
|
||||
that cannot share one process lifetime. `MAX_TEST_RUNS` is a safety cap,
|
||||
never a budget to spend.
|
||||
- Delete the overlay-bearing Debug executable on every terminal test exit so
|
||||
the user cannot launch it accidentally.
|
||||
|
||||
@@ -313,7 +317,8 @@ contiguous run this is the `BASE_REF..GREEN_REF` diff; for a resumed older task,
|
||||
combine the exact task commits and inspect their current code at `RUN_REF`
|
||||
without treating intervening tasks as this task's changes. It writes checks
|
||||
before running, covers every acceptance surface, declares a falsifiable oracle
|
||||
for each, and never reuses a generic navigate-and-screenshot scenario. Missing
|
||||
for each, compresses all checks into the fewest possible runs — normally
|
||||
exactly one — and never reuses a generic navigate-and-screenshot scenario. Missing
|
||||
or ambiguous evidence is `TEST_FLAW`; no expected task delta is `IMPL_BUG`. Two
|
||||
identical consecutive failure signatures block early. A known implementation
|
||||
bug at the attempt cap is implementation-blocked, not a successful retained
|
||||
|
||||
Reference in New Issue
Block a user