testing.nvim · Debug & inspect · vimdoc

:help testing

Test runner and orchestration for Neovim plugins (pre-alpha)

doc/testing.txt — rendered from the plugin's own vimdoc

*testing.txt*  Test runner and orchestration for Neovim plugins (pre-alpha)

Author: Stefan Bartl
License: MIT

CONTENTS *testing-contents*

  1. Introduction ........................................ |testing-intro|
  2. Requirements ........................................ |testing-requirements|
  3. Setup ............................................... |testing-setup|
  4. Configuration ....................................... |testing-config|
  5. Commands ............................................ |testing-commands|
  6. Keymaps and autocommands ............................ |testing-bindings|
  7. Health check ........................................ |testing-health|
  8. Command-line runner ................................. |testing-cli|
  9. Project file .testing.lua ........................... |testing-project|
 10. Exit codes .......................................... |testing-exit-codes|
 11. Dialects ............................................ |testing-dialects|
 12. Reporters and the Result-IR ......................... |testing-reporters|
 13. Migrating from busted or plenary .................... |testing-migrating|

1. Introduction *testing-intro*

testing.nvim is a test runner for Neovim plugins. It runs existing spec files
unchanged (the return function(H) styles of the author's plugins and busted
style describe / it specs, without plenary or busted), reports through a
terminal reporter, GitHub annotations, JUnit XML and a JSON result format, and
refuses to report a green run it cannot stand behind.

This release is pre-alpha. Spec files can run in a child editor of their own,
guards name what a spec leaves behind, unchanged spec files can come from a
result cache, and there is no editor UI beyond |:Testing|. The long form of this
help is in the repository's docs/ directory.

2. Requirements *testing-requirements*

  - Neovim 0.10 or newer
  - lib.nvim (hard dependency): https://github.com/WKDSteve/lib.nvim, at
    least the commits 6304829 (fs.write.atomic) and 89cb912 (safe_call keeps
    non-string errors)

The command-line runner looks for lib.nvim in four places, in this order:
$LIB_NVIM_DIR, <repo>/.deps/lib.nvim, <repo>/../lib.nvim and
stdpath('data')/lazy/lib.nvim. If it finds none, it names all four and exits
with 3. An override ($LIB_NVIM_DIR) that is set but invalid is a failure, never
skipped.

3. Setup *testing-setup*

With lazy.nvim:
    {
      "WKDSteve/testing.nvim",
      dependencies = { "WKDSteve/lib.nvim" },
      cmd = "Testing",
      opts = {},
    }
Without a plugin manager spec, call:
    require("testing").setup({})
setup() is safe to call twice. The plugin file registers :Testing on its own,
so cmd = "Testing" lazy-loading works without setup().

4. Configuration *testing-config*

These are the options of setup(). The per-project file of the command-line
runner is a different thing, see |testing-project|. All keys are optional. A key
with the wrong type, and any unknown key, is dropped and reported with a
warning; the default stays in place.

                                                          *testing-notify_prefix*

notify_prefix

    Type: string, non-empty. Default: "[testing]".
    Prefix of every message the plugin shows.

                                                              *testing-keymaps*

keymaps

    Type: table<string, string|string[]|false>|false. Default: {}.
    Overrides for the plugin's named keymap actions: action name -> new left-hand
    side, a list of them, or false to drop that action. false instead of a
    table binds no key at all. The one action, hover, has no default key:
    keymaps = { hover = "<leader>th" } binds it, so nothing is bound by default.

                                                                *testing-hover*

hover

    Type: { max_width?, max_height?, max_diff_lines? } (positive integers).
    Default: { max_width = 80, max_height = 20, max_diff_lines = 12 }.
    Size of the hover over a test (:Testing hover, hover.nvim :Hover show):
    longest line in display columns, most lines, lines of a failure diff.

                                                        *testing-surface_view*

surface_view

    Type: { keys?: { jump?, status?, kind?, export? } }, each an lhs or false.
    Default: { keys = { jump = "<CR>", status = "s", kind = "t", export = "e" } }.
    Keys of the matrix view (|testing-matrix-view|), only in that view;
    false leaves a key out. Legend, ? cheatsheet and keymaps come from
    this one table.

                                                                 *testing-lint*

lint

    Type: { on_save?: boolean }. Default: { on_save = false }.
    on_save = true: a written file is linted in the background (the lint
    checks that may run on save, never luals) and only its diagnostics are
    updated, see |:Testing-lint|.

                                                                 *testing-tree*

tree

    Type: boolean | { keys?: { run?, failed? } }, each an lhs, a list of them
    or false. Default: true.
    The last result in a tree view that reads lib.nvim.tree.registry
    (filetree.nvim's own tree): a badge next to spec files and the view
    tests (folder, spec file, describe, case; :Filetree source tests).
    In that view r runs what the node stands for and R what failed last
    time; keys moves them, false leaves an action in the menu only.
    tree = false registers nothing. See docs/STATE.md, "In a tree view".

5. Commands *testing-commands*

                                                                       *:Testing*
:Testing [run] [{root}] [--file=..] [--filter=..] [--reporter=..] [--cached]
        [--no-cache] [--changed] [--since={rev}] [--shard={i}/{n}]
        Run the specs of the project in a headless child Neovim, never in this
        editor. {root} defaults to the nearest directory upwards that holds
        .testing.lua, TESTS or .git. --cached skips spec files whose
        inputs did not change since a green run (reported as cached);
        --changed and --since run only the specs the changes can reach (a
        partial run, never the "all green" last line).
:Testing file [{spec}]    Run the spec of the current buffer.
:Testing last             Repeat the last run.
:Testing hover            Last result of the test under the cursor (float); also hover.nvim
                         :Hover show over a test file, a describe
                         block or a spec path in any buffer. With the hover
                         keymap action (no default key) on a key of your own.
                         A buffer with unsaved changes is worded as outdated;
                         only _spec.lua files and files with a recorded
                         result answer. hover.nvim is never required by
                         setup(): a retry in the augroup
                         TestingHoverRegister waits for it.
:Testing terminal         Repeat the last run in a terminal.nvim terminal.
:Testing list [{root}]    List what would run.
:Testing init [{root}] [--force] [--plugin={name}] [--workflow={kind}]
        Generate the test setup of a plugin repository. An existing file is
        never overwritten without --force. --workflow=two-runner,
        two-runner-private or hosted writes only .github/workflows/ci.yml
        in that layout of the generator (the local runner plus a GitHub-hosted
        macOS leg, or hosted runners only). --workflow=public generates
        that file from ci.workflow of .testing.lua.
:Testing conformance [{root}] [--gate] [--only={ids}] [--skip={ids}] [--bridge]
        Run the conformance checks K1..K16 on the project (a report; a failed
        gate is a warning, not an error).
:Testing surface [{root}] [--from={ir.json}] [--threshold={n}] [--markdown]
        [--matrix] [--csv] [--no-tags]
        The keymaps, commands and autocmds of the plugin and how much of them
        the specs exercised. --baseline={file} compares with a baseline;
        with --require-signed-baseline a baseline that was edited or has
        no digest fails the run (exit 1). --matrix shows entries x spec
        files, --csv a tab separated table; binding:/command: tags and
        -- @covers headers of passing cases count as exercised (not with
        --no-tags). --matrix alone opens the matrix view (see
        |testing-matrix-view|).

                                                        *testing-matrix-view*
The matrix view of :Testing surface --from={ir.json} --matrix is a ui.kit
viewer, one line per entry, one column per spec file. Its keys (buffer-local,
from |testing-surface_view|):
    <CR>    on a spec file column: that spec file; elsewhere on a row: the
            source of the entry at its line
    s       status filter: all, hit, missing, untracked, ignored
    t       kind filter: all, binding, command, autocmd
    e       export the shown rows as a tab separated file (asks for a path)
    ?       the cheatsheet; q and <Esc> close
:Testing budget [{root}] [--update] [--factor={x}] [--allow-new]
        Measure the performance budgets of the runner. A measured case
        without a baseline entry compared nothing: exit 2 unless
        --allow-new.
                                                                 *:Testing-act*
:Testing act [--root={dir}]
        A finding of the last run to a task, a rule or the AI (testing act).
        The target is the current quickfix entry, else the line under the
        cursor, else a choice. Then two steps: what (task, rule, AI) and how
        (form, AI draft, agent; skipped when the target takes one). A task and
        a rule are a dry run of the target's engine first, then a question;
        a duplicate names the task that is already there. The AI gets a
        hand-over file in testing's state folder and the command for Claude
        Code in register +; nothing is sent or started.
                                                              *:Testing-export*
:Testing export [{dir}] [--repos-root={dir}] [--remote=always|never|ask]
        Write the export folder testing-web reads, in a headless child
        (testing export). Runs no suite and no pipeline. {dir} defaults to
        the setting ci.export_dir, the repositories to ci.repos.roots,
        --remote to ci.export_remote. Choose that folder in the page.
                                                                *:Testing-lint*
:Testing lint [{check}...] [--file[={a,b}]|--changed|--all] [--fix] [--root={dir}]
        Run the lint checks (stylua, luacheck, luals: the lint steps of the
        ci pipeline, or the named ones) in a headless child
        (testing lint --json). Default: the files changed against HEAD and
        the untracked ones; a bare --file the file of the current buffer,
        --file=a.lua,b.lua (repeatable) the named files, --all
        everything the step checks in testing ci. The findings fill the
        quickfix list "testing lint" (opened only when there are some) and
        become |vim.diagnostic| in the loaded buffers, one namespace per check
        (testing.lint.{check}); a checked file without findings is cleared.
        --fix rewrites what stylua can fix and reloads the buffers (refused
        while a buffer in its scope has unsaved changes). A missing
        tool skips its check and is named in the message (exit 3, never green).
        On the command line: testing lint --help.
:Testing cache stats | clear [{root}]
        Size of the result cache of the project, or delete it.
:Testing health           Run :checkhealth testing.
:Testing config           Show the effective plugin configuration.
:Testing doctor [{root}]  Show the project configuration and the dependency
                          report.

Subcommands and flags are completed with <Tab>. The complete reference is
docs/BINDINGS.md.

6. Keymaps and autocommands *testing-bindings*

The plugin binds no key and registers no autocommand by default. Its one keymap
action, hover (:Testing hover), has no default key; keymaps = { hover = "<lhs>" }
binds it. Every action can be moved or dropped through the keymaps option.

7. Health check *testing-health*

:checkhealth testing reports, in separate sections: the Neovim version; the
lib.nvim modules the plugin calls, plus two behaviors it relies on (safe_call
keeps non-string errors, fs.write.atomic refuses an invalid call), each probed
without writing anything; whether the kernel modules load; the dialect and
reporter registries; whether the command-line runner can find lib.nvim; the
.testing.lua of the current directory; the guards and the warm pool as that
file configures them (modes, allowances, pool size; a warning when every guard
is off, an error when a module of the isolation machinery does not load); the
effective configuration; whether the result cache directory can be written, whether
the docmap.nvim module graph of the affected selection is available and
current, and whether the conformance suite and the surface report load; and
whether |:Testing| is registered.

It is read-only and loads no plugin that is lazy-loaded; the processes it
starts are the version calls of the lint tools and one question to the
container engine (whether the image of testing.container is built, 5 s at
most).
.testing.lua is a Lua file and loading it executes it, so the health check
does not: it evaluates the file in an empty environment with an instruction
budget. A plain table is validated for real; a file that needs more is reported
as "not evaluated here" (never as valid). testing doctor evaluates it.

A level never contradicts its text: an error is something broken, a warning is
something that will fail elsewhere (for example the command-line runner not
finding lib.nvim while the editor works), an info is something not checked.

8. Command-line runner *testing-cli*

    nvim -n -i NONE --headless -u NONE -l scripts/testing.lua [run|fast|list|doctor|budget|conformance|surface|explain|stamp|verify] [{root}] [options]
    nvim -n -i NONE --headless -u NONE -l scripts/testing.lua migrate [dry-run|apply] [{path}] [options]
Run it from {root}: specs that look at "this repository" read the working
directory and the runner does not change it. scripts/test.sh runs this
repository's own specs this way.

                                                            *testing-command*
testing {args} is the same line as a command: put the checkout's bin/
folder on the PATH (bin/testing, bin/testing.cmd on Windows). It hands
the exit code through; without nvim on the PATH it says so and exits 3.

Subcommands: run (default), list (= run --list), doctor (resolved
configuration and dependency report), migrate (plan or write the move of a
repository to testing.nvim, see |testing-migrating|), budget (measure the hot
paths of a run against a baseline), conformance (the checks K1..K16) and
surface (the plugin's keymaps, commands, ... and what the specs exercised);
those two have their own arguments and exit codes (docs/CONFORMANCE.md,
docs/SURFACE.md). explain says why a spec file is selected, taken from the
cache, run or left out, and what its cache key is made of (see "Explain and
audit" below). stamp runs the suite and, after a complete green run, writes
the green stamp; verify answers from the cache keys, without running a spec,
whether the tree is still the stamped one (see "Stamp and verify" below).
init is available as |:Testing| init only. ci runs the pipeline of
.testing.lua locally; ci remote runs nothing and reports the latest GitHub
Actions run of the CI workflow per repository and system (docs/CI.md, "The
GitHub Actions status"; --history[=N] reads the last N runs and tells when
one system was red alone; --wait[=<duration>] asks again every 30 seconds
until no run of the head commits is going, at most 60 minutes or the duration,
past that exit 1 "not finished"); ci overview runs nothing either and writes one
static HTML page of the fleet from ci verify, ci.json and that adapter
(docs/CI.md, "Fleet overview").
ci commit {rev} checks a commit in a git worktree of its own (another
session's uncommitted work does not count) and sorts red into new, known and
fixed against the parent: clean and known-red exit 0, regression 1
(docs/CI.md, "Check a commit"). ci commit --repos-root={dir} --since={time} does it for the new commits of every
repository below a directory. From the editor: :Testing ci commit (HEAD or the
revisions given, --since, --repos-root; the quickfix list holds only the new
findings). ci commit {rev} --quick runs only the spec files the diff to the
base reaches (CLEAN quick); it is no proof of the tree: before the push one
full ci commit HEAD --base=@{upstream} -- --cached. On Windows --wsl runs
the check in WSL2 (about ten times faster; docs/CI.md, "Check a commit").
ci land [{rev}] pushes a commit to a check branch ci/<repo>-<sha>, waits
for the hosted CI and moves main to it only when that is green (needs
ci.workflow.land = true; docs/CI.md, "Land through the hosted CI").
Heavy runs (a whole suite, ci with the whole suite, ci commit) take turns
on one machine: a second one waits and says for what (docs/CI.md, "One heavy
run at a time"). TESTING_QUEUE=off turns that off.
stats [--since=7d] [--repo=<name>] [--json] sums up what the heavy runs of
this machine cost (one line each in metrics.jsonl; docs/CI.md, "Metrics").

Workspace trust ~                                             *testing-trust*
The runner executes a project's .testing.lua, minimal init and specs, so it
runs only a trusted root; any other is exit 2 before anything of it runs. A
root is trusted when CI is set (not false/0), TESTING_TRUST=1, the
root lies below a directory of TESTING_TRUST_ROOTS (a list like PATH), or
the store (stdpath("state")/testing/trust.json) has it with the hash of its
current .testing.lua (a changed file asks again). --trust on any command
stores the root and goes on; testing trust list|check|add|revoke [{root}]
manages the store. From the editor, |:Testing| and :Testing ci ask once
(vim.ui.select) instead: yes stores the root and runs again, no leaves the
refusal; a headless nvim is never asked, :Testing ci repos never asks.
Details: docs/CONFIG.md, "Workspace trust".

Selection

  --file {text}      Only spec files whose file name contains {text}
                     (repeatable). --only is an alias.
  --filter {text}    Only cases whose name contains {text} (literal).
  --tags {a,b}       Only cases with one of these tags: a #word in a busted
  --exclude-tags     title, or a -- @tags a b header line in any spec.
  --lf, --ff         Only what failed last time, or that first.
  --order priority   Run first what failed last time, what the changes touch
                     and reach (nearest first), what has not run for long.
                     Orders only, never filters; excludes --shuffle.
  --time-budget {t}  Stop starting spec files after {t} (90s, 2m, 1h). A running
                     file is never cut short. The rest is "not run (budget)":
                     verdict green-partial (exit 0, never the sentinel), or red.
  --skip-uncacheable With the cache: leave out files that have no cache key, on
                     purpose: green-partial, never the sentinel. Refused (exit
                     2) when CI is set.
  --order slowest-first
                     With --jobs 2 or more and child editors: the longest
                     files (remembered duration) start first. Results stay in
                     file order.
  --retry-failed {n} Run a red case again up to {n} times (1 to 10). A case
                     that passes on a retry is flaky: the run stays red, it is
                     listed and never cached.
  --allow-flaky      With --retry-failed: count such a case as green (still
                     listed). The run is then green-partial: no sentinel.
  --list, --dry-run  List what would run, run nothing.
  --shard {i}/{n}    Only shard {i} of {n} (CI matrices).
  --changed, --since {rev}, --affected[={rev}]
                     Only the specs the changes can reach (git, working tree
                     against HEAD, against {rev}, or the last commit). A
                     partial run: no "all green" last line. Anything that
                     cannot be placed selects every spec. Never the default.
  --consumers {dir}
                     With one of the three above: name the specs of the
                     checkouts below {dir} (projects that use this one) that
                     the change reaches (docmap.nvim). A hint, it never
                     changes what runs here. Also affected.consumers in
                     .testing.lua.
  --state-dir {dir}  State directory of the run history (default stdpath('state')).
                     scripts/test.sh hands the real one down; the specs keep
                     a throwaway one.
  --cache-dir {dir}  Base directory of the result cache (default stdpath('cache');
                     env TESTING_CACHE_HOME; the flag wins). `cache = {
                     project_key = ... }` in .testing.lua names the project
                     folder after a key instead of the checkout path
                     (docs/CI-CACHE.md).
  --cached, --no-cache, --cache-refresh, --cache-clear
                     The result cache: a spec file whose inputs are
                     byte-identical to an earlier green run does not run and is
                     reported as cached. Off unless asked for; --no-cache
                     always wins. docs/CACHE.md.
  --cache-audit {0..1|all}
                     Run that share of the cache hits anyway and compare with
                     the stored result. A difference is the finding
                     cache.stale_pass and exit 1; the measured stale-pass
                     rate is on the cache line and in run.cache of the IR.
                     Implies --cached; excludes --cache-refresh.
  --config {file}    Project file inside the root.
  --rtp {dir}        Add {dir} to the runtimepath (repeatable).

Control

  -x, --maxfail {n}  Stop after the first / {n}th failure.
  --shuffle, --seed {n}
                     Random order; the seed is printed.
  --case-timeout {ms}, --file-timeout {ms}
                     Timeouts. In this editor best effort (Lua code and
                     vim.wait, not a spec that blocks inside C); in a child
                     editor hard: the child and its process tree are killed.
  --strict           A skipped case or a discovery finding makes the run red.
  --isolated {none|file|case|soft}
                     file: every spec file in a child editor of its own (a
                     crash, a hang or a prompt ends that file only). case:
                     a child per case (busted). soft: this editor, what a
                     file changed is restored before the next. Default
                     auto: busted files file, the others none; a
                     script is always a child.
  --jobs {n|auto}    Child editors at once; the report is the same for any n.
  --watch            Run, then re-run on every change.
  --watch-max-wait {ms}  Longest a change waits before a re-run (cooldown).
  --profile          Where the time went (stderr; run.profile in the IR).
  --pool-reuse, --no-pool-reuse
                     The warm pool: with --isolated file the files share
                     child editors that are reset and verified clean between
                     files (a member that cannot prove it is clean is
                     replaced and a finding says why). script files and
                     case never use it.
  --pool-size {n}    Editors the pool keeps (default min(--jobs, 4)).
  --guard {name}={mode}
                     Guards: fs, state, scheduled_error, prompt, deprecation,
                     process_net (off|warn|error), clock (on|off).
  --allow-fs {path}, --allow-spawn {exe}, --allow-network {host}
                     What the guards let through (repeatable).
  --no-determinism   A child keeps the parent's LANG/LC_ALL/TZ.
  --no-trace         No trace artifact for a child that timed out or died.
  --isolate-home     A child gets a home directory of its own (HOME,
                     USERPROFILE, APPDATA, LOCALAPPDATA in its sandbox).
  --update-snapshots Rewrite screen snapshots that differ (testing.screen);
                     implies --no-cache.
  --host {c|l}       A child starts like plenary's host (c, default) or like
                     nvim -l.
  --env-allow {name} An environment variable (or PREFIX*) a child may inherit.

Output

  --reporter {name}  Terminal reporter: term, github, junit, json or agent.
                     Default: $TESTING_REPORTER, then agent in a coding
                     agent environment ($TESTING_AGENT=1 forces it, =0 stops
                     the detection), else term.
  --agent-budget {n} Characters of the failure part of agent (default 4000).
  --format {text|jsonl}  Shape of agent (jsonl: one object per line).
  --json {file}      Write the Result-IR (schema_version 1), validated.
  --junit {file}     Write a JUnit XML report.
  --events {file}    Write the events of the run (NDJSON) while it goes; - is stdout.
  --github           GitHub Actions annotations and step summary.
  --durations {n}    Name the {n} slowest cases.
  --sentinel {name}  Last line of a complete green run (default TESTING_OK).
  --no-timings       No timing line.

Explain and audit

  explain {spec}...  Say why a spec file is selected, taken from the cache, run
                     or left out: the reason of the selection, hit, miss
                     (which file, with old and new hash, which environment
                     variable by name, the Neovim version or the configuration
                     changed since the stored entry) or uncacheable (the
                     file and line of what makes it so, and the way out:
                     -- @cache-allow, -- @cache-inputs, -- @cache-env,
                     env_allow,
                     -- @cache off). {spec} is a path, a directory or a part
                     of a file name; the options of a run are accepted (the
                     key depends on them). It changes nothing.
    --all            Every spec file: the hit rate and the reasons of the
                     files without a key, most frequent first.
    --json           One JSON document (testing-explain/1).
    --parts          The lines the key is the hash of.

Stamp and verify

  stamp              A run that writes the green stamp after the verdict
                     green only (never for a partial or red run: exit 1;
                     options that select part of the suite are refused, exit
                     2). The stamp holds, per spec file, the cache key or the
                     reason there is none, plus runner, Neovim, OS and
                     configuration digests, commit and tree, all taken before
                     the first spec runs (an input that changed while the
                     specs ran: no stamp, exit 1). The report says so in its
                     first line (RED ... exit 1, verdict: red), never
                     green. No secrets.
    --out {file}     Where to write it (default: stamp.json in the state
                     directory of the project, not in the checkout).
    --note           Also attach it to the tree object as a git note under
                     refs/notes/testing.
  verify             Recompute the keys (after the minit of the project, as a
                     run does) and answer: verified (exit 0 and the
                     sentinel, only when EVERY file is proven), partial
                     (files without a key are not proven; never green),
                     changed, rejected (other runner, Neovim, OS or
                     configuration), expired, dirty, untrusted,
                     invalid, no-stamp (all exit 1).
    --stamp {file}   Read this stamp instead of the default one.
    --from-note      Read the git note of the tree of HEAD.
    --max-age {dur}  Oldest accepted stamp (7d; s, m, h, d).
    --allow-dirty    Check a dirty working tree anyway (said in the answer).
    --require-hmac   Refuse a stamp unless TESTING_STAMP_SECRET is set.
    --allow-unsigned In CI, accept a stamp without a checked HMAC (otherwise never green).
    --json           One JSON document (testing-verify/1).
                     TESTING_STAMP_SECRET (16 characters or more) adds an
                     HMAC when writing and checks it when verifying.
                     docs/CACHE.md, "Stamp".
  --cache-audit {0..1|all}
                     see Selection: a share of the hits runs anyway; a
                     difference is cache.stale_pass. A spec whose key gave
                     two different results is nondeterministic and not
                     cached (-- @cache-allow nondeterministic lifts it).
                     docs/CACHE.md.

An option that is parsed but not implemented is refused with exit code 2, never
ignored. Only a complete green run (no skipped case, no --maxfail stop, no
case accepted by --allow-flaky) prints the sentinel; anything else prints a
distinct last line. The same
distinction is the verdict line of every reporter: green, green-partial, red.

9. Project file .testing.lua *testing-project*

A Lua file in the project root that returns a table; every key is optional. It
is executed, so it is loaded only from the root named on the command line (or
--config, inside that root), as text, below 256 KiB. Invalid values and
unknown keys are warnings (the default stays); an unusable file is exit 2.

  plugin       string       Module root of the project.
  roots        string[]     Spec roots, relative. Default { "TESTS" }.
  spec_pattern string[]     Lua patterns that make a file a spec. Default
                            { "_spec%.lua$" }. A malformed pattern is a
                            warning (the default stays).
  dialect      string|table "auto" (default), "testing", "a", "b", "c", "d",
                            "h", "busted", "script"; or { [path or glob] =
                            name, ["*"] = name }.
  assertions   string       "error" (default): a case without assertions
                            fails; "warn": it passes and is listed.
  isolated     string       "auto" (default), "none", "file", "case", "soft".
  guards       table        fs, state, deprecation, scheduled_error, prompt,
                            process_net: "off"|"warn"|"error" or a table
                            { mode = ..., categories, ignore_*, keep, allow,
                            ... } (docs/GUARDS.md); clock: boolean.
                            Runner default: process_net "warn".
  guard_allow  table        { fs, spawn, network } string lists.
  pool         table        { size = n, reuse = boolean }: the warm pool.
  determinism  boolean      Fixed LANG/LC_ALL/TZ in a child. Default true.
  isolate_home boolean      A home directory in the child's sandbox. Default false.
  trace        boolean      Trace artifact of a child that died. Default true.
  soft_keep    string[]     Modules the soft isolation never unloads.
  jobs         integer|"auto" Child editors at once. Default 1.
  shard, watch, budget
               tables       --shard balance, --watch debounce, cooldown and poll,
                            testing budget factor and baseline.
  cache        table        { enabled = boolean }: reuse results without
                            --cached (never in CI).
  conformance  table        gate, skip, waivers (each with a reason),
                            keymaps_off, timeout_ms, load_budget_ms,
                            rules_bridge: testing conformance.
  surface      table        track (the runner records what the cases
                            exercise), threshold, kinds, ignore, setup_chunk.
  host         string       "c" (default) or "l".
  filetype     boolean      Children run filetype plugin indent on.
  disable_first_run boolean Set lib.nvim's first-run opt-out
                            (vim.g.lib_nvim_deps_disable_first_run) in every
                            editor the runner starts, before minit. Default
                            true.
  env_allow    string[]     Environment names a child may inherit on top of
                            the allowlist (never NVIM*, never a bare "*").
  minit        false|string Minimal init, run before the specs.
  deps         string[]     Dependencies that must resolve (exit 3 if not).
  timeouts     table        case_ms (10000), file_ms (60000); hard in a child.
  setup        table        Options setup() is called with by conformance
                            and surface.
  coverage      table       bindings, commands, autocmds: 0..1 thresholds of
                            testing surface.
  snapshots    table        dir is the default of testing.screen
                            (TESTS/__snapshots__); the runner
                            hands it to the specs; opts.dir wins.
  backends                  Validated, reserved: nothing acts on it yet.

A dependency is looked up in $<NAME>_DIR, <root>/.deps/<name>, <root>/../<name>
and stdpath('data')/lazy/<name>; the message of a miss names all four.

10. Exit codes *testing-exit-codes*

  0   Every spec file passed.
  1   A case failed, errored, timed out or crashed (a child editor that died);
      under --strict also a skip or a discovery finding.
  2   Usage or configuration error, or nothing to run (also a selection that
      matches nothing).
  3   Infrastructure error: a dependency is missing, minit raised, a report
      could not be written, the editor was quit under the run, an internal
      error.

A spec cannot end the run: os.exit inside a spec is an error of that file
(in a child editor it is a crash of that file; the others run on).

testing migrate: 0 done or nothing to do, 1 (--check) work to do, 2 refused
or bad usage, 3 the root cannot be analysed.

testing sheets report [<root>...] [--from <ir.json>]... [--out <file>]: the
checklist of live checks from the test sheets (TESTS/sheets/*.sheet.lua) and
their last results (the editor's state, or the IR of a run with --from); runs
nothing. --tasks {vault} writes the state of each sheet into its task through
tasks.nvim (--dry-run: only say what would change); a project is named
there by sheets.project of its .testing.lua, else by its folder. With a
feature catalog (TESTS/sheets/features.lua) it ends with the section
Features: the state of each functionality (the worst of its sheets), the ones
without a sheet, the sheets without a feature, and unknown features. 0 written or printed, 2 bad
usage, 3 the file could not be written. See docs/SHEETS.md.

testing sheets new --from-task {area/slug} [--repo <dir>] [--vault <dir>]:
writes a draft sheet TESTS/sheets/<slug>.sheet.lua from a live-check task
(read through tasks.nvim): a skeleton of steps and one judge entry per
acceptance checkbox. --feature {id} (with or without --from-task) sets the
sheet's feature and its title from the catalog; an id the catalog does not
know is exit 2. Never overwrites a sheet (--name for another file and
id). 0 written, 2 bad usage or the sheet exists, 3 no vault or engine, the task
could not be read, the file not written or the draft does not load.

testing sheets run --task {area/slug} [--root <dir>] [--tasks <vault>]:
runs the sheets that name the task now and writes their state into its block
(trusted roots only; a missing snapshot fails; on the workstation only
manual sheets run, unless TESTING_ALLOW_WORKSTATION=1). 0 green, 1 red or not written, 2 bad
usage, refused or no sheet, 3 the engine of tasks.nvim could not be reached.
--result (any form of sheets run) prints one more line, the last one on
stdout: a document of testing-sheet-run/1 (schemas/testing-sheet-run/1/)
with the status of each sheet, red, the outcome of each task and written,
so a program that started the run tells a red sheet from a task that was not
written (both are exit 1).
testing sheets run --all [--root <dir>] [--json <ir>] [--strict] runs every
sheet of the roots; --json writes the Result-IR, --strict makes a skip red.
testing sheets run --sheet {id}... runs only the named sheets (id or file
name), as --all would. --live (any form) lets live sheets (`live = { env,
hosts, cost }`) run with their real keys, only the named hosts and the cost
said first; never on CI, a skip without it. See docs/SHEETS.md, "Live sheets".

`testing sheets step {sheet} [--root <dir>] [--headed] [--delay <ms>]
[--break <n>]... [--run]`: one sheet step by step. Events on stdout
(testing-sheet-session/1, one JSON line each), commands on stdin (next,
continue, until <n>, note <text>, explain <n> <kind>, abort; closed
stdin is abort). explain answers with the request an AI would get about
a step, as an event for the client to show, and sends nothing. Writes
nothing but the notes of the run (beside its pictures, for the report).
--run --format=text writes one line per step instead (size, n/total,
status, ms, action and value, # note, : reason, =>  a measured value)
and the end. A measure = expr step (name, min, max, unit, runs)
reads a number in the child, takes the median of its runs, fails outside the
bounds and keeps the value as a note of the run (docs/SHEETS.md).
0 green or a skip, 1 a step failed, 2 bad usage or refused, 3 aborted.
See docs/SHEETS.md, "Step by step".

:Testing sheet step [{sheet}] is the same in a buffer: n next, c continue,
u until the cursor, b breakpoint, N note on the step, a abort, q close, y or n to
a question. E explains the step under the cursor with AI through ai.nvim (what
it does, what to watch out for, what its result means): the request is shown
before it is sent, and only sent on a yes; without ai.nvim nothing happens.

testing sheets prompt draft --from-task {id} / prompt explain --task {id}:
a Markdown hand-over for an AI (draft a sheet from the task; explain a red
run). testing sheets accept --task {id} --from <file> writes the AI's sheet
only when it loads and names the task. A vault or root naming tricentis is
refused. See docs/SHEETS.md, "Hand over to an AI".

11. Dialects *testing-dialects*

The dialect of a file is sniffed from its text (dialect = "auto") or forced.

  a, b, c   return function(H) with the helpers of lib.nvim, of
            markdown/diff.nvim, of images.nvim. One case per file.
  d         require("harness") and M.run() (spotlight.nvim). One case per
            file.
  h         return function(H) on the project's own TESTS/harness.lua.
            One case per file. Never greener than that harness: collected
            failures, printed failure lines and fields named like a failure
            that grew are failures.
  script    A self-running file (own counters, os.exit): one case, run in a
            child editor of its own; the exit code and the printed [FAIL]
            lines are the verdict.
  busted    describe / it with before_each, after_each, setup, teardown,
            pending and a luassert subset. One case per it. Not supported,
            raising by name: spy, stub, mock, insulate, expose, finally,
            xdescribe, luassert extensions.

A failed check is recorded and the file goes on (H.ask(fn) asks whether a check
fails; inside a pcall the spec wrote it is raised too: docs/DIALECTS.md).
A case without an assertion fails; a file whose dialect cannot be told is an error case with the reason
(never a skip: set dialect in .testing.lua).

12. Reporters and the Result-IR *testing-reporters*

The Result-IR (schema_version 1) is deterministic JSON: run, cases, summary.
Statuses: pass, fail, error, skip, xfail, xpass, timeout, crash. Paths become
<REPO>, <HOME>, <TMP>, <STATE>; with --json the free text is redacted (user and
host name, environment pairs, e-mail addresses, Users/<name> path shapes) and validated
after writing. A privacy finding that redaction cannot remove becomes a top-level warnings
list; it never discards the verdict.
effects are filled by the guard layer; a case says so in its notes when the
guard that fills them is off. Additive fields: cases[].cached (not executed
in this run, from the result cache), cases[].surface (surface.track),
run.cache, run.profile, run.verdict.

The verdict (run.verdict, a line of every reporter) is green (every file ran
green or has a cache hit; the sentinel), green-partial (exit 0, but a
selection, a case filter or a skip means not everything was looked at: no
sentinel, the reasons are named) or red (exit 1; with the last green run and
the files changed since). It counts spec files: n from cache, m ran, k skipped
on purpose. The exit codes do not change.

Reporters read the IR only and treat everything from the code under test as
hostile input: term (default; a line that would start with :: is written
\x3A: and every ##[ is written #\x23[, so a failure message cannot forge
a workflow command), github
(annotations escaped so a test name cannot start a workflow command, step
summary appended), junit (well-formed
XML for any input), agent (compact, for coding agents: the verdict as the
first line, then only the failures with file:line, expected/actual and a repeat
command; equal failures are one entry with a counter; a character budget counts
what it leaves out). A reporter that fails is exit code 3.

13. Migrating from busted or plenary *testing-migrating*

Specs stay unchanged. testing migrate reads the repository and prints the
plan (a .testing.lua, scripts/test.sh, TESTS/minimal_init.lua with the
state of the old scripts/minimal_init.lua carried over (that file is deleted),
the CI workflow as diffs, notes and risks); the created Lua files follow the
repository's stylua.toml (stylua on PATH; otherwise a note). assertions and
timeouts are notes, not written keys. testing migrate apply writes the plan
into a clean git working tree. Or by hand: check out lib.nvim where the runner finds
it, run the runner from the plugin root, read what it says about spec
locations, unknown dialects and unsupported luassert features, then point
scripts/test.sh and CI at the runner and remove the old one. See
docs/MIGRATING.md.