doc/testing.txt — rendered from the plugin's own vimdoc
*testing.txt* Test runner and orchestration for Neovim plugins (pre-alpha) Author: Stefan Bartl License: MIT
CONTENTS
1. Introduction ........................................ |testing-intro| 2. Requirements ........................................ |testing-requirements| 3. Setup ............................................... |testing-setup| 4. Configuration ....................................... |testing-config| 5. Commands ............................................ |testing-commands| 6. Keymaps and autocommands ............................ |testing-bindings| 7. Health check ........................................ |testing-health| 8. Command-line runner ................................. |testing-cli| 9. Project file .testing.lua ........................... |testing-project| 10. Exit codes .......................................... |testing-exit-codes| 11. Dialects ............................................ |testing-dialects| 12. Reporters and the Result-IR ......................... |testing-reporters| 13. Migrating from busted or plenary .................... |testing-migrating|
1. Introduction
testing.nvim is a test runner for Neovim plugins. It runs existing spec files unchanged (thereturn function(H)styles of the author's plugins and busted styledescribe/itspecs, without plenary or busted), reports through a terminal reporter, GitHub annotations, JUnit XML and a JSON result format, and refuses to report a green run it cannot stand behind. This release is pre-alpha. Spec files can run in a child editor of their own, guards name what a spec leaves behind, unchanged spec files can come from a result cache, and there is no editor UI beyond |:Testing|. The long form of this help is in the repository's docs/ directory.
2. Requirements
- Neovim 0.10 or newer
- lib.nvim (hard dependency): https://github.com/WKDSteve/lib.nvim, at
least the commits 6304829 (fs.write.atomic) and 89cb912 (safe_call keeps
non-string errors)
The command-line runner looks for lib.nvim in four places, in this order:
$LIB_NVIM_DIR, <repo>/.deps/lib.nvim, <repo>/../lib.nvim and
stdpath('data')/lazy/lib.nvim. If it finds none, it names all four and exits
with 3. An override ($LIB_NVIM_DIR) that is set but invalid is a failure, never
skipped.
3. Setup
With lazy.nvim:
{
"WKDSteve/testing.nvim",
dependencies = { "WKDSteve/lib.nvim" },
cmd = "Testing",
opts = {},
}
Without a plugin manager spec, call:
require("testing").setup({})
setup()is safe to call twice. The plugin file registers:Testingon its own, socmd = "Testing"lazy-loading works withoutsetup().
4. Configuration
These are the options of setup(). The per-project file of the command-line
runner is a different thing, see |testing-project|. All keys are optional. A key
with the wrong type, and any unknown key, is dropped and reported with a
warning; the default stays in place.
*testing-notify_prefix*
notify_prefix
Type: string, non-empty. Default: "[testing]".
Prefix of every message the plugin shows.
*testing-keymaps*
keymaps
Type: table<string, string|string[]|false>|false. Default:{}. Overrides for the plugin's named keymap actions: action name -> new left-hand side, a list of them, orfalseto drop that action.falseinstead of a table binds no key at all. The one action,hover, has no default key:keymaps = { hover = "<leader>th" }binds it, so nothing is bound by default. *testing-hover*
hover
Type: { max_width?, max_height?, max_diff_lines? } (positive integers).
Default: { max_width = 80, max_height = 20, max_diff_lines = 12 }.
Size of the hover over a test (:Testing hover, hover.nvim :Hover show):
longest line in display columns, most lines, lines of a failure diff.
*testing-surface_view*
surface_view
Type: { keys?: { jump?, status?, kind?, export? } }, each an lhs or false.
Default: { keys = { jump = "<CR>", status = "s", kind = "t", export = "e" } }.
Keys of the matrix view (|testing-matrix-view|), only in that view;
false leaves a key out. Legend, ? cheatsheet and keymaps come from
this one table.
*testing-lint*
lint
Type: { on_save?: boolean }. Default: { on_save = false }.
on_save = true: a written file is linted in the background (the lint
checks that may run on save, never luals) and only its diagnostics are
updated, see |:Testing-lint|.
*testing-tree*
tree
Type: boolean | { keys?: { run?, failed? } }, each an lhs, a list of them
or false. Default: true.
The last result in a tree view that reads lib.nvim.tree.registry
(filetree.nvim's own tree): a badge next to spec files and the view
tests (folder, spec file, describe, case; :Filetree source tests).
In that view r runs what the node stands for and R what failed last
time; keys moves them, false leaves an action in the menu only.
tree = false registers nothing. See docs/STATE.md, "In a tree view".
5. Commands
*:Testing* :Testing [run] [{root}] [--file=..] [--filter=..] [--reporter=..] [--cached] [--no-cache] [--changed] [--since={rev}] [--shard={i}/{n}] Run the specs of the project in a headless child Neovim, never in this editor. {root} defaults to the nearest directory upwards that holds.testing.lua,TESTSor.git.--cachedskips spec files whose inputs did not change since a green run (reported as cached);--changedand--sincerun only the specs the changes can reach (a partial run, never the "all green" last line). :Testing file [{spec}] Run the spec of the current buffer. :Testing last Repeat the last run. :Testing hover Last result of the test under the cursor (float); also hover.nvim:Hover showover a test file, a describe block or a spec path in any buffer. With thehoverkeymap action (no default key) on a key of your own. A buffer with unsaved changes is worded as outdated; only_spec.luafiles and files with a recorded result answer. hover.nvim is never required bysetup(): a retry in the augroup TestingHoverRegister waits for it. :Testing terminal Repeat the last run in a terminal.nvim terminal. :Testing list [{root}] List what would run. :Testing init [{root}] [--force] [--plugin={name}] [--workflow={kind}] Generate the test setup of a plugin repository. An existing file is never overwritten without--force.--workflow=two-runner,two-runner-privateorhostedwrites only.github/workflows/ci.ymlin that layout of the generator (the local runner plus a GitHub-hosted macOS leg, or hosted runners only).--workflow=publicgenerates that file fromci.workflowof.testing.lua. :Testing conformance [{root}] [--gate] [--only={ids}] [--skip={ids}] [--bridge] Run the conformance checks K1..K16 on the project (a report; a failed gate is a warning, not an error). :Testing surface [{root}] [--from={ir.json}] [--threshold={n}] [--markdown] [--matrix] [--csv] [--no-tags] The keymaps, commands and autocmds of the plugin and how much of them the specs exercised.--baseline={file}compares with a baseline; with--require-signed-baselinea baseline that was edited or has no digest fails the run (exit 1).--matrixshows entries x spec files,--csva tab separated table;binding:/command:tags and-- @coversheaders of passing cases count as exercised (not with--no-tags).--matrixalone opens the matrix view (see |testing-matrix-view|). *testing-matrix-view* The matrix view of:Testing surface --from={ir.json} --matrixis a ui.kit viewer, one line per entry, one column per spec file. Its keys (buffer-local, from |testing-surface_view|): <CR> on a spec file column: that spec file; elsewhere on a row: the source of the entry at its line s status filter: all, hit, missing, untracked, ignored t kind filter: all, binding, command, autocmd e export the shown rows as a tab separated file (asks for a path) ? the cheatsheet; q and <Esc> close :Testing budget [{root}] [--update] [--factor={x}] [--allow-new] Measure the performance budgets of the runner. A measured case without a baseline entry compared nothing: exit 2 unless--allow-new. *:Testing-act* :Testing act [--root={dir}] A finding of the last run to a task, a rule or the AI (testing act). The target is the current quickfix entry, else the line under the cursor, else a choice. Then two steps: what (task, rule, AI) and how (form, AI draft, agent; skipped when the target takes one). A task and a rule are a dry run of the target's engine first, then a question; a duplicate names the task that is already there. The AI gets a hand-over file in testing's state folder and the command for Claude Code in register+; nothing is sent or started. *:Testing-export* :Testing export [{dir}] [--repos-root={dir}] [--remote=always|never|ask] Write the export folder testing-web reads, in a headless child (testing export). Runs no suite and no pipeline. {dir} defaults to the settingci.export_dir, the repositories toci.repos.roots,--remotetoci.export_remote. Choose that folder in the page. *:Testing-lint* :Testing lint [{check}...] [--file[={a,b}]|--changed|--all] [--fix] [--root={dir}] Run the lint checks (stylua, luacheck, luals: the lint steps of thecipipeline, or the named ones) in a headless child (testing lint --json). Default: the files changed against HEAD and the untracked ones; a bare--filethe file of the current buffer,--file=a.lua,b.lua(repeatable) the named files,--alleverything the step checks intesting ci. The findings fill the quickfix list "testing lint" (opened only when there are some) and become|vim.diagnostic|in the loaded buffers, one namespace per check (testing.lint.{check}); a checked file without findings is cleared.--fixrewrites what stylua can fix and reloads the buffers (refused while a buffer in its scope has unsaved changes). A missing tool skips its check and is named in the message (exit 3, never green). On the command line:testing lint --help. :Testing cache stats | clear [{root}] Size of the result cache of the project, or delete it. :Testing health Run:checkhealth testing. :Testing config Show the effective plugin configuration. :Testing doctor [{root}] Show the project configuration and the dependency report. Subcommands and flags are completed with <Tab>. The complete reference is docs/BINDINGS.md.
6. Keymaps and autocommands
The plugin binds no key and registers no autocommand by default. Its one keymap action,hover(:Testing hover), has no default key;keymaps = { hover = "<lhs>" }binds it. Every action can be moved or dropped through thekeymapsoption.
7. Health check
:checkhealth testingreports, in separate sections: the Neovim version; the lib.nvim modules the plugin calls, plus two behaviors it relies on (safe_call keeps non-string errors, fs.write.atomic refuses an invalid call), each probed without writing anything; whether the kernel modules load; the dialect and reporter registries; whether the command-line runner can find lib.nvim; the.testing.luaof the current directory; the guards and the warm pool as that file configures them (modes, allowances, pool size; a warning when every guard is off, an error when a module of the isolation machinery does not load); the effective configuration; whether the result cache directory can be written, whether the docmap.nvim module graph of the affected selection is available and current, and whether the conformance suite and the surface report load; and whether |:Testing| is registered. It is read-only and loads no plugin that is lazy-loaded; the processes it starts are the version calls of the lint tools and one question to the container engine (whether the image oftesting.containeris built, 5 s at most)..testing.luais a Lua file and loading it executes it, so the health check does not: it evaluates the file in an empty environment with an instruction budget. A plain table is validated for real; a file that needs more is reported as "not evaluated here" (never as valid).testing doctorevaluates it. A level never contradicts its text: an error is something broken, a warning is something that will fail elsewhere (for example the command-line runner not finding lib.nvim while the editor works), an info is something not checked.
8. Command-line runner
nvim -n -i NONE --headless -u NONE -l scripts/testing.lua [run|fast|list|doctor|budget|conformance|surface|explain|stamp|verify] [{root}] [options]
nvim -n -i NONE --headless -u NONE -l scripts/testing.lua migrate [dry-run|apply] [{path}] [options]
Run it from {root}: specs that look at "this repository" read the working
directory and the runner does not change it. scripts/test.sh runs this
repository's own specs this way.
*testing-command*
testing {args} is the same line as a command: put the checkout's bin/
folder on the PATH (bin/testing, bin/testing.cmd on Windows). It hands
the exit code through; without nvim on the PATH it says so and exits 3.
Subcommands: run (default), list (= run --list), doctor (resolved
configuration and dependency report), migrate (plan or write the move of a
repository to testing.nvim, see |testing-migrating|), budget (measure the hot
paths of a run against a baseline), conformance (the checks K1..K16) and
surface (the plugin's keymaps, commands, ... and what the specs exercised);
those two have their own arguments and exit codes (docs/CONFORMANCE.md,
docs/SURFACE.md). explain says why a spec file is selected, taken from the
cache, run or left out, and what its cache key is made of (see "Explain and
audit" below). stamp runs the suite and, after a complete green run, writes
the green stamp; verify answers from the cache keys, without running a spec,
whether the tree is still the stamped one (see "Stamp and verify" below).
init is available as |:Testing| init only. ci runs the pipeline of
.testing.lua locally; ci remote runs nothing and reports the latest GitHub
Actions run of the CI workflow per repository and system (docs/CI.md, "The
GitHub Actions status"; --history[=N] reads the last N runs and tells when
one system was red alone; --wait[=<duration>] asks again every 30 seconds
until no run of the head commits is going, at most 60 minutes or the duration,
past that exit 1 "not finished"); ci overview runs nothing either and writes one
static HTML page of the fleet from ci verify, ci.json and that adapter
(docs/CI.md, "Fleet overview").
ci commit {rev} checks a commit in a git worktree of its own (another
session's uncommitted work does not count) and sorts red into new, known and
fixed against the parent: clean and known-red exit 0, regression 1
(docs/CI.md, "Check a commit"). ci commit --repos-root={dir} --since={time} does it for the new commits of every
repository below a directory. From the editor: :Testing ci commit (HEAD or the
revisions given, --since, --repos-root; the quickfix list holds only the new
findings). ci commit {rev} --quick runs only the spec files the diff to the
base reaches (CLEAN quick); it is no proof of the tree: before the push one
full ci commit HEAD --base=@{upstream} -- --cached. On Windows --wsl runs
the check in WSL2 (about ten times faster; docs/CI.md, "Check a commit").
ci land [{rev}] pushes a commit to a check branch ci/<repo>-<sha>, waits
for the hosted CI and moves main to it only when that is green (needs
ci.workflow.land = true; docs/CI.md, "Land through the hosted CI").
Heavy runs (a whole suite, ci with the whole suite, ci commit) take turns
on one machine: a second one waits and says for what (docs/CI.md, "One heavy
run at a time"). TESTING_QUEUE=off turns that off.
stats [--since=7d] [--repo=<name>] [--json] sums up what the heavy runs of
this machine cost (one line each in metrics.jsonl; docs/CI.md, "Metrics").
Workspace trust ~ *testing-trust*
The runner executes a project's .testing.lua, minimal init and specs, so it
runs only a trusted root; any other is exit 2 before anything of it runs. A
root is trusted when CI is set (not false/0), TESTING_TRUST=1, the
root lies below a directory of TESTING_TRUST_ROOTS (a list like PATH), or
the store (stdpath("state")/testing/trust.json) has it with the hash of its
current .testing.lua (a changed file asks again). --trust on any command
stores the root and goes on; testing trust list|check|add|revoke [{root}]
manages the store. From the editor, |:Testing| and :Testing ci ask once
(vim.ui.select) instead: yes stores the root and runs again, no leaves the
refusal; a headless nvim is never asked, :Testing ci repos never asks.
Details: docs/CONFIG.md, "Workspace trust".
Selection
--file {text} Only spec files whose file name contains {text}
(repeatable). --only is an alias.
--filter {text} Only cases whose name contains {text} (literal).
--tags {a,b} Only cases with one of these tags: a #word in a busted
--exclude-tags title, or a -- @tags a b header line in any spec.
--lf, --ff Only what failed last time, or that first.
--order priority Run first what failed last time, what the changes touch
and reach (nearest first), what has not run for long.
Orders only, never filters; excludes --shuffle.
--time-budget {t} Stop starting spec files after {t} (90s, 2m, 1h). A running
file is never cut short. The rest is "not run (budget)":
verdict green-partial (exit 0, never the sentinel), or red.
--skip-uncacheable With the cache: leave out files that have no cache key, on
purpose: green-partial, never the sentinel. Refused (exit
2) when CI is set.
--order slowest-first
With --jobs 2 or more and child editors: the longest
files (remembered duration) start first. Results stay in
file order.
--retry-failed {n} Run a red case again up to {n} times (1 to 10). A case
that passes on a retry is flaky: the run stays red, it is
listed and never cached.
--allow-flaky With --retry-failed: count such a case as green (still
listed). The run is then green-partial: no sentinel.
--list, --dry-run List what would run, run nothing.
--shard {i}/{n} Only shard {i} of {n} (CI matrices).
--changed, --since {rev}, --affected[={rev}]
Only the specs the changes can reach (git, working tree
against HEAD, against {rev}, or the last commit). A
partial run: no "all green" last line. Anything that
cannot be placed selects every spec. Never the default.
--consumers {dir}
With one of the three above: name the specs of the
checkouts below {dir} (projects that use this one) that
the change reaches (docmap.nvim). A hint, it never
changes what runs here. Also affected.consumers in
.testing.lua.
--state-dir {dir} State directory of the run history (default stdpath('state')).
scripts/test.sh hands the real one down; the specs keep
a throwaway one.
--cache-dir {dir} Base directory of the result cache (default stdpath('cache');
env TESTING_CACHE_HOME; the flag wins). `cache = {
project_key = ... }` in .testing.lua names the project
folder after a key instead of the checkout path
(docs/CI-CACHE.md).
--cached, --no-cache, --cache-refresh, --cache-clear
The result cache: a spec file whose inputs are
byte-identical to an earlier green run does not run and is
reported as cached. Off unless asked for; --no-cache
always wins. docs/CACHE.md.
--cache-audit {0..1|all}
Run that share of the cache hits anyway and compare with
the stored result. A difference is the finding
cache.stale_pass and exit 1; the measured stale-pass
rate is on the cache line and in run.cache of the IR.
Implies --cached; excludes --cache-refresh.
--config {file} Project file inside the root.
--rtp {dir} Add {dir} to the runtimepath (repeatable).
Control
-x, --maxfail {n} Stop after the first / {n}th failure.
--shuffle, --seed {n}
Random order; the seed is printed.
--case-timeout {ms}, --file-timeout {ms}
Timeouts. In this editor best effort (Lua code and
vim.wait, not a spec that blocks inside C); in a child
editor hard: the child and its process tree are killed.
--strict A skipped case or a discovery finding makes the run red.
--isolated {none|file|case|soft}
file: every spec file in a child editor of its own (a
crash, a hang or a prompt ends that file only). case:
a child per case (busted). soft: this editor, what a
file changed is restored before the next. Default
auto: busted files file, the others none; a
script is always a child.
--jobs {n|auto} Child editors at once; the report is the same for any n.
--watch Run, then re-run on every change.
--watch-max-wait {ms} Longest a change waits before a re-run (cooldown).
--profile Where the time went (stderr; run.profile in the IR).
--pool-reuse, --no-pool-reuse
The warm pool: with --isolated file the files share
child editors that are reset and verified clean between
files (a member that cannot prove it is clean is
replaced and a finding says why). script files and
case never use it.
--pool-size {n} Editors the pool keeps (default min(--jobs, 4)).
--guard {name}={mode}
Guards: fs, state, scheduled_error, prompt, deprecation,
process_net (off|warn|error), clock (on|off).
--allow-fs {path}, --allow-spawn {exe}, --allow-network {host}
What the guards let through (repeatable).
--no-determinism A child keeps the parent's LANG/LC_ALL/TZ.
--no-trace No trace artifact for a child that timed out or died.
--isolate-home A child gets a home directory of its own (HOME,
USERPROFILE, APPDATA, LOCALAPPDATA in its sandbox).
--update-snapshots Rewrite screen snapshots that differ (testing.screen);
implies --no-cache.
--host {c|l} A child starts like plenary's host (c, default) or like
nvim -l.
--env-allow {name} An environment variable (or PREFIX*) a child may inherit.
Output
--reporter {name} Terminal reporter: term, github, junit, json or agent.
Default: $TESTING_REPORTER, then agent in a coding
agent environment ($TESTING_AGENT=1 forces it, =0 stops
the detection), else term.
--agent-budget {n} Characters of the failure part of agent (default 4000).
--format {text|jsonl} Shape of agent (jsonl: one object per line).
--json {file} Write the Result-IR (schema_version 1), validated.
--junit {file} Write a JUnit XML report.
--events {file} Write the events of the run (NDJSON) while it goes; - is stdout.
--github GitHub Actions annotations and step summary.
--durations {n} Name the {n} slowest cases.
--sentinel {name} Last line of a complete green run (default TESTING_OK).
--no-timings No timing line.
Explain and audit
explain {spec}... Say why a spec file is selected, taken from the cache, run
or left out: the reason of the selection, hit, miss
(which file, with old and new hash, which environment
variable by name, the Neovim version or the configuration
changed since the stored entry) or uncacheable (the
file and line of what makes it so, and the way out:
-- @cache-allow, -- @cache-inputs, -- @cache-env,
env_allow,
-- @cache off). {spec} is a path, a directory or a part
of a file name; the options of a run are accepted (the
key depends on them). It changes nothing.
--all Every spec file: the hit rate and the reasons of the
files without a key, most frequent first.
--json One JSON document (testing-explain/1).
--parts The lines the key is the hash of.
Stamp and verify
stamp A run that writes the green stamp after the verdict
green only (never for a partial or red run: exit 1;
options that select part of the suite are refused, exit
2). The stamp holds, per spec file, the cache key or the
reason there is none, plus runner, Neovim, OS and
configuration digests, commit and tree, all taken before
the first spec runs (an input that changed while the
specs ran: no stamp, exit 1). The report says so in its
first line (RED ... exit 1, verdict: red), never
green. No secrets.
--out {file} Where to write it (default: stamp.json in the state
directory of the project, not in the checkout).
--note Also attach it to the tree object as a git note under
refs/notes/testing.
verify Recompute the keys (after the minit of the project, as a
run does) and answer: verified (exit 0 and the
sentinel, only when EVERY file is proven), partial
(files without a key are not proven; never green),
changed, rejected (other runner, Neovim, OS or
configuration), expired, dirty, untrusted,
invalid, no-stamp (all exit 1).
--stamp {file} Read this stamp instead of the default one.
--from-note Read the git note of the tree of HEAD.
--max-age {dur} Oldest accepted stamp (7d; s, m, h, d).
--allow-dirty Check a dirty working tree anyway (said in the answer).
--require-hmac Refuse a stamp unless TESTING_STAMP_SECRET is set.
--allow-unsigned In CI, accept a stamp without a checked HMAC (otherwise never green).
--json One JSON document (testing-verify/1).
TESTING_STAMP_SECRET (16 characters or more) adds an
HMAC when writing and checks it when verifying.
docs/CACHE.md, "Stamp".
--cache-audit {0..1|all}
see Selection: a share of the hits runs anyway; a
difference is cache.stale_pass. A spec whose key gave
two different results is nondeterministic and not
cached (-- @cache-allow nondeterministic lifts it).
docs/CACHE.md.
An option that is parsed but not implemented is refused with exit code 2, never
ignored. Only a complete green run (no skipped case, no --maxfail stop, no
case accepted by --allow-flaky) prints the sentinel; anything else prints a
distinct last line. The same
distinction is the verdict line of every reporter: green, green-partial, red.
9. Project file .testing.lua
A Lua file in the project root that returns a table; every key is optional. It is executed, so it is loaded only from the root named on the command line (or--config, inside that root), as text, below 256 KiB. Invalid values and unknown keys are warnings (the default stays); an unusable file is exit 2. plugin string Module root of the project. roots string[] Spec roots, relative. Default { "TESTS" }. spec_pattern string[] Lua patterns that make a file a spec. Default { "_spec%.lua$" }. A malformed pattern is a warning (the default stays). dialect string|table "auto" (default), "testing", "a", "b", "c", "d", "h", "busted", "script"; or { [path or glob] = name, ["*"] = name }. assertions string "error" (default): a case without assertions fails; "warn": it passes and is listed. isolated string "auto" (default), "none", "file", "case", "soft". guards table fs, state, deprecation, scheduled_error, prompt, process_net: "off"|"warn"|"error" or a table { mode = ..., categories, ignore_*, keep, allow, ... } (docs/GUARDS.md); clock: boolean. Runner default: process_net "warn". guard_allow table { fs, spawn, network } string lists. pool table { size = n, reuse = boolean }: the warm pool. determinism boolean Fixed LANG/LC_ALL/TZ in a child. Default true. isolate_home boolean A home directory in the child's sandbox. Default false. trace boolean Trace artifact of a child that died. Default true. soft_keep string[] Modules the soft isolation never unloads. jobs integer|"auto" Child editors at once. Default 1. shard, watch, budget tables--shardbalance,--watchdebounce, cooldown and poll,testing budgetfactor and baseline. cache table { enabled = boolean }: reuse results without --cached (never in CI). conformance table gate, skip, waivers (each with a reason), keymaps_off, timeout_ms, load_budget_ms, rules_bridge:testing conformance. surface table track (the runner records what the cases exercise), threshold, kinds, ignore, setup_chunk. host string "c" (default) or "l". filetype boolean Children runfiletype plugin indent on. disable_first_run boolean Set lib.nvim's first-run opt-out (vim.g.lib_nvim_deps_disable_first_run) in every editor the runner starts, beforeminit. Default true. env_allow string[] Environment names a child may inherit on top of the allowlist (never NVIM*, never a bare "*"). minit false|string Minimal init, run before the specs. deps string[] Dependencies that must resolve (exit 3 if not). timeouts table case_ms (10000), file_ms (60000); hard in a child. setup table Optionssetup()is called with by conformance and surface. coverage table bindings, commands, autocmds: 0..1 thresholds oftesting surface. snapshots tablediris the default of testing.screen (TESTS/__snapshots__); the runner hands it to the specs;opts.dirwins. backends Validated, reserved: nothing acts on it yet. A dependency is looked up in $<NAME>_DIR, <root>/.deps/<name>, <root>/../<name> and stdpath('data')/lazy/<name>; the message of a miss names all four.
10. Exit codes
0 Every spec file passed.
1 A case failed, errored, timed out or crashed (a child editor that died);
under --strict also a skip or a discovery finding.
2 Usage or configuration error, or nothing to run (also a selection that
matches nothing).
3 Infrastructure error: a dependency is missing, minit raised, a report
could not be written, the editor was quit under the run, an internal
error.
A spec cannot end the run: os.exit inside a spec is an error of that file
(in a child editor it is a crash of that file; the others run on).
testing migrate: 0 done or nothing to do, 1 (--check) work to do, 2 refused
or bad usage, 3 the root cannot be analysed.
testing sheets report [<root>...] [--from <ir.json>]... [--out <file>]: the
checklist of live checks from the test sheets (TESTS/sheets/*.sheet.lua) and
their last results (the editor's state, or the IR of a run with --from); runs
nothing. --tasks {vault} writes the state of each sheet into its task through
tasks.nvim (--dry-run: only say what would change); a project is named
there by sheets.project of its .testing.lua, else by its folder. With a
feature catalog (TESTS/sheets/features.lua) it ends with the section
Features: the state of each functionality (the worst of its sheets), the ones
without a sheet, the sheets without a feature, and unknown features. 0 written or printed, 2 bad
usage, 3 the file could not be written. See docs/SHEETS.md.
testing sheets new --from-task {area/slug} [--repo <dir>] [--vault <dir>]:
writes a draft sheet TESTS/sheets/<slug>.sheet.lua from a live-check task
(read through tasks.nvim): a skeleton of steps and one judge entry per
acceptance checkbox. --feature {id} (with or without --from-task) sets the
sheet's feature and its title from the catalog; an id the catalog does not
know is exit 2. Never overwrites a sheet (--name for another file and
id). 0 written, 2 bad usage or the sheet exists, 3 no vault or engine, the task
could not be read, the file not written or the draft does not load.
testing sheets run --task {area/slug} [--root <dir>] [--tasks <vault>]:
runs the sheets that name the task now and writes their state into its block
(trusted roots only; a missing snapshot fails; on the workstation only
manual sheets run, unless TESTING_ALLOW_WORKSTATION=1). 0 green, 1 red or not written, 2 bad
usage, refused or no sheet, 3 the engine of tasks.nvim could not be reached.
--result (any form of sheets run) prints one more line, the last one on
stdout: a document of testing-sheet-run/1 (schemas/testing-sheet-run/1/)
with the status of each sheet, red, the outcome of each task and written,
so a program that started the run tells a red sheet from a task that was not
written (both are exit 1).
testing sheets run --all [--root <dir>] [--json <ir>] [--strict] runs every
sheet of the roots; --json writes the Result-IR, --strict makes a skip red.
testing sheets run --sheet {id}... runs only the named sheets (id or file
name), as --all would. --live (any form) lets live sheets (`live = { env,
hosts, cost }`) run with their real keys, only the named hosts and the cost
said first; never on CI, a skip without it. See docs/SHEETS.md, "Live sheets".
`testing sheets step {sheet} [--root <dir>] [--headed] [--delay <ms>]
[--break <n>]... [--run]`: one sheet step by step. Events on stdout
(testing-sheet-session/1, one JSON line each), commands on stdin (next,
continue, until <n>, note <text>, explain <n> <kind>, abort; closed
stdin is abort). explain answers with the request an AI would get about
a step, as an event for the client to show, and sends nothing. Writes
nothing but the notes of the run (beside its pictures, for the report).
--run --format=text writes one line per step instead (size, n/total,
status, ms, action and value, # note, : reason, => a measured value)
and the end. A measure = expr step (name, min, max, unit, runs)
reads a number in the child, takes the median of its runs, fails outside the
bounds and keeps the value as a note of the run (docs/SHEETS.md).
0 green or a skip, 1 a step failed, 2 bad usage or refused, 3 aborted.
See docs/SHEETS.md, "Step by step".
:Testing sheet step [{sheet}] is the same in a buffer: n next, c continue,
u until the cursor, b breakpoint, N note on the step, a abort, q close, y or n to
a question. E explains the step under the cursor with AI through ai.nvim (what
it does, what to watch out for, what its result means): the request is shown
before it is sent, and only sent on a yes; without ai.nvim nothing happens.
testing sheets prompt draft --from-task {id} / prompt explain --task {id}:
a Markdown hand-over for an AI (draft a sheet from the task; explain a red
run). testing sheets accept --task {id} --from <file> writes the AI's sheet
only when it loads and names the task. A vault or root naming tricentis is
refused. See docs/SHEETS.md, "Hand over to an AI".
11. Dialects
The dialect of a file is sniffed from its text (dialect = "auto") or forced. a, b, creturn function(H)with the helpers of lib.nvim, of markdown/diff.nvim, of images.nvim. One case per file. drequire("harness")andM.run()(spotlight.nvim). One case per file. hreturn function(H)on the project's ownTESTS/harness.lua. One case per file. Never greener than that harness: collected failures, printed failure lines and fields named like a failure that grew are failures. script A self-running file (own counters,os.exit): one case, run in a child editor of its own; the exit code and the printed[FAIL]lines are the verdict. busteddescribe/itwith before_each, after_each, setup, teardown, pending and a luassert subset. One case perit. Not supported, raising by name: spy, stub, mock, insulate, expose, finally, xdescribe, luassert extensions. A failed check is recorded and the file goes on (H.ask(fn) asks whether a check fails; inside a pcall the spec wrote it is raised too: docs/DIALECTS.md). A case without an assertion fails; a file whose dialect cannot be told is an error case with the reason (never a skip: setdialectin .testing.lua).
12. Reporters and the Result-IR
The Result-IR (schema_version1) is deterministic JSON: run, cases, summary. Statuses: pass, fail, error, skip, xfail, xpass, timeout, crash. Paths become <REPO>, <HOME>, <TMP>, <STATE>; with --json the free text is redacted (user and host name, environment pairs, e-mail addresses,Users/<name>path shapes) and validated after writing. A privacy finding that redaction cannot remove becomes a top-levelwarningslist; it never discards the verdict.effectsare filled by the guard layer; a case says so in its notes when the guard that fills them is off. Additive fields:cases[].cached(not executed in this run, from the result cache),cases[].surface(surface.track),run.cache,run.profile,run.verdict. The verdict (run.verdict, a line of every reporter) isgreen(every file ran green or has a cache hit; the sentinel),green-partial(exit 0, but a selection, a case filter or a skip means not everything was looked at: no sentinel, the reasons are named) orred(exit 1; with the last green run and the files changed since). It counts spec files: n from cache, m ran, k skipped on purpose. The exit codes do not change. Reporters read the IR only and treat everything from the code under test as hostile input:term(default; a line that would start with::is written\x3A:and every##[is written#\x23[, so a failure message cannot forge a workflow command),github(annotations escaped so a test name cannot start a workflow command, step summary appended),junit(well-formed XML for any input),agent(compact, for coding agents: the verdict as the first line, then only the failures with file:line, expected/actual and a repeat command; equal failures are one entry with a counter; a character budget counts what it leaves out). A reporter that fails is exit code 3.
13. Migrating from busted or plenary
Specs stay unchanged.testing migratereads the repository and prints the plan (a.testing.lua,scripts/test.sh,TESTS/minimal_init.luawith the state of the oldscripts/minimal_init.luacarried over (that file is deleted), the CI workflow as diffs, notes and risks); the created Lua files follow the repository'sstylua.toml(stylua on PATH; otherwise a note).assertionsandtimeoutsare notes, not written keys.testing migrate applywrites the plan into a clean git working tree. Or by hand: check out lib.nvim where the runner finds it, run the runner from the plugin root, read what it says about spec locations, unknown dialects and unsupported luassert features, then point scripts/test.sh and CI at the runner and remove the old one. See docs/MIGRATING.md.