Testing and evidence

Hostwright separates deterministic test coverage from evidence that exercises real local systems. Every support claim in these docs traces back to one of these classes — a report uses exactly one:

Evidence class	What it proves	What it cannot prove
unit-contract	Deterministic logic, parsing, policy, redaction, failure injection, clocks, retries, recovery branches.	Real process, database, Keychain, Apple container, hardware, or artifact behavior.
local-integration	Real local files, subprocesses, loopback networking, SQLite connections, file locks, Keychain operations.	Apple container mutation, hardware efficiency, signing, notarization, or install artifacts unless exercised.
live-runtime	Real runtime behavior against uniquely named disposable Hostwright-owned resources with exact cleanup.	Unmeasured hardware efficiency, provider behavior, distribution trust.
hardware-benchmark	Measured work on the recorded physical hardware with raw samples and exact tool versions.	Capacity guarantees, other hardware, unmeasured accelerator behavior.
distribution-artifact	Actual build, checksum, SBOM/provenance, signing, notarization, install/upgrade/uninstall stages that ran.	Any stage recorded as blocked or failed.
migration-upgrade	Real forward migration, upgrade, rollback window, backup, restore, and mixed-version behavior.	A schema unit test or invented prior artifact alone.
security-assessment	Threat/adversarial tests, fuzzing, scans, penetration work, and remediation for the reviewed boundary.	A blanket claim for an unreviewed deployment.
resilience-chaos	Recovery under injected process/storage/network/timing/checkpoint faults with exact cleanup.	Faults or soak time that were not exercised.
multi-host	Physical Mac cluster quorum, fencing, failover, partitions, identity, upgrade, and cleanup behavior.	Multiple local processes or simulation alone.
interop-conformance	Named upstream suite and exact protocol/client/version results.	Untested endpoints, versions, clients, or silently dropped fields.
ux-accessibility	Completed workflows, API parity, accessibility and assistive-technology results.	Visual review alone.

Reports conform to schemas/hostwright-evidence.schema.json in the core repository. Status is passed, failed, or blocked — there is no skipped-success status.

The default repository gate runs swift test (unit-contract plus local integration), then scripts/integration.sh against the built CLI. Live runtime, hardware, and distribution lanes are separate because they require explicit resources or credentials.

Passing rules

A passing report must:

• identify the exact source commit and whether the worktree was dirty;
• record OS, architecture, hardware, memory, and tool versions;
• include every command and its real exit status;
• include raw executed/passed/failed/blocked counts;
• contain no failures or blockers;
• record cleanup as not-required or succeeded;
• use evidence from the declared class — never a fixture standing in for a higher class.

Release evidence must come from a clean checkout. A dirty report can support development diagnosis but cannot satisfy a release gate.

Failure and blocking rules

• A command failure makes the report failed with a redacted message.
• A missing executable, image, credential, signing identity, second host, or other prerequisite makes the report blocked.
• Blocked work may not be converted to passed with a fixture, no-op implementation, conditional early return, or silently skipped test.
• Exact cleanup failure makes live-runtime, hardware, resilience, multi-host, or interoperability evidence fail even when the measured operation succeeded.

Why this matters for these docs

The v0.0.2 Phase 01 documentation rules require every current-support claim on this site to trace to reference docs or tests in the core repository. When a page here says “implemented”, that means covered by passing evidence in one of these classes for the reviewed commit — not “should work”.