Claude Code First, specification version 0.1.0. The plain-language guide
Version 0.1.0. Grade R3/S3.
F302 A gate reported a pass over work that did not run or a check that looked at the wrong thing
| Field | Value |
|---|---|
| Status | active |
| Topic | Gates, verification, and hooks against instructions |
| Grade | R3/S3 |
| Cases | winwright, Shio, freewilly |
| Contrary | none |
| Harness facts | none |
Claim
In three projects a gate went green, or exited 0, when the tests it stood for had not run, when its exit status was not the suite's, or when the thing it inspected was not the thing under test; each project answered by making "did not run" a verdict distinct from a pass or by asserting on the signal it actually needed.
Mechanism
A green is the absence of a reported failure. A test host that dies, a pipeline whose status is its last command's, a stale build directory, a check that matches nothing, or a picture taken before the window arrived all produce no failure, and so read as success. The fixes count what ran against what should have run, keep the suite's own exit code, or wait for the positive signal before judging.
Observations
- winwright. A dying test host still printed a pass, "measured here at 352 of 374, a green covering twenty-two tests that" [winwright@861b82e:.github/workflows/ci.yml#L42]
never ran. WW117 added a roll call of "what discovery listed against what the results file recorded" [winwright@861b82e:docs/CHANGELOG.md#L14] [winwright@6351b80], and WW138 made a bare dotnet test take it [winwright@2eeb059]. The shipped skill makes the same distinction for the product's own checks: "A check that could not run is a third verdict and never a pass" [winwright@861b82e:skills/winwright/SKILL.md#L71].
- Shio. Four separate forms. A stale build: "mvn compile can report success on code that does not compile" [shio@052550e68].
A piped gate: the shell reported the filter's status, so a failing pnpm test printed an exit status of 0 [shio@821f18d74:agents.md#L155-L156], fixed in [shio@c1b9e8947]; an earlier pipe had lost the failure itself, one run in six failed and "identical runs around it passed and the failing class is still unknown" [shio@821f18d74:shio-app/src/test/java/com/viglet/shio/ShTestCommandDocLintTest.java#L31], because the failure line "went into a pipe nobody kept" [shio@821f18d74:shio-app/src/test/java/com/viglet/shio/ShTestCommandDocLintTest.java#L33]. A vacuous check: the assertion-debt test's own first run misread itself, "on the first run its own declaration made all forty entries read as" [shio@821f18d74:agents.md#L234] "asserted and the debt came back as three." [shio@821f18d74:agents.md#L235] The runner that replaced four wrappers keeps the output and states that "the exit code is not swallowed" [shio@821f18d74:scripts/run-gate.mjs#L9].
- freewilly. A desktop capture case "photographed the guest's WALLPAPER, and passed" [freewilly@c1c2eaf];
the same commit records that an earlier copy of its kind "twice photographed something else" [freewilly@c1c2eaf]. The case now waits for the menu to report an entry before it copies anything. The token budget gate had a band so wide that a "response that grew by 100 tokens landed inside it silently" [freewilly@c1c2eaf:agent-budget.json#L109], and now asserts every figure exactly [freewilly@c1c2eaf:agent-budget.json#L103].
Independence
Three independent origins: winwright's test host, Shio's build and shell pipeline, and freewilly's capture and budget gates. Each is a failure of that project's own gate, on a different stack (.NET test runner, Maven and pnpm, a WPF capture), and none of the remedies was copied between them.
Counter-evidence
None found at the pins. roadkeep's round-trip test over Shio's and Turing's files skips when they are absent, "which is what CI does" [roadkeep@91754240:.claude/skills/roadkeep-dev/SKILL.md#L26]; that is a green over an unrun check by design, stated, with no incident recorded, and it is neither support nor contrary evidence here.
Grade
R3: independent failures in three projects. S3: winwright's is counted (352 of 374, twenty-two unrun), Shio's in its sources (one run in six; forty entries read as asserted), and freewilly's budget failure in tokens (100). Not S4: winwright's roll call and Shio's command lint encode the remedy, but no source at the pins records either one failing on a real run.