Reading the verdict

If you are here because a run answered 2 and you are not sure what you did wrong: probably nothing. Read on.

The four readings

The enum’s member values are the process exit codes. CI reads the number rather than the word, and a mapping written twice is a mapping that drifts.

Exit code	Verdict	What it means
0	Passed	Every assertion ran and every one of them held.
1	Failed	At least one assertion ran and did not hold.
2	Degraded	Everything that ran passed, and something could not be evaluated at all.
3	Broken	The harness broke — something threw, and what it says is about this tool rather than about the application under test.

They do not rank in that order

A run is folded to one reading, and the order is not the enum’s:

Broken outranks Failed outranks Degraded outranks Passed.

Taking the largest member value would rank a hole above a failure, which is the one comparison the numbers happen to get backwards. 3 outranks everything because a reader told the build failed opens the wrong repository, and nothing after a throw was observed at all.

A run that produced no readings at all is {holes.empty} rather than a pass. A run of no cases has no failure in it, so it would otherwise read as a green about nothing — and before it gets that far, a suite with nothing to run and a selector matching no case are both refused, naming what there was to choose from.

A hole is an assertion that never ran

Not one that failed. The precondition it needed was absent, so the check was never made, and the summary names it — by name, in the line, before the tally.

Passed: 1 of 9 cases, 8 not run, 3 assertions over case 'renaming a profile writes it back'.

Collapsing that into a pass is the defect this project was started over: a suite reported a pass with no failures over a total of 352 where the run before it had 374, twenty-two tests gone, because the host had died partway through. Collapsing it into a failure is the other half of the same mistake — it sends you to read code that is fine.

So the run says whose the hole was, which is what decides what you do next.

Desk: The desk's. The machine could have been arranged differently: who holds the foreground, what is standing over a rectangle, whether the shell answers. Nothing to fix in any repository.
UnderTest: The thing under test's. A stale binary, a page still computing, an application in the wrong language. A repository to open, and re-running changes nothing.
Unclassified: Neither, because nobody has said. The condition was composed at the throw site, or it is one this engine declares and DeskFacts has never classified — or the hole carries no condition at all, which is worse than a failure and is counted here rather than rounded into one of the other two.

This engine declares 24 conditions an assertion can be held up by. 9 of them are the desk’s and 15 are about the thing under test, and both lists below are read off the engine on every build.

The desk’s — nothing to fix in any repository

These are the ones an adopter meets first, and none of them is your code being wrong. A run that could not get what it needed of the machine says so instead of guessing.

The reading that was absent	Why it is the desk's and not yours
the foreground belongs to the window under test	Windows grants the foreground to whoever it grants it to, and once this process has been refused once it stops being granted
the control under test holds the keyboard focus	a key goes where the focus is, and the focus is the desk's to give
the focus is inside the application under test	an element belonging to another application is not an answer about this one
the notification area can be searched	the shell decides whether the overflow opens, and a taskbar something is covering has no chevron to open it with
an overflow flyout this run can work	the chevron belongs to the shell and so does the flyout it opens, and a taskbar something is covering answers neither
the tray icon can be reached and asked	the route to a tray menu is focus and then the application key, and a desk that gives neither stops the act before it starts
no input this run did not synthesise	somebody using the machine during a run is the machine's business, and a run cannot tell its own synthesised input from a second person's
nothing stands over the region being captured	a window somebody else left over the region is on the desk, and no capture of that rectangle is a capture of what was underneath
this desk draws the shadow behind a menu	WW450: a menu's shadow is a setting of the machine, and a desk tuned for speed switches it off — so a menu with nothing behind it is the desk and never the application

What to do: give the run a desk of its own. A VM with an interactive session, a dedicated runner, or any machine nobody is sitting at. Re-running on the same busy desk is the one response that changes nothing — and if you are working alongside the run, your own typing is landing in the application under test.

Your application’s — a repository to open

Everything else the engine declares. Re-running changes nothing about these: a stale binary is stale on the second run too.

a binary built from this source
an application that renders its own tree when asked
every launch this run made was still running when it was asked to stop
everything this run started has left the machine
nothing outlived the run that started it
the application is in the language this scenario is written for
the arguments this run passed at launch
the page has finished computing
the pointer acts say why they are pointer acts
the region held still while the capture was taken
the run leaves the machine as it found it
the running instance is the binary this run named
the window's own glass carries nothing from behind it
the window's own pixels carry nothing from behind it
two renders of the same code can be compared

What to do: read the condition. Most of them name exactly what was not true — a binary built from this source, a page that has finished computing, an application in the language the case was written for. Several are answered by a key winwright.json does not declare yet, which is the last column of Declaring a project.

Where to go next

What this is — why the third verdict exists at all.
The verbs — which verbs need a desk, so you can tell before a run rather than after it.
Declaring a project — the keys whose absence is recorded as a reading not taken.