XAT reports: separating failure, missing results, and exit codes

In a batch, A passes, B fails an assertion, and C’s process exits abruptly. How should the summary represent it? Counting only normal reports makes C disappear. Marking the whole batch failed without detail leaves the reader unable to distinguish business assertions, script problems, and runner failures.

Each test needs its own account: A completed, B returned an assertion failure, and C never returned a normal result. The batch also needs to say whether the information is complete and provide an exit code to command-line callers. The same example will show how XAT keeps these three layers separate.

Method states follow execution facts

unittest already provides assertion-failure and error callbacks. XAT collects structured issues from them, adding system errors and deadlines detected by the parent.

State	Trigger	What it does not establish
SUCCESS	Successful method-level execution; class/module fixture errors are recorded separately	HTTP must be 200; negative assertions can pass
ASSERT_ERROR	An assertion fails, or expectedFailure unexpectedly succeeds	The product is necessarily wrong; the assertion might be wrong
SCRIPT_ERROR	An unhandled script or lifecycle exception	A root cause from its name alone; request timeouts can appear here
SYSTEM_ERROR	Worker, result protocol, or framework execution failure	A business assertion ran and failed
TIMEOUT_ERROR	Runner determines that the method exceeded its limit	The same event as an HTTP client raising TimeoutError

An exception correctly checked with assertRaises can still produce success. Skips and expected failures use separate execution annotations. They are neither a sixth failure state nor successful methods.

One method can have several causes

If an assertion fails and cleanup then divides by zero, the method has two issues. Its primary state follows SYSTEM_ERROR > TIMEOUT_ERROR > SCRIPT_ERROR > ASSERT_ERROR, while all detailed causes remain.

The summary can classify this method as a script error while its details still show the earlier assertion failure. Counts follow the same relationship: one failed method, two error events. Two issues do not mean that two methods ran.

The following fault example runs with the current XAT version. It uses local processes and no business service. os._exit simulates a worker unable to return a report.

# Save as test_report_example.py
import os
from xat import BaseCase

class ReportExample(BaseCase):
    def test_a_ok(self):
        self.assertEqual(2 + 2, 4)

    def test_b_two_failures(self):
        self.addCleanup(lambda: 1 / 0)
        self.assertEqual(1, 2, "business assertion")

    def test_c_worker_exit(self):
        os._exit(7)

xatrun test_report_example.py -c 2 --json-report result.json

Under method mode in verified revision d8a6f672, this produces three method records: A is SUCCESS; B is SCRIPT_ERROR with assertion_failed and script_error issues; C is SYSTEM_ERROR marked interrupted. Completion order may change display order, but identity-based interpretation stays the same. C’s exit reason is a batch-level issue whose target identifies the method, so a reader must inspect batch issues as well as each method’s own issues.

Completeness and exit codes

B failed its assertion, but the runner received a full account of that failure. For C, the parent can record interruption; adding an error placeholder does not recover the report C should have returned. complete distinguishes those situations, so it cannot be calculated by comparing the passed count with the total.

Situation	complete	Exit code
Methods return normally and pass	true	0
Assertion or script failures with complete reporting	true	1
Worker crash, missing report, or untrusted report	false	1
Method terminated or classified after exceeding its deadline	false	1
User interrupt	false	130

The three-method example is therefore complete=false, exit 1. Remove C and B still fails, but the batch can be complete=true.

Two further boundaries matter. An all-skipped batch can exit 0 while having no successful methods. An empty collection exits 3; invalid input exits 2. A caller requiring at least one executed, successful method needs method records rather than exit 0 alone.

Which report belongs to which task

A process returning JSON establishes only that data exists. The parent also checks the relationship to its submitted task: matching file and method identities, valid result counts, a supported schema, and agreement between process and report exit codes.

In method mode, a method disappearing after discovery, a report naming another method, or duplicate or missing records must not merge unchecked. A task with known identity receives the corresponding system error. Discovery failure has no established method list and can only produce file or discovery-level errors. Class/module fixture failure may also prevent a method from starting; a fictitious executed-method result would be misleading.

Checkpoints and final reports serve different purposes. A checkpoint preserves progress and helps identify interrupted work. A final report carries a complete return. If a checkpoint is untrusted, the parent can still mark a submitted task as failed at the system level, but cannot infer which business statement it reached.

Break the result channel deliberately

Success and assertion-failure tests alone miss report corruption. Regression tests also modify the schema when a worker exits or create method-identity changes, checking that the parent rejects incorrect reports. Interrupt tests check retention of received results.

Another test makes the final JSON write fail. The method already succeeded, yet the command must exit nonzero and record report_write_error. Report-writing failure must not rewrite that method as an assertion failure. This is another reason to separate method and command states.

These checks cover representation and transmission of execution results. For side-effecting tests, a disconnected worker still cannot tell the runner whether a server-side operation occurred. Business state needs checking before replay. The result model exposes that uncertainty; it does not supply an idempotency guarantee for the business API.