XAT: making concurrent test results trustworthy

XAT is a Python API testing tool built on unittest and requests. It runs cases, controls concurrency, collects logs, and produces reports. I define the scope and execution behavior and collaborate with AI on implementation and code review. The project develops questions from my payments testing work while leaving company-specific authentication, test data, and business assertions to the integrating team.

Suppose one test passes, another fails an assertion, and a third loses its process before returning. A final “failed” message leaves several questions open: which methods finished, which result is missing, and which logs are still available? This article follows those tasks from discovery to reporting to explain how XAT keeps that information.

Requirements and a minimal check

The reviewed version is 0.6.0 (d8a6f672), requiring Python ≥3.11 and requests ≥2.32. Obtain authorized access to the XAT source first; this site is not a hosted runner, and a similarly named PyPI package is not an installation source for this project. In a virtual environment inside the authorized checkout:

python -m pip install -e .
xatrun test_report_example.py -c 2 --json-report result.json

Reuse the three-method input in the result-integrity article. It needs no business server: a passing assertion, an assertion followed by a cleanup error, and a worker exit yield SUCCESS, SCRIPT_ERROR, and SYSTEM_ERROR. Expect complete=false and exit code 1. That failure is the intended observation, not an installation error; the article explains batch-level issues as well.

When an execution layer is useful

A small set of independent tests can run directly with unittest. Another runner becomes useful when multiple methods need consistent scheduling and other results must still be collected after one task stops responding.

Consider a file with two methods. A completes normally; B calls os._exit(7) during execution. B bypasses unittest’s exception collection and never returns a normal report. Counting only returned reports makes B disappear, potentially turning the batch into “all passed.” Keeping only the process exit code discards the fact that A completed.

To notice B is missing, the runner must first know that the batch should contain A and B, then match the returned results against that list. The same comparison works under concurrency: task identity establishes what completed, regardless of which log appeared first.

Freeze the task list before execution

The default scheduling unit is one test_* method. Separate discovery processes import files and produce method lists. The parent checks the file and method identities, freezes the list, and queues tasks. A file that fails discovery does not execute a partially collected list.

The diagram focuses on method mode. A worker is a Python child process executing a test; the parent’s thread pool starts and waits for those processes.

Scroll horizontally if needed.Open full-size diagramOpen full-size diagram

xatrun tests/orders tests/users -c 4 --case-timeout 600 --json-report result.json

-c 4 limits simultaneous workers. After a task finishes and its log is printed, the parent fills the available slot. Tests run in fresh interpreters, so module variables and request Sessions are not directly shared across methods. Concurrent logs are collected by task before being printed, reducing interleaving between lifecycle phases.

Discovery also imports modules. Top-level code may therefore run during discovery and again during execution. Creating orders or changing database records belongs in managed fixtures, the preparation and cleanup around a test, rather than code that assumes a module will be imported once.

Process isolation changes fixture scope

A fresh interpreter separates in-process state and lets the parent terminate an unresponsive task. It also repeats interpreter startup, imports, and fixture preparation. Class and module fixtures run again for each method in method mode.

Mode	Work assigned to one process	Conditions and cost
Default method mode	One method and its fixtures	Methods prepare data independently; methods in one file can overlap, but shared preparation is repeated
--parallel-scope file	A file, executed serially within the worker	Preserves shared fixtures within the file; a stuck method affects later methods in that file

There is no performance comparison here, so more processes are not presented as a measured speedup. The choice depends on data dependencies and preparation cost. Separate interpreters can still use the same account or modify the same database row; they do not isolate external resources.

Execution units and fixture scope explains discovery, scheduling, and timing in more detail.

Record test failure separately from missing execution information

A negative test receiving HTTP 400 can pass when its assertions match. HTTP 200 can still fail a business assertion. XAT uses unittest’s assertion and exception outcomes rather than deciding from the HTTP status alone.

For this batch, method reports first describe each outcome: success, assertion failure, script error, system error, or method timeout. The batch field complete then says whether execution information was fully collected, while exit_code lets a command-line caller decide whether to proceed. An assertion failure and a missing report can therefore lead to different explanations.

What happened	Method records	Batch meaning
A passes; B fails an assertion; both return normally	Success and assertion failure	complete=true, exit 1: results returned, tests did not pass
A passes; B’s worker exits early	Success and a system error marked interrupted	complete=false, exit 1: B’s normal execution result is missing
A passes; the parent times out B	Success and timeout	complete=false, exit 1: B was terminated; cleanup is not guaranteed

Readable JSON alone does not establish that a report is valid. The parent checks the schema, file and method identities, collected count, and agreement between the reported exit code and the process exit code. A method disappearing after discovery or a corrupted report must not become a success or silently vanish from the totals.

Completeness does not mean every method ran. A class fixture can prevent a method from starting while its failure is still fully reported. Skips and expected failures have separate records and do not count as successful methods. See result states and completeness for examples.

What happens after a method deadline

An HTTP timeout is an exception business code may handle. A method deadline requires the runner to act when that code does not return voluntarily.

Workers write checkpoints containing the active method, its start time, and completed results. The parent polls progress. After detecting an expired deadline, it reads again to confirm that the active method has not changed, then terminates the worker. This matters in file mode: the worker might have moved from A to B, and A’s old start time must not be used to kill B. Normal exits also receive an elapsed-time check, covering completion between polls.

A checkpoint is written to a temporary file and replaced into place. Its report is always marked incomplete: progress cannot stand in for a successful final report. In method mode the deadline covers the suite’s class/module fixtures, setup, test, teardown, cleanup, and fixture finalization. It excludes queuing, discovery, and interpreter startup. It is not a command-wide deadline; a stalled import or discovery step remains outside this protection.

addCleanup handles exceptions within a running Python lifecycle. After forced process termination, Python cleanup cannot continue. External orders, accounts, or locks need business-specific reclamation. Resource acquisition and cleanup order includes runnable examples.

Check the model with faults

This account uses XAT revision d8a6f672. Verification exercises real subprocesses and the observations below. It checks execution behavior, not production throughput or defect-detection effectiveness.

Verification input	Observation checked	Test file
One method exits abruptly; another completes	Preserve success; attach identity and system error to the exited method; mark the batch incomplete	test_method_scheduling.py
Assertion failure followed by a cleanup exception	Preserve both causes; select script error as the primary state	test_results.py
CPU loop that catches exceptions	Parent terminates it; later method-mode tasks run; cleanup does not run	test_deadline.py
Modified report version or method identity	Reject the mismatched report and record an execution error	test_results.py, test_method_scheduling.py

Related problems occur in remote runners, but this implementation uses local subprocesses. Leases, ownership after a network partition, cross-host retries, and durable recovery are outside its guarantees. A future remote executor can reuse the idea of task identity and result validation, but must separately define ownership and side-effect handling after disconnection.