AI testing workflow: from API material to acceptance

In payments testing, I built a staged LLM workflow: read requirements and API documentation, learn the structure of existing tests and the business domain, produce test scenarios, then fill a contract-testing skeleton with setup, business calls, and assertions. Human review and execution checks were part of delivery.

The work has a recorded delivery of more than 200 test files in roughly four weeks, with approximately sevenfold productivity improvement. Those results depended on the existing business APIs, framework, and review process. This article explains the transferable method. The separate inventory implementation, prompts, Skill, and commands are in the Chinese end-to-end tutorial.

Establish the rules before generating code

Suppose the model needs to generate an insufficient-funds test. An API table tells it that the amount is an integer, but may not say whether partial execution is allowed, repeated requests reuse a result, or resources must remain unchanged after rejection. Without those rules, runnable code can still have the wrong setup and assertions.

The use-case workflow I organized separates interaction steps, field lists, and business rules, then connects them through step identifiers. A rule has an explicit place: which condition is checked at which step, and what follows a failure. Funds workflows also distinguish synchronous completion, acceptance followed by processing, and acceptance alone. An accepted request does not establish the correctness of the final funds state.

For another team, start with five kinds of material. They need identifiable versions and source locations; they do not all need a new documentation format.

Input	Question it must answer	Consequence if missing
Requirements and interaction flow	What does the system promise, and where can a path terminate?	Coverage cannot be tied to the requirement
API contract	What can be sent, and what do responses mean?	Parameters or error codes may be bound incorrectly
Domain rules	How are objects related, and what state changes?	Response checks may miss side effects
Existing tests	How do base classes, fixtures, assertions, and cleanup work?	Generated code may not run in the framework
Testability APIs	How can state be prepared, observed, and cleaned up?	A scenario may be impossible in the available environment

Choose existing tests by purpose. A successful case shows ordinary calls, a negative case shows how invalid input reaches the system, and a cleanup example shows the resource lifecycle. One supposedly universal script can spread its business exceptions into every new case.

Preserve the business decision in a scenario

Consider the tutorial’s inventory reservation example. The requirement says insufficient stock must reject the reservation, leave stock unchanged, and create no reservation record. Write those three expectations before choosing observation APIs.

Part	Insufficient-stock scenario
Rule	Reject without changing either resource
Setup	Item exists, available stock is 1, other inputs are valid
Action	Request 2 units with a new request identifier
Response assertion	Insufficient-stock code
State assertions	Stock remains 1; no reservation exists for that identifier

If the item does not exist, validation may stop before reaching the stock check. A rejected request would not test the intended rule. Reviewers need to check that the target condition fails while the prerequisites for reaching that condition hold.

Bind setup, invocation, observation, and cleanup to available capabilities only after this review. If lookup by request identifier is unavailable, retain that assertion as a capability gap. If an SDK replaces invalid quantities with defaults, it cannot test the server’s rejection of the original input.

Generate business logic within the framework’s conventions

The existing payments APIs and SDK provide reusable operations. The framework provides discovery, resource lifecycle, and reporting conventions. The model maps reviewed scenarios onto those capabilities to generate setup, calls, and checks.

In the inventory example, first check whether the scenario requires no reservation record. Then check whether the code actually queries for that record and asserts its absence. This separates review of testing intent from review of its implementation. If an assertion fails, preserve requests, responses, and state snapshots before changing anything; replacing the expected value with the observed one would erase the evidence that needs diagnosis.

Problem	Stage to revisit	Check after correction
Acceptance mistaken for completion	Requirements and rules	Define the terminal state and how to observe it
SDK rewrites the invalid input	Capability binding	Inspect the request that actually reaches the system
Setup did not take effect	Preparation and environment	Read back the precondition before the business action
Rejected request leaves a business record	Implementation or rule-conflict investigation	Preserve the failure and judge it against the confirmed rule
Worker exits without a complete report	Runner	Treat missing results as incomplete, never as passes

XAT result integrity distinguishes individual outcomes from batch completeness. The workflow uses that execution evidence to decide whether delivery is acceptable.

Keep the process and replace the business adapter

XAT was extracted from an internal contract-testing framework by removing company-specific protocols, data formats, authentication, and tickets. The workflow needs the same separation.

Material preparation, scenario review, capability binding, skeleton generation, and acceptance can transfer. Business vocabulary, rule sources, API adapters, environments, and organizational approvals must be replaced. The public tutorial’s single-item inventory format also needs to change for another domain; changing a URL is insufficient.

Start a pilot with one successful case and one business rejection. Confirm setup, actual request delivery, observable results, and cleanup before adding repeated requests or asynchronous terminal states. Business reviewers own rule meaning, testers own scenarios, and test developers maintain the adapter and execution boundaries.

The public ai-test-workflow is a teaching implementation with executable reference cases and seeded defects. Its measurements are separate from the workplace results above. Usecase Factory explores selective blocking and recovery; the post-training plan asks when changing the model itself would be justified.