Usecase Factory · AI testing workflow design

A model generates a runnable negative test. The request fails and the test passes, but its setup never created the account. Execution stopped at “account not found” before reaching the intended insufficient-balance branch. Neither syntax checks nor a passing run expose this drift in testing intent.

Usecase Factory addresses this by separating intent, rule sources, executable capabilities, and observations. I defined the direction from an existing testing process, organized refactoring and MVP plans, and established generation, review, recovery, and acceptance requirements through design review. The deliverables are design and review materials; a complete execution implementation has not been verified.

Relationships a scenario needs to preserve

The inventory example below is an independent explanation of the design. Insufficient stock must reject the request without changing resources. A scenario connects that rule to inputs, calls, and observations.

Object	Example	Reason to keep it distinct
Rule	R2: reject without reducing stock or creating a record	Supplies expectations independently of execution
Scenario	C02: stock 1, request 2, other conditions valid	Explains how to reach the target branch
Capability binding	Create item, reserve, read stock, look up reservation by request ID	Establishes what can be executed and observed
Generated artifact	Test file and digest for the scenario version	Detects code that no longer matches review
Observation	Response code, stock, record, and logs	Supplies facts for diagnosis

Suppose the available API can query stock but cannot look up a reservation. C02 can check the response and stock, but its last assertion has no observation method. Keep that gap with the scenario until the capability is available. Removing the assertion might make the script run, but it would change the rule being tested.

Block only affected scenarios

The same task may also contain independent scenarios, such as a successful reservation. C02 being blocked should not discard their completed reviews or execution results. Pause the work that depends on the missing capability and record what recovery will need: the scenario, capability, affected assertion, and resumption condition:

{
  "case_id": "C02",
  "case_version": 3,
  "status": "blocked",
  "missing_capability": "reservation.lookup_by_request_id",
  "affected_assertion": "no_reservation_created",
  "resume_when": "capability registered, semantics reviewed, scenario rebound"
}

This is a design example, not an existing platform API. Adding the lookup capability requires rebinding C02, reviewing affected scenarios, and regenerating code. Unchanged independent scenarios can retain their results. If rule R2 changes instead, every referencing scenario needs reconsideration, not just the latest failure.

Manual edits are inputs too. Keep the original artifact, patch, and rationale, then create a new version. Overwriting the output loses diagnostic evidence and conceals the difference between a correct model output and a human-repaired deliverable.

Diagnose a failure before changing expectations

Suppose C02 returns insufficient stock but a query finds a reservation record. Present the rule, scenario, actual request, and observation together. The rule may omit an allowed failure-record convention; the query may mistake an audit record for a reservation; or the implementation may violate the rule.

Those explanations require different corrections: confirm the business contract, correct capability semantics, or retain the failure and fix the product. Simply changing the assertion to allow a record bypasses that decision.

Generation, publication, platform synchronization, and execution therefore need separate records. A generated file does not prove platform receipt. Receipt does not prove execution, and execution does not establish a complete, acceptable result. Recovery should resume from the stage that actually stopped.

How to evaluate the MVP

Give the same rules and capability material to a single-prompt baseline and a staged workflow. Fix tools, budgets, and review standards. Preserve raw outputs and manual patches, then compare useful scenarios, defect detection, review time, and repeated work after recovery.

Start with three faults: a capability gap blocks only dependent scenarios; a rule change invalidates affected reviews; a missing result prevents batch acceptance. Broaden the domain after these behaviors work, rather than implementing platform synchronization before expectation sources and acceptance are settled.

The public ai-test-workflow implements staged artifacts, review association, template generation, and execution reports for single-item inventory. The Chinese tutorial documents its commands and formats. It is not the complete general-purpose Usecase Factory platform: dependency-aware recovery, publication, and synchronization remain separate implementation and validation work.