Post-training for a testing workflow

A model may repeatedly reduce “insufficient stock leaves all resources unchanged” to “returns an error code.” The output has valid structure but omits valuable state assertions. That is worth recording, though it does not yet justify training: missing context or a missing observation API can produce the same symptom.

This planned work follows the AI testing workflow. The candidate task is generating test scenarios from confirmed rules. It starts with a baseline before deciding whether task-specific fine-tuning is warranted. No training or model-effectiveness results are available yet; this page defines the experiment.

Attribute errors before choosing a remedy

Return to the scenario with the missing assertion and freeze the requirements, contract, domain model, capabilities, and output format used to generate it. Keep the raw output. Then check whether the rule required unchanged resources, whether the model received that rule, and whether the tools could query those resources. These checks locate the omission before choosing a remedy.

Observation	First area to investigate	Attribution
Output invents a query API	Capability retrieval and calling constraints	First establish that the material was complete
Expected code comes from old documentation	Input versions and retrieval	Training cannot repair incorrect source material
Rules and observation APIs are present, but state assertions are repeatedly omitted	Prompt and examples, then possible fine-tuning	Candidate model-behavior error
Correct scenario fails because the environment is unavailable	Execution environment	Not a reasoning failure

If a reviewer added “rejection must create no record” only after generation, the earlier omission cannot be counted as failure to follow that input rule. Keep this attribution with the candidate sample and record environment failures separately.

Turn a reviewed correction into a sample

A candidate record can link the input snapshot, rule IDs, original output, reviewed output, and correction rationale. This is a conceptual sample manifest, not a provider’s training format:

{
  "task_id": "inventory-insufficient-stock",
  "source_group": "inventory-spec-v1",
  "input_snapshot": "inputs/manifest.json",
  "rule_ids": ["R2"],
  "original_output": "candidate/cases.json",
  "reviewed_output": "reviewed/cases.json",
  "correction": "Add unchanged-stock and absent-reservation assertions",
  "reason": "The rule requires both, and the capability list supports observation"
}

The corrected output still needs business review. Passing tests alone do not make it a training target: incorrect expectations can agree with an incorrect implementation. Material must also meet the team’s data-use rules, without real credentials, tickets, or customer data in samples.

Split by business source and compare three alternatives

Paraphrases of one rule and parameter variants of one script belong to the same group. Randomly scattering them across training and evaluation can underestimate the difficulty of handling a new rule. Split by requirement source or business operation first, then organize samples within each group. Keep the evaluation set out of repeated prompt tuning.

The planned comparison has three arms: the current prompt baseline, an improved prompt with rule explanations and a few examples, and a fine-tuned candidate using the same inputs and tools. Record the untrained baseline first. Keep budgets, human involvement, and retry allowances comparable. Model choice, training method, sample count, and hyperparameters follow data inspection; no untested configuration is prescribed here.

Decide using testing value and total cost

Alongside valid JSON, check whether scenarios reach the intended branch, assertions cover side effects, known defects are detected, and human edits are needed. Record execution costs too. The Chinese agent regression article explains the evaluation approach.

Set adoption conditions before the experiment: no regression on critical held-out cases, an acceptable improvement in review effort, and inference and maintenance costs within the team’s budget. Include training, data review, deployment, and rollback. If better material and prompts solve the problem, stop there.

The next deliverable is therefore a versioned task set, an untrained baseline, error categories, and reviewed corrections. If the model repeatedly misses the same assertions despite complete material and tools, those records can justify a training experiment and provide the basis for evaluating it.