Minctrl Docs
Guides

Testing processes

Write test cases for a process — inputs, the answer each gate gets, the path and outcome the run must take — run them against any version without touching a live system, compare two versions on the same suite, and make publishing require a green suite.

A change to a process changes what happens to real cases. A test suite says, before you publish, what the process must do: for these inputs, with these answers at the gates, the run goes through these steps and ends like this. You run the suite against any version — a draft too — and compare two versions on it side by side.

A suite run is never live:

What a run doesWhat a suite run does instead
Calls the connectors bound to each stepEvery connector is shadowed: it records what it would have sent and sends nothing.
Parks at a gate and waits for a personThe gate is answered from the case, checked the way the sign-off card checks a signature.
Scores agent steps with the runtime's step scorerAnswers each agent step from its recorded fixture (unless the case says llm: "live").
Is a row in Runs, in analytics, in someone's queue, on the SLA clockIt is none of those. No run row, no park, no notification, no run_started or gate_signed entry.

What it leaves is the suite result and one audit entry, process_suite_run: who ran it, on which version (number and executable hash), and the result counts.

A test case

Open a process and go to the Tests tab. New case opens a JSON editor that checks the case as you type; the server checks it again on save and names every problem (422).

{
  "name": "High risk: the analyst approves, the number is released",
  "inputs": { "risk": "high", "confidence": 0.9 },
  "llm": "fixtures",
  "answers": [
    { "step": "signoff", "visit": 1, "outcome": "approved", "form": { "ticket": "FR-77" } }
  ],
  "fixtures": {
    "score": {
      "tool_results": [{ "tool": "fraud-scoring-model", "ok": true, "signal": { "risk": "high" } }],
      "confidence": 0.95
    }
  },
  "expect": {
    "status": "completed",
    "path": ["intake", "score", "risk", "stage", "signoff", "execute", "monitor"],
    "path_mode": "exact",
    "outputs": [
      { "step": "execute", "path": "$.tool_results[2].mode", "equals": "shadow" },
      { "step": "signoff", "path": "$.form.ticket", "equals": "FR-77" }
    ]
  }
}
KeyWhat it says
inputsThe run's starting facts, as POST /process-runs takes them.
auto_resolveDefault true — the autopilot, as the console runs it.
answersOne per gate pass: step (its id or its gate: label), visit (default 1 — a rework loop passes a gate again), outcome (approved, denied or a branch label), form (the gate form's values), approvers (see N-of-M gates).
fixturesRecorded outputs of agent steps, keyed "<step>" or "<step>#<visit>": tool_results (one per tool, matched by tool, else by position) and an optional confidence (0..1). A result's signal feeds the run state, so a fixture can route a gateway.
llm"fixtures" (default): steps with a fixture answer from it, the others run on the shadow connectors with the deterministic scorer — the suite is deterministic and free. "live": every step is scored by the runtime's step scorer, as a real run is.
expect.statusHow the run ends: completed, awaiting_human (it waits at a gate the case does not answer), max_steps or dead_end.
expect.pathStep ids in order. path_mode: "exact" — every step, nothing else; "in_order" — these steps, in this order, others may come between.
expect.outputsPer step (and visit): a JSONPath ($.a.b, $.list[0], $['odd key']) that equals a value or contains one (a substring, a list element, or a sub-object). An agent step's output is {tool_results, decision, resolver}; a gate's is {decision, branch, form}.

A scripted answer is checked like a signature: an outcome that names no branch of the gate, or form values the form refuses, is a failure of the case (answer_refused, naming the field), never a signature.

Make a case from a run

Make a case from a run (or POST /process-tests/{slug}/cases/from-run with a run_id) turns a run that already happened into a case: its inputs, every answer its gates were given — by step and visit, with the form values and, on an N-of-M gate, who approved — its path and final status as the expectation, and its agent steps' outputs as fixtures. Run the suite on the next version to see whether it still does the same.

Record fixtures from a live run

Record (or POST /process-tests/{slug}/cases/{id}/record with a version) runs the case once live — the runtime's step scorer, connectors still shadowed — and stores what each step produced as the case's fixtures; the case runs from them (llm: "fixtures") afterwards. A live run that crashed records nothing (409). The record is audited as a case change (process_test_case_changed, action recorded).

Run the suite

Pick a version and Run suite (POST /process-tests/{slug}/runs with {"version": 4}). The result:

  • status — passed (every case passed), failed, or error (a case could not run — it crashed; an error is never a pass);
  • per case: pass / fail / error, the first divergence — the step, what the case expected and what the run did (kind: path, status, output, answer_refused, approvals) — the path it took, its duration, and any answers or fixtures it never used.

The first divergence is found in this order: the path (every later difference follows from it), then the final status, then each output assertion.

A suite with no cases is refused (409); a version that is not runnable is refused (422, with its violations). If the audit entry cannot be written, the result is not stored (503).

Compare two versions

Compare (or POST /process-tests/{slug}/compare with {"a": 3, "b": 4}) runs the suite on both versions — two stored, audited suite runs — and lines them up per case, each side with its status and first divergence. A row is marked changed when the verdict, the divergence or the path differs: that is the behaviour change the new version brings.

N-of-M gates

On a gate with an approvals policy (approvals: {required: 2, …}, see N-of-M approvals) an answer says who approves:

  • "approvers": 2 — two distinct testers, each with the role the policy's slot needs;
  • "approvers": ["[email protected]", {"by": "[email protected]", "role": "admin"}] — named people; a delegate is {"by": …, "on_behalf_of": …} and counts as both.

The approvals are checked like the sign door checks them — different people when distinct_people is on, the role of each slot when order is sequential — and recorded for the gate's visit; the engine clears the gate only at quorum, as on a real run. Fewer approvals than required is a failure of kind approvals ("2 approvals" expected, "1 approval" given). A denial by anyone ends the gate whatever the count.

Publishing requires a green suite

An owner or admin can turn on Publishing needs a green test suite on the Settings page (PUT /governance/policy with {"publish_requires_green_suite": true}; off by default; the change is audited as policy_changed). With it on, publishing a version — and rolling back to one — is refused (409, saying why) unless the latest suite run on that version:

  • passed;
  • ran that version's exact executable graph (its hash);
  • ran the cases the process has now — a case added, changed or deleted since makes the result stale.

The Versions panel shows each version's suite (suite passed, suite stale, suite failed, not tested) and why Publish waits. When the results cannot be read the panel says not read — never "not tested" — and with the rule on, publishing waits. The process_published entry records suite_required and the suite_run_id it relied on.

API keys

ScopeRoutes
processes:readGET /process-tests/{slug}/cases, …/cases/{id}, …/runs, …/runs/{id}
processes:testPOST /process-tests/{slug}/runs, POST /process-tests/{slug}/compare

A key never writes a case (create, change, delete, record, from a run): those are for people above viewer. A viewer reads cases and results but does not run a suite.

import minctrl

ci = minctrl.MinctrlClient("https://app.minctrl.com", api_key="mk_live_…")   # processes:test
result = ci.tests.run("prior-auth", version=4)
if not ci.tests.passed(result):
    for case in result["results"]:
        if case["status"] != "pass":
            print(case["name"], case["divergence"] or case["error"])
import { createMinctrl, suitePassed } from "@minctrl/sdk";

const mc = createMinctrl({ baseUrl: "https://app.minctrl.com", apiKey: process.env.MINCTRL_API_KEY });
const run = await mc.tests.run("prior-auth", 4);
if (!suitePassed(run)) process.exit(1);

On this page