Arazzo Conformance

Loading…

Each case is a small Arazzo document that is either valid or breaks exactly one rule of the specification. Every tool is run against every case and judged on one question: did it call the document valid or invalid, and was it right? Schema cases are decided by the official JSON Schema; semantic cases break a rule only the spec's prose states (unique stepIds, operationIds that resolve, runtime expressions that parse), which a schema cannot catch.

Cases come from the OAI's own schema tests (Apache-2.0) and from API Evangelist's suite. Source, cases and the runner: api-evangelist/arazzo-conformance.

Scores by tool

Every case

✓ right verdict · ✗ wrong verdict · — tool declined the document's Arazzo version · ! tool crashed · ○ contested, shown but not scored. Hover a cell for the tool's output.

Execution

Validation asks whether a tool reads a document correctly; execution asks whether a runner does what the workflow says. Each case runs one workflow against a mock petstore that records every request. A case passes when the run ends the way the spec says it should and the mock saw exactly the expected requests, in order, carrying the expected values. Contested cases are run and shown but not scored.

✓ right outcome and requests · ✗ wrong · ○ contested, shown but not scored. Hover a cell for what the runner did.