Anyone can claim
this one you can check
Every model MatrixAI builds carries a receipt, a manifest and a package that reproduces. Somebody else — an auditor, a reviewer, a customer — can re-run the check without asking you for anything and without believing you.
This page is about what can be proved about a model — where it came from, with what data, what was measured and over what — and about nothing else. It does not say a model is good. The last block on this page lists what none of this demonstrates, and it is not small print: it is half the argument.
Inside the Studio it is the Audit phase (or the Verification view in the Workbench). From a terminal it is matrixai verify, matrixai attest, matrixai report --tripod and matrixai bom. And with nothing installed at all, it is the three cases at /casos, already run, with their literal output.
A number nobody can re-check is an opinion
A model ships as a file and a figure: «94% accuracy». Everything that decides whether that figure means anything — which data, which split, which seed, which library version, whether the weights that travel are the ones that were trained — stays behind, in a notebook, in somebody's head, or nowhere.
That is fine until someone has to answer for it: an audit, a regulator, a customer asking how the model that decides about them was built. Then the figure is worth what the person saying it is worth.
What a receipt is, and what it is for
A receipt is a small JSON document that records one thing that happened: this model, over this data, produced this number, here and now. It carries the digests (sha256) of the model and of the data, the environment it ran in, the metric with its direction and its tolerance, and — always — a line saying what it does not attest.
- It is emitted when something runs: a pipeline, an evaluation. A simulation emits none, and says why — paper that describes nothing reads as proof that something happened.
- It can be signed, and signing is a separate act: whoever signs decides with which key. An unsigned receipt is level A0 and says so.
- Whoever receives it can compare two receipts of the same model and read the difference without opening a terminal.
A0 to A4: what a receipt actually holds up
A level anybody can write for themselves means nothing, so the level is deduced from what the receipt carries — the assurance_level written inside the document is ignored on purpose.
| Level | What it holds up |
|---|---|
A0 | Unsigned. Structured and readable, but it does not prove whose it is: anyone can write one just like it. |
A1 | Signed. It has not been altered between whoever issued it and whoever reads it. |
A2 | Signed and with reproducible evidence: the recipe and the dataset digest travel, so the result can be rebuilt. |
A3–A4 | Reserved for attested runtime environments and proofs this route does not give today. They are not claimed. |
Four stages, and one of them is «I could not check this»
matrixai verify rebuilds the package and reports four stages separately. Each one has its own verdict, and there are four possible verdicts — not two:
| Stage | What it checks |
|---|---|
manifest | Every artifact in the package matches the sha256 the manifest declares — and nothing travels that the manifest does not name. |
R1 | The dataset rebuilt from the recipe has the full sha256 the package declares. |
training | Training runs to completion with what the package carries inside. |
R3 | The metrics of that fresh run fall inside their declared tolerance. |
The interesting verdict is INCOMPARABLE. A package trained on data that cannot be rebuilt is an honest, common case — calling it FAIL would accuse it of being tampered with. So the report says «I could not compare this, and here is why», and the exit code lets a script tell the three cases apart without reading the JSON.
The manifest: what would be checked, and with which numbers
Every package carries a reproduce.json: the seeds (data, split, weight initialisation), the three epoch counts —declared, effective, actually run—, the artifacts with their digests, and the metrics with the dataset digest they were measured on.
- Three epoch numbers, not one. What the contract asked for, what the cap left, and what actually ran are three different questions, and a package that answers only the first is hiding the other two.
- A metric without its dataset digest is unusable. «94%» over which rows? The manifest says which, and almost nobody publishes that.
- What is missing is listed. Not blank, not filled in: named, so you know what to ask whoever made the package for.
Measuring a model MatrixAI did not train
You can bring an ONNX model from anywhere — PyTorch, TensorFlow, scikit-learn, a vendor — plus an evaluation CSV. It runs, it is measured, and the number is tied to the digests of both files in a receipt somebody else can re-check.
matrixai attest model.onnx --data eval.csv --target-column y \
--key <hex> --in-totoWhat that is worth, and what it is not:
- You can compare vendors on the same data. Two receipts, same CSV digest, two numbers — and nobody has to trust anybody's slide deck.
- The measurement is pinned. Change one row of the CSV and the digest changes: a receipt cannot be quietly reused for other data.
- It tops out at A1, and the screen says so before you upload anything. Higher levels speak about runtime environments and proofs this route does not give.
The clinical checklist, filled in from what the run recorded
TRIPOD+AI (BMJ, 2024) is the 27-item checklist required to publish a study that develops or validates a clinical prediction model. The measured reality is that most papers still do not meet it: it gets filled in by hand, at the end, from memory.
matrixai report <package> --tripod --locale en -o record.mdThe command invents nothing. It writes what the run already recorded, in the format that audience has to hand in — and it names the boxes it cannot fill.
| What TRIPOD+AI asks for | Where it comes from |
|---|---|
| Data source and sample size | the data recipe and artifacts.dataset.rows — and with a recipe, the source is declared synthetic, written as such |
| Predictors and excluded columns | field_types, field_ranges, excluded_identifiers |
| Split and randomisation | the three seeds and the contract's SPLIT |
| Modelling method and hyperparameters | the .mxai and the .mxtrain, both readable, both travelling in the package |
| Environment and versions | python, platform, packages, backend and device |
| Performance metrics | with the dataset digest they were measured on |
| Model availability | the package: ONNX + predict.py, usable with no MatrixAI installed |
| Missing data · Calibration · Subgroup fairness | Not computed today — and the record says so, one box at a time |
| Funding · Conflicts · Ethical approval · Intended use | A person writes these. The report asks for them; it does not invent them |
Not a private format
A receipt only a MatrixAI can read is a receipt worth nothing outside MatrixAI. So the same content also travels in the envelopes the rest of the world already reads:
| Standard | What it gives you |
|---|---|
| DSSE | the signature envelope, with the payload in base64 as the spec requires |
| in-toto | the same receipt as a Statement, with its subjects and their digests — matrixai attest --in-toto |
| CycloneDX 1.6 ML-BOM | the package's bill of materials, validated against the official schema and deterministic: its serial number comes from the manifest digest, not from a random one — matrixai bom |
| Sigstore | keyless signing with a verifiable identity — matrixai attest --sigstore |
--sigstore never falls back to HMAC in silence — if the library or the identity is missing it says which, and writes no receipt. Falling back to another signature would assume you do not care who answers for the document.Three cases, already run, with their literal output
You do not have to take any of this on trust. The gallery ships three models with their recipe, their training contract, the digests of everything, the package that predicts with no MatrixAI installed, and the verification report exactly as it comes out.
- Celsius to Kelvin — the four stages in PASS. If the package returns 273.15 for 0 °C it works, and if it returned anything else you would see it without knowing any machine learning.
- Readmission risk (synthetic) — the same, plus the full TRIPOD+AI record produced by the command.
- Will it rain tomorrow (real data) — this one does not come out clean, and it is published anyway. There is no recipe that regenerates real observations, so two of the four stages come back INCOMPARABLE and the report says so instead of keeping quiet.