How we measure which engine wins

The new MatrixAI network, on the field of the other engines

What "the best engine" means here

On each dataset, every engine is trained with its default settings — no hyperparameter search — and scored with the metric that task uses. The engine with the best average is the winner of that dataset; any other engine that lands within a fixed number of points of that winner still counts as having met the bar there. A crash or a timeout counts as a lost dataset for that engine, not as a zero.

the engine with the best mean over the fits on THAT dataset, excluding the dummy baseline; a failure (timeout/crash) counts as a lost dataset for that engine

kc2 — Binary classification

Worked from the same JSON this whole page reads — nothing here is typed by hand. · 522×22 · OpenML ↗

The previous network is shown for reference only: on the new field it is replaced, so it has no distance to the best here.

On kc2, the best engine was MatrixAI network, at 0.919. Of the 6 engines that competed, 3 stayed within 2 points of the best and so meet the rule on this dataset.

Who builds the model: MatrixAI directs, the engines learn

The general downloadable package never carries ONNX. Only when the winner is LightGBM, on a binary classification with its decision already chosen, can MatrixAI also generate a page that predicts in the browser with no server, in ONNX format, with its parity checked against real rows before it is published — neither the baseline nor the dense net export to ONNX.

The table below measures exactly the engine’s part: every engine learns with the same prepared data, the same split and the same time, and MatrixAI does the rest the same way for all of them.

One case is entirely MatrixAI’s own: our neural network. It is not a third-party library — MatrixAI’s core generates it and trains it. Its architecture builds on TabM; the section on the new network below says so.

40 datasets × 7 engines

Same field as the counts above. Filter by task or by dataset size. On a phone the table keeps its own horizontal scroll — the page itself never scrolls sideways.

This table is Phase 0: every engine got 4 threads and between 120 and 600 s per attempt, measured on each dataset’s own test set. A study in the Studio gives each attempt 1 thread and at most about 22 s, and picks its champion by cross-validation on the data of whoever runs it — so when the gaps are small, it can pick a different engine than this table, and both measurements are right about what they measure.

• ✓ meets the bar (within 2 points of the best on that dataset)
• ★ best engine on that dataset, on the new field (the baseline never competes)
• — crashed or timed out (counted as a lost dataset, not a zero)

Datasets marked SEALED are the portfolio’s final exam: they are never used to pick or tune anything, only to grade the result once, at the end.

Baseline is a dummy predictor shown for reference only — it never competes for "the best" and is excluded from every verdict below.

The last column is the previous network as measured in the frozen v2 table. On this field it is replaced by the new network, so it carries no ✓ or ★ here.

The two bottom rows count all 40 datasets, whatever the filter above. For the previous network, the count in its own v2 field.

MatrixAI’s new neural network, measured

It is a neural network built for tabular data, like a spreadsheet. Each row goes through several small networks that share most of their weights, and their answers are averaged. Before that, every numeric value is turned into a small vector of sines and cosines whose frequencies the network learns, so that it reads numbers well.

It runs on a CPU, with no GPU (a GPU is not required), and it is the engine that replaces our previous dense network in this measurement.

Its architecture builds on TabM with PLR embeddings (Gorishniy et al., Yandex Research, 2022 and 2025). The CPU adaptation, its integration into MatrixAI, the protocol and the measurement are ours.

31 of 32 on the datasets we could tune against, plus 7 of 8 on the sealed ones. The previous network: 11 of 32, plus 5 of 8.

The 8 sealed datasets were measured one single time: no second try, no adjustments afterwards.

The records of this measurement ↓

It does not pass the rule: it met the bar on 15/40 datasets in the September 16, 2026 measurement and on 16/40 in the September 22, 2026 one, where it was the best engine on 4. On the field above it no longer competes: the new network takes its place.

In the downloadable Studio, since v2.8: in the GPU package the previous network keeps competing, as an experimental engine, and it only wins when it is the best on that dataset; in the CPU package the new network takes its place and the previous one no longer appears.

• Allstate_Claims_Severity trains only 5 epochs: the time limit cuts it short, well below the 17 the rule asks for as a minimum. It counts as measured, with that handicap.
• In classification, early stopping uses the log-loss of the ensemble, not the accuracy that the original source of the method uses. It changes which epoch is kept, so it changes the numbers; the protocol was already sealed, so it was not changed.
• The other networks and engines come from the frozen table of September 22, 2026 (v2); they were not measured again with the new network.
• The sealed datasets were measured once (110 runs, none repeated).

Under our own decision rule, a network that meets the bar on at least 32 of 40 datasets joins the Phase 0 portfolio of engines, and this one does.

That is a Phase 0 statement: 4 threads per attempt and between 120 and 600 s each. In the downloadable Studio every attempt gets 1 thread and about 22 s, and that is where it was measured on October 1, 2026, on 13 datasets, with the rule written before measuring: it completes 18 of 26 attempts, it is within 2 points or less of the best on 4 of 13 datasets, and it loses to the previous network on 5 (the limit was 2). It did not pass the rule, so on that date the downloadable Studio kept the previous network. That is history: see the next paragraph for what changed.

Then contract 120 put the new network into the study as one more engine, with its own time (60 s per attempt, added to the budget: no other engine loses a second), and measured it that way. It only competes if it was measured on every part of the cross-validation, and the study picks per dataset: it only wins where it is the best. It ships in the Studio since v2.8.

Measured in the Studio’s study, on 13 public datasets, without a GPU and with 6 GB of memory for the container. CPU package: The champion improves on 2 of 13 datasets (difference in points of its test score: dresses-sales +4.4, pc1 +1.6), drops on none and none stops completing. On 2 the study did not end with a champion, with the network or without it. On 2 the estimated memory exceeded the container’s and the network did not compete (Allstate_Claims_Severity: 15.3 GB estimated against 6 GB; KDDCup09_appetency: 7.8 GB estimated against 6 GB).

GPU package: The champion improves on 3 of 13 datasets (difference in points of its test score: dresses-sales +4.4, house_16H +1.8, pc1 +1.6), drops on none and none stops completing. On 3 the study did not end with a champion, with the network or without it. This run was also done without a GPU: what it does on a GPU has not been measured.

It competes in both packages. The CPU package downloads PyTorch for CPU (about 200 MB) on its first start, on the computer that installs it, like the GPU package already did; later starts download nothing. If the study’s estimated memory (about 1 GB plus half a GB per million cells, with a 15 % margin) exceeds the container’s, the network does not compete in that study and Results says so, with both figures. The hosted demo does not carry it.

Where each figure comes from, and whether it can be anchored to one exact commit

Three sources make up the field. The other engines come from the frozen table of the September 22, 2026 measurement. The new network comes from two sealed records, measured on September 30, 2026 and October 1, 2026, each tied to an exact commit; their seals were recomputed before publishing this page.

110 of the 3,479 attempts in this run were measured with the matrixAI tree dirty after a restart — they do not come from a clean commit, so this run is NOT fully anchorable to one exact point in history.

Measurement seals (JCS): C3 640137070dc30a28 · C4 a44a38c0e1a1fac4 · v2 600ebdaad2ca206e

These are seals of the content, not file hashes; see the explanation in the downloads below.

The protocols and the raw artefacts

This page does not redistribute the datasets — each row of the table links out to its own page on OpenML. What you can download here are the sealed records: first those of the 38/40 measurement of the new network, then those of the frozen v2 table — the base of the field — each with the digest that proves it has not been touched.

File SHA-256  8ec2e05bed0db6f8e38c99c6b5da4dc9f1097579bc55065a998b2813c3582cd3

File SHA-256  665c1c83b7b80ca8d49bcbd0f314a470902c8d76c9985e3527e3fae93d088d3b

File SHA-256  fcb154c67e2d7f0e06ec62d9ba843de4766b76d50fa1f622cb94032eace4d118

File SHA-256  7b300b179ffd210163f52764778b8a51ef1c39e607328490dffa871e2f31ee57

File SHA-256  efce65c3b74bfd762e7c911c1f3a996632789b1e50c4107cc49537889711a27b

File SHA-256  40029421c095292fd77490cca4ac8c664df797cd3b1424c931a3826806537912

Measurement seal  640137070dc30a282a1cfc4d7702fd5f06b4e24ca9e2d720950b0c0b8bd995ce

File SHA-256  c8fdf337184f236d6874e8f72711e92828352fac0643dbd219cabb9b6e60cca4

Measurement seal  a44a38c0e1a1fac450d958bfdec6d5a3153c84edb65d2d88cf6f73c31471e078

The previous measurement and the base of the field

These are the records of the earlier measurements — the previous network and the frozen v2 table the field is built on — not those of the 38/40, which are listed above.

File SHA-256  40ef7614d12bc5f727779d878ad430db9eb340b79d40894e4f174125d1f85b22

Registered protocol digest  1b4e902e61e06edcfda8e9f4ccef1232a233d46718089c5f0a2f814feb9dc878

File SHA-256  3ab8bd3589855e074f1bf21e0f82993f12c08553d8b1950396eaa8a565eecfb1

Measurement seal  600ebdaad2ca206eac148e25c38b57add8ad2d64652fafca3d8226758010c0d0

Nothing on this page redistributes a dataset — every dataset name links to its own OpenML page.

The earlier measurements, kept with their date

Until October 1, 2026 this page showed the September 16, 2026 measurement (v1). It is kept here, not erased. The September 22, 2026 measurement (v2) is the base of the field above: its counts are the first column of the two counts.

The v2 measurement was the first one with the recipe of the previous network (the one the Studio shipped until v2.7.3, and still ships in the GPU package as an experimental engine), and it decided a portfolio change on September 22, 2026. Part of it was measured with a dirty working tree (see the provenance table above), so it is not fully anchorable to one commit.

File SHA-256  cbc0f42b928df18d359cb0471b7b5741711abed429a54f4f073a3c8762e5091f

Registered protocol digest  ea50ca482a815627364b3439bfcf10e78295a13ae5f821219d0fb40329e65301

File SHA-256  dfb37c28e9aca72305efee319be46e5b968f30d981aeb781e8358881d30ae2d5

Measurement seal  3646af61b155c1321ad3fd71f25e9300ed9797b9b5dec0f37ed835cd835ecf48