How we measure which engine wins

What "the best engine" means here

On each dataset, every engine is trained with its default settings — no hyperparameter search — and scored with the metric that task uses. The engine with the best average is the winner of that dataset; any other engine that lands within a fixed number of points of that winner still counts as having met the bar there. A crash or a timeout counts as a lost dataset for that engine, not as a zero.

the engine with the best mean over the fits on THAT dataset, excluding the dummy baseline; a failure (timeout/crash) counts as a lost dataset for that engine

kc2 — Binary classification

Worked from the same JSON this whole page reads — nothing here is typed by hand. · 522×22 · OpenML ↗

On kc2, the best engine was scikit-learn HGB, at 0.913. Of the 6 engines that competed, 3 stayed within 2 points of the best and so meet the rule on this dataset.

40 datasets × 7 engines

Filter by task or by dataset size. On a phone the table keeps its own horizontal scroll — the page itself never scrolls sideways.

• ✓ meets the bar (within 2 points of the best on that dataset)
• ★ best engine on that dataset
• — crashed or timed out (counted as a lost dataset, not a zero)

Datasets marked SEALED are the portfolio’s final exam: they are never used to pick or tune anything, only to grade the result once, at the end.

Baseline is a dummy predictor shown for reference only — it never competes for "the best" and is excluded from every verdict below.

Is this run anchorable to one exact commit?

All 3479 attempts in this run were measured with the matrixAI and matrixai-engines repositories CLEAN — each one comes from an exact commit, so this run is fully anchorable to one specific point in history.

The protocol and the raw artefact

This page does not redistribute the datasets — each row of the table links out to its own page on OpenML. What you can download here is the sealed protocol and the raw measurement artefact this whole page reads, each with the digest that proves it has not been touched.

SHA-256  ea50ca482a815627364b3439bfcf10e78295a13ae5f821219d0fb40329e65301

SHA-256  3646af61b155c1321ad3fd71f25e9300ed9797b9b5dec0f37ed835cd835ecf48

Nothing on this page redistributes a dataset — every dataset name links to its own OpenML page.

The most recent measurement, kept apart

Everything above — the table, the verdict, the downloads — comes from the run that backs the current engine portfolio, because it is fully anchorable to one exact commit. This run is newer and was the one that decided a portfolio change on 2026-09-22, but part of it was measured with a dirty working tree, so it is shown here on its own rather than mixed into the numbers above.

Measured September 22, 2026 · 3,479 attempts · 5.82 h of wall clock

SHA-256  1b4e902e61e06edcfda8e9f4ccef1232a233d46718089c5f0a2f814feb9dc878

SHA-256  600ebdaad2ca206eac148e25c38b57add8ad2d64652fafca3d8226758010c0d0