Check with your own cases whether a scale or risk model you already use, or are being sold, works in your population. It trains nothing: it measures what is already recorded next to the real outcome.
What it is for
A hospital, a bank or an insurer often uses a score built on another population: a published scale, a vendor’s model or one from its own service. Almost nobody checks whether it works on their cases.
This check takes the score that is already recorded next to the real outcome and answers five questions:
Does it rank cases well? Whether the cases with the event score higher than those without (AUROC, with its interval).
Is its risk right? Only if the score is a probability: whether the risk it states is the one that then happens (calibration).
At today’s threshold Sensitivity and specificity at the cut-off it is used with, and how many alerts it raises.
Who does it fail? The same analysis by subgroup (age, sex, service…), with no number when a group has fewer than 30 cases, or fewer than 10 events or 10 non-events.
Against another score Whether one ranks better than the other, on the same rows and with its interval. It is the most directly interpretable comparison.
The simulation
Try the example case or your own CSV. On this page your CSV is read in this browser and is not sent anywhere; the demo works differently (see “With your data”). This page calculates the figures with JavaScript, not the MatrixAI core: they show how a report reads, they are not a result.
MatrixAI Studio · Check a score you already useSimulation · made-up data
1 · Your data
On this page it is read in this browser and is not sent anywhere. Up to 30 MB.
The example case is 2,400 made-up admissions with two scores: a vendor’s risk (a probability that overestimates the risk and fails more in patients over 80) and a 0 to 10 points scale. The Excel version brings semicolons, decimal commas, “NA” and the outcome written three ways.
Loading the example…
The CSV it reads
Made for the CSV a hospital really has, which usually comes out of Excel or a database query:
Semicolon separator (Excel in Spanish), comma, tab or vertical bar.
Decimal comma (0,3). It is detected per column and shown to you.
Windows-1252, which is what Excel on Windows writes as “CSV (comma delimited)”. If the file is not valid UTF-8 it is read that way, and the page says so.
Empties written as NA, N/A, NULL, NaN, “-” or “?”, or left blank: they do not count, and you are told how many.
The outcome as people write it: Yes, YES, yes, 1, true. You say which value is the event; it is never guessed.
Probability in % (23.5): you declare it as “probability in %”; it is not deduced.
Several admissions per patient: with an identifier column, the intervals are calculated by patient.
An unnamed first column of row numbers (what R’s write.csv writes) is detected and dropped.
Excel .xlsx files: no. Export them to CSV first.
The limit on this page is 30 MB.
What this report is not
It is not a validation for clinical use or a regulatory assessment. It is a retrospective evaluation with the data you provide.
If the score was already used to decide, it will look worse than it is (the treatment paradox): a high risk may have led to treating, and the treatment avoided the outcome.
Every figure depends on how the outcome was recorded: one recorded wrongly or late moves them all.
You say what is predicted and by when: the outcome horizon (during the stay, at 30 days, at 6 months) and when the score was calculated are declared, never deduced. If these cases took part in developing the score, or you do not know, the report says it is not an external validation.
A sample that does not represent the population where the score is used does not say what its alerts are worth there: that is why PPV and NPV only appear if you say it does represent it. Calibration and net benefit then describe the sample provided, not the population of use (a sample with more deaths than usual distorts them). If you say it does represent it, that is declared by whoever runs the check, and MatrixAI does not verify it.
A published figure is a reference, not a verdict: it is set against a local estimate, and the population, the outcome, the follow-up and the uncertainty of the original figure may not be comparable. Falling outside the local interval does not show that the original figure is wrong. The comparison of two scores on the same cases is the most directly interpretable.
With fewer than 30 cases in a group, or fewer than 10 events or 10 non-events, no number is published. Hiding small groups helps, but it does not guarantee anonymisation.
Yes/no outcomes only. Multiclass, regression and survival (time to event) are left out.
How this simulation is calculated
JavaScript in this page, not the MatrixAI core. The Studio’s figures come from the core.
Intervals by resampling: 200 resamples and a fixed seed (137), stratified by outcome or, if you give a patient identifier, by whole patient.
AUROC counts ties as half a hit. Calibration is the intercept and slope of a logistic regression on the logit of the stated risk.
A group is marked “worse” when its AUROC is more than 0.08 below the total.
With your data
In the demo
It accepts your CSV, with a limit of 10,000 rows or 5 MB. Your CSV leaves your computer for the demo server, is processed in memory and is discarded. Do not upload patient data or personal data to it: the demo is for public or made-up data.
Everything is local: the check runs on your machine and the CSV does not leave it. No row limit beyond memory, and no connection needed. It gives a one-page report in Spanish and English and a receipt signed with your installation’s key.