Does free text in Spanish help a tabular model?
4 public Spanish-language tasks, 6 ways to use the free text that comes with each row — from ignoring it to asking an external service — measured against one rule sealed before a single number came in.
Artefact format: 107-C30.v1
| Task | Metric | No text | Best way to use the text | Verdict vs. no text |
|---|---|---|---|---|
| A · Fake news | AUROC (higher is better) | 0.724 | Jev, asked directly (untrained): 0.885 | Improves |
| B · Clinical cases | AUROC (higher is better) | 0.5 | TF-IDF + linear: 0.819 | Improves |
| C · Public procurement | RMSE (lower is better) | 0.653 | Multilingual embedding (MiniLM): 0.635 | Practically equivalent |
| D · BOE — consolidated legislation | AUROC (higher is better) | 0.855 | Multilingual embedding (MiniLM): 0.875 | Practically equivalent |
- Text helps when it is the main signal: alone (TF-IDF), it improves on the no-text model in fake news (AUROC 0.795 vs. 0.724) and in clinical cases (AUROC 0.819 vs. 0.5).
- Asked directly and untrained, Jev is the best measured way to use the text in fake news (AUROC 0.885 against 0.724 with no text and 0.795 for TF-IDF) — with one caution: the corpus has been public for years, so this cannot separate Jev reading the text from remembering it from training.
- As a column generator, neither Jev nor the local model practically improves on the no-text model in any task.
- In public procurement (RMSE) and in the BOE, nothing practically crosses the equivalence margin.
The question and the six conditions
Plenty of MatrixAI Studio's models are trained on tables, and some of those tables carry a free-text column — a description, a note, a report — that today gets ignored. This measures whether that text is worth reading at all, and if so, which of six ways to read it earns its keep, on four public tasks written in Spanish.
- No text (reference) — The tabular model alone, with the free-text column dropped — the reference every other condition is measured against.
- Multilingual embedding (MiniLM) — The text turned into a single multilingual sentence embedding (MiniLM), fed to the tabular model as extra columns.
- TF-IDF + linear — The text alone: a TF-IDF bag of words (single words and pairs) with its own linear model — logistic regression, or ridge for regression — without the task's other columns and without the tabular model.
- Local responder (Qwen2.5-0.5B) — A small model running on this machine (Qwen2.5-0.5B) asked 5 fixed yes/no questions about the text; the answers go into the same tabular model as 5 extra columns.
- Jev (making columns) — The same 5 yes/no questions, answered by Jev — an external service — instead of the local model.
- Jev, asked directly (untrained) — No columns at all: Jev is asked the task's own question directly, having trained on none of our data, and its raw answer (0–1) is used as the score itself.
Every condition is compared to condition (0) — no text — in a paired comparison, row by row: the same rows, with and without the text signal. The interval comes from a row-level bootstrap with 1,000 resamples, and a condition only counts as practically equivalent to «no text» when its whole interval fits inside a fixed equivalence margin of 0.05 — set before any of these numbers existed, not chosen after seeing them.
Four public tasks, all in Spanish
Each task is a real, publicly available Spanish-language dataset — nothing built for this measurement.
| Source | What it predicts | Rows | |
|---|---|---|---|
| A · Fake news | FakeNewsCorpusSpanish · CC BY 4.0 | Telling a fake Spanish-language news article from a genuine one. | Train 676 · Dev 295 · Test 572 · 1,543 rows in total |
| B · Clinical cases | CodiEsp · CC BY 4.0 | Predicting, from a Spanish clinical case report, whether ICD-10 code R52 (pain) applies. It has no other columns: «no text» has nothing to learn from. | Train 500 · Dev 250 · Test 250 · 1,000 rows in total |
| C · Public procurement | PLACSP · August 2025 tenders file | Regressing the base-10 logarithm of each tender's amount excluding VAT, from the contract's description and its other columns. | Train 21,323 · Dev 4,569 · Test 4,570 · 30,462 rows in total |
| D · BOE — consolidated legislation | Spain's official gazette (BOE) · consolidated legislation · reusable (Law 37/2007) | Predicting whether a piece of Spanish legislation is still in force. | Train 3,500 · Dev 750 · Test 750 · 5,000 rows in total |
Every condition, per task
On a phone each table keeps its own horizontal scroll.
A — Fake news · FakeNewsCorpusSpanish
AUROC (higher is better)
| Condition | Score | Difference vs. (0), with interval | Verdict |
|---|---|---|---|
| No text (reference) | 0.724 | reference | |
| Multilingual embedding (MiniLM) | 0.645 | -0.079 [-0.123, -0.033] | Underperforms |
| TF-IDF + linear | 0.795 | 0.071 [0.021, 0.115] | Improves |
| Local responder (Qwen2.5-0.5B) | 0.728 | 0.004 [-0.008, 0.018] | Practically equivalent |
| Jev (making columns) | 0.715 | -0.009 [-0.029, 0.012] | Practically equivalent |
| Jev, asked directly (untrained) | 0.885 | 0.161 [0.122, 0.200] | Improves |
Other differences described in the artefact, without a verdict (not in the pre-registration):
- Local responder (Qwen2.5-0.5B) − TF-IDF + linear: -0.067 — Described, no verdict — not part of the sealed comparison plan.
- Jev (making columns) − TF-IDF + linear: -0.08 — Described, no verdict — not part of the sealed comparison plan.
- Jev (making columns) − Local responder (Qwen2.5-0.5B): -0.013 — Described, no verdict — not part of the sealed comparison plan.
- Jev, asked directly (untrained) − Multilingual embedding (MiniLM): 0.24 — Described, no verdict — not part of the sealed comparison plan.
- Jev, asked directly (untrained) − TF-IDF + linear: 0.09 — Described, no verdict — not part of the sealed comparison plan.
- Jev, asked directly (untrained) − Jev (making columns): 0.17 — Described, no verdict — not part of the sealed comparison plan.
B — Clinical cases · CodiEsp
AUROC (higher is better)
| Condition | Score | Difference vs. (0), with interval | Verdict |
|---|---|---|---|
| No text (reference) | 0.5 | reference | |
| Multilingual embedding (MiniLM) | 0.711 | 0.211 [0.134, 0.287] | Improves |
| TF-IDF + linear | 0.819 | 0.319 [0.261, 0.374] | Improves |
| Local responder (Qwen2.5-0.5B) | 0.5 | 0 [0.000, 0.000] | Practically equivalent |
| Jev (making columns) | — | Never sent to Jev — clinical data | |
| Jev, asked directly (untrained) | — | Never sent to Jev — clinical data | |
Other differences described in the artefact, without a verdict (not in the pre-registration):
- Local responder (Qwen2.5-0.5B) − TF-IDF + linear: -0.319 — Described, no verdict — not part of the sealed comparison plan.
C — Public procurement · PLACSP (agosto 2025)
RMSE (lower is better)
| Condition | Score | Difference vs. (0), with interval | Verdict |
|---|---|---|---|
| No text (reference) | 0.653 | reference | |
| Multilingual embedding (MiniLM) | 0.635 | -0.019 [-0.024, -0.013] | Practically equivalent |
| TF-IDF + linear | 0.669 | 0.016 [-0.001, 0.032] | Practically equivalent |
| Local responder (Qwen2.5-0.5B) | — | Not measured — cost | |
| Jev (making columns) | 0.648 | -0.005 [-0.008, -0.002] | Practically equivalent |
| Jev, asked directly (untrained) | — | Regression — Jev does not give a continuous number | |
Other differences described in the artefact, without a verdict (not in the pre-registration):
- Jev (making columns) − TF-IDF + linear: -0.021 — Described, no verdict — not part of the sealed comparison plan.
D — BOE — consolidated legislation · BOE - Legislación consolidada (muestra 5000)
AUROC (higher is better)
| Condition | Score | Difference vs. (0), with interval | Verdict |
|---|---|---|---|
| No text (reference) | 0.855 | reference | |
| Multilingual embedding (MiniLM) | 0.875 | 0.02 [-0.004, 0.045] | Practically equivalent |
| TF-IDF + linear | 0.836 | -0.019 [-0.052, 0.013] | Inconclusive |
| Local responder (Qwen2.5-0.5B) | — | Not measured — cost | |
| Jev (making columns) | — | Outside the pre-registration | |
| Jev, asked directly (untrained) | 0.828 | -0.028 [-0.065, 0.011] | Inconclusive |
Other differences described in the artefact, without a verdict (not in the pre-registration):
- Jev, asked directly (untrained) − Multilingual embedding (MiniLM): -0.048 — Described, no verdict — not part of the sealed comparison plan.
- Jev, asked directly (untrained) − TF-IDF + linear: -0.008 — Described, no verdict — not part of the sealed comparison plan.
What Jev answered
Jev is an external service from TypeSafe, reached through OpenRouter, that answers typed yes/no questions about a piece of text. The model behind it is fixed and read from the artefact: typesafe/jev-1.13-20260917.
As a column generator (condition 4), Jev only runs on tasks A and C — both public datasets; asked directly, without training (condition 5, below), it runs on A and D instead — C is a regression and Jev's answer is not a continuous number. Task B is clinical case reports, and clinical text never leaves this machine, so it is never sent to Jev under either use; that is why the tables above show B as omitted, with its reason.
Why Jev is not in the 40-dataset table of «How we measure»: those datasets are tables with no free text, Jev does not learn from labelled rows and does not do regression. Its place is here — turning text into columns for the model that does learn. How we measure: the 40 datasets →
Total spent on the Jev API key for this measurement: $0.5963, covering tasks A and C, plus the test calls.
Task A — Fake news
Jev and the local responder agreed on 76.1% of the comparable answers (5,872 of 7,715).
Total wall-clock time: 485 s (8.1 min)
| Pass | Outcome | Duration |
|---|---|---|
107c30-jev-a | completed | 485 s (8.1 min) |
Task C — Public procurement
Total wall-clock time: 15,208 s (4.2 h)
The measurement had to resume from its cache of already-answered rows, so it took several runs: the total above is the sum of all of them, listed below — not just the last one.
| Pass | Outcome | Duration |
|---|---|---|
107c30-jev-c | error (network cut) | 212 s (3.5 min) |
107c30-jev-c2 | error (no credit left on the account) | 3,053 s (50.9 min) |
107c30-jev-c3 | hit the time limit | 10,801 s (3 h) |
resto-a-mano | completed | 1,142 s (19 min) |
Condition (5): asking Jev directly, without training
A different way to use Jev: instead of manufacturing columns for our own model, this asks Jev the task's own question exactly as written, having trained on none of our data — its raw answer (0 to 1) is used directly as the score, on the same test rows as every other condition.
A — Fake news
¿Es falsa esta noticia? Responde solo sí o no.Missing answers: 0 (0%)
D — BOE — consolidated legislation
¿Esta norma ha dejado de estar vigente? Responde solo sí o no.Missing answers: 0 (0%)
Both corpora have been public for years, so this measurement cannot separate Jev reading the text from Jev remembering it from training. And its score is AUROC — it ranks — which says nothing about whether that 0–1 number is calibrated.
The «NUMBER» placeholder
FakeNewsCorpusSpanish replaces digits with the literal placeholder NUMBER in some of its versions. Counted directly in the copy of the dataset this measurement used: 749 of 1,543 rows contain it — and how often depends on which version and which partition, not on whether the article is fake or genuine.
Detection pattern: \bNUMBER\b (case-sensitive)
| Version | Partition | Rows with NUMBER | Rows | Share |
|---|---|---|---|---|
| v1 | dev | 234 | 295 | 79.3% |
| v1 | train | 515 | 676 | 76.2% |
| v2 | test | 0 | 572 | 0% |
| Label | Rows with NUMBER | Rows | Share |
|---|---|---|---|
Fake | 221 | 338 | 65.4% |
True | 294 | 338 | 87% |
| Term (χ²) | χ² |
|---|---|
number de | 1.55 |
mil number | 1.329 |
el number | 1.27 |
mil | 1.254 |
de number | 1.152 |
The version used to pick task A's yes/no questions is soaked in this placeholder; the version actually scored is not. A responder's answer to a question built around NUMBER says something about which corpus version a row came from, not about whether it can read Spanish news — so task A's numbers describe that mismatch, and cannot be used to grade Jev or the local responder as readers of this text.
Placed next to the tables on purpose, with the same weight — not a footnote at the bottom of the page.
- Four tasks and one equivalence margin: a result here does not generalize beyond them.
- Only Spanish was measured — nothing here says anything about any other language.
- The yes/no questions were picked by an automatic rule (χ²), not written by a person: with different questions, the same responders could score differently.
- Jev's response time is not comparable to the local responder's — they are different things (a remote HTTP service vs. a model running on this machine), and this page never puts the two side by side.
What produced this JSON, with its fingerprints
Every artefact this page reads from, with its SHA-256 so that anyone holding the file can check it is the same one, and the hash of the script that wrote it where the artefact records one.
18c922c9df6c6bee13ac2953b394296fd8ac67306a71ec9873bf8ab84e694b49
| File | Generated | Script SHA-256 | File SHA-256 |
|---|---|---|---|
marcador_number_A.json | September 29, 2026 | 41858b30bc492d4b32d0fc34429c70e59b4e43a36d67f7514d80ff1fcde6182c | 072ed3dfc2f498ff1201ab1c3fb32dd18807ec12649569133b10ec98ff81a95e |
pasadas_c30_jev.json | — | 97baf79a97dc17240267a49e6cd38ceca323b91f8cb9d15976acfcf6b07aeebe | 2d5b0963b43b1f1e9da95f60646521ff69c44aad2dc328958205d38c55c04190 |
respuestas_c30_jev_A.json | — | — | 5edfd489f48a25aec7499a0739f31b5c9097c3bd91b9891d9f109e6de76aece8 |
respuestas_c30_respondedor_a.json | — | — | c3873430d4382bf75018705dd11eaeff19d97f6b684da59deff34e0f9eeee12c |
resultado_c30.json | September 25, 2026 | d2bf7c07a10de2e1b67e09178a0fc92b9be7c211248764b45bdb96b8862e9581 | 0a85571c8db563adb6d07cb3725a60129a399dffa0ec2889f08e10faf3b7e0f9 |
resultado_c30_jev_A.json | September 29, 2026 | 0abc5e112747929b5256a41b0e7447534ce470eb61b251b142d2c0ca64067777 | 8a9a42171ed83bc0711a68597c5c06097b675f74e55ca2ef69aeff621aa8029d |
resultado_c30_jev_C.json | September 29, 2026 | 0abc5e112747929b5256a41b0e7447534ce470eb61b251b142d2c0ca64067777 | 5051fa8649944e379c10da1e98efa219a89241cf0841eb13e5d4450eda79eed3 |
resultado_c30_jev_directo.json | September 29, 2026 | f55f492ad008faac9c6a341c8ab1a4546c1f27d1c60471f1ad259ab33b5206bd | 5140f7b79926505a29ff5b5ba6bd801a6b887543584a6fb4b6f5bc4984c3ba8a |
resultado_c30_respondedor_a.json | September 27, 2026 | 77d23914451e3946fe219533915cc00d1f3e1d2da6e68b388b082e66a683f95a | 403074a1876c8437ffaf22e785b035eedbddb5485f1d9c0377204885d18240c7 |
resultado_c30_respondedor_b.json | September 27, 2026 | 77d23914451e3946fe219533915cc00d1f3e1d2da6e68b388b082e66a683f95a | f32115499363e1785242914a125bce6587a14456c82336d8f1a374c76a176ec3 |
resumen_tareas.json | — | — | 158f31b29a89c9ccc7d78850ac8367a81ed272e35f507829e90d690e58fc4661 |
Every number on this page is read live from src/datos/texto_107_publico.json, copied byte-for-byte from the artefact MatrixAI generates from the sealed 107-C30 run — nothing here is typed by hand.