A round yellow face reads a document through a magnifying glass; typed chips (text, number, date, null) pop out of the page. text number date null

SystemOne Extract Index 0.1

A benchmark for one-pass extraction models. The model reads a document (state) and a map of typed fields, and returns a typed value or null for each field in one forward pass, with no generated tokens. The suite is frozen and public, the scorer is deterministic, and every listed run publishes its results.

edition 0.1

Leaderboard

Loading data/board.json

The Index is the mean of the 7 test scores, 0 to 100. Hover or tap a bar for its parts; click it to open the row in the table. System Two rows (generating models) are reference points and are not ranked. How it works

By area

Each area is the mean of its tests. The dashed tick is the null floor for that area.

Field types

Micro score over every field of that type in the suite. One dot per run; the dashed tick is the null floor.

Median latency vs Index

Wall time per request, one request at a time; hosted APIs include the network round trip. Up and left is the corner you want.

Full table

Area columns are means of their tests. Latency is the median wall time per request. Open a row for per-test and per-type scores, null calibration, hardware and links.

How it works

One request per document, one forward pass, one score per field. The engine under test does the first three steps; the kit does the rest, deterministically.

  1. 1 · Requestfrom the suite

    state (the document) and fields: a map of name to type, with optional instructions, unit and decimals.

  2. 2 · One passmodel under test

    One forward pass over state and fields. No decoding, no generated tokens, no thinking out loud.

  3. 3 · Typed answersmodel under test

    One typed value or null per field. Optional p_null, confidence and span.

  4. 4 · Scorerkit, deterministic

    Each field against gold with the rule for its type: a score in 0 to 1. Mean field score × 100 per test.

  5. 5 · Indexkit, deterministic

    Mean of the 7 test scores, 0 to 100. null until every test is present.

Request

The same field specs as Cloudflare's Clairic, so engines stay thin.

{
  "state": "Invoice INV-2291 from Northwind Supply, issued 2026-09-02, net 30. ...",
  "fields": {
    "invoice_number": {"type": "text"},
    "due_date": {"type": "date", "instructions": "Issue date plus the net payment days"},
    "total": {"type": "number", "decimals": 2, "unit": "USD", "instructions": "Amount due including shipping"},
    "contacts": {"type": "list", "items": "text"},
    "po_number": {"type": "text", "instructions": "Purchase order number"}
  },
  "reference_date": "2026-09-02"
}

Answer

Every requested field is present. p_null, confidence and span are optional and never affect the Index.

{"answers": {
  "invoice_number": {"type": "text", "value": "INV-2291", "span": [8, 16], "p_null": 0.0, "confidence": 1.0},
  "total": {"type": "number", "value": 1080, "confidence": 0.99},
  "contacts": {"type": "list", "value": ["Dana Ruiz", "Priya Shah"]},
  "po_number": {"type": "text", "value": null, "p_null": 1.0}
}}
Format details
  • Field names match ^[A-Za-z_][A-Za-z0-9_.\-]{0,63}$. Per-field settings: instructions, unit (durations: seconds to years), integer, decimals (0 to 6), items for lists (text, number or date). reference_date is optional.
  • Value forms: text a string; number and duration a number (durations in the field's unit); date YYYY-MM-DD; time HH:MM or HH:MM:SS; datetime YYYY-MM-DDTHH:MM[:SS] with no zone; list an array, [] or null when nothing matches.
  • value is null when the document does not determine the field. About half of the fields in some tests are null, so saying nothing at the right time counts as much as reading the right span.
  • An engine may declare limits (context length, field count) and answer unsupported instead of truncating. Those fields score 0.

Tests

Scoring

Per field, a score in 0 to 1. The rule depends on the field type.

Unanswered
Request unsupported or error, field missing, or wrong value form: 0.
Gold null
1 if the answer is null (or [] for a list), else 0.
Gold non-null, answer null
0.
text
SQuAD token F1 against the gold value and each listed alternative; the best one counts. Tokens are lowercased, punctuation and the articles a, an, the are removed, whitespace is collapsed.
number, duration
The answer is parsed (numeric strings allowed; , thousands separators, % and currency symbols stripped). 1 when |a − g| ≤ 0.5 × 10−d + 1e-9 for a field with decimals d; without decimals, when |a − g| ≤ 1e-6 × max(1, |g|). Else 0.
date, time, datetime
date: exact YYYY-MM-DD. time: exact to the minute; seconds are compared only when gold has them. datetime: exact to the minute. Common unambiguous forms (ISO with T or a space, a trailing Z or offset dropped) are normalised first.
list
Multiset F1 between answer items and gold items. Each item is matched with its item type's rule; text items match at token F1 ≥ 0.5 and count 1. Both empty: 1.
Per test
Mean field score over the test's fields (micro), × 100.
Index
Mean of the 7 test scores. null until every test is present.
Also reported
Per area (mean of its tests), per field type (micro over the suite), null precision and recall, Brier score over fields that report p_null, and latency (median, mean, p80 of request wall time). A run is complete only when every suite row has a result with status ok or unsupported.

Rules and submissions

Submissions are pull requests. Every listed run is public.

Rules

  • No truncation. One configuration for every test; no per-test prompt tuning.
  • Do not train on the suite's rows or on the docs generator's templates. The generator is public; Turion AI's own models are trained on a separate one.
  • A System One listing reads the input in a single pass per field group and generates no tokens. System Two (generating) models may be shown as reference rows; they are not ranked.
  • Declared limits are allowed. The engine answers unsupported for inputs it cannot take, and those fields score 0.
  • Open-weights entries are re-run by the maintainers on one NVIDIA RTX PRO 6000 to confirm scores and latency.
  • The maintainers (Turion AI) also submit their own models. All code, the suite and every run's results are public, so anyone can check them.

Submit a run

  1. Install the kit and fetch the frozen suite. 
    git clone https://github.com/balys/extract-index
cd extract-index && pip install -e ".[hub]"
python -m extract_index suite download
  2. Run the whole suite with your engine (http for the request format above, clairic, or your own module:Class) and score it. Requests go one at a time; a rerun resumes and retries errors. 
    python -m extract_index pipeline --engine http \
  --option base_url=https://your-host --compact --out runs/<name>
  3. Check runs/<name>/scores.json: complete must be true and index must not be null.
  4. Upload the run (results, environment, scores) to a Hugging Face dataset. 
    python -m extract_index upload --run runs/<name> \
  --repo <you>/<name> --public
  5. Open a pull request that adds a row to submissions/README.md (model, weights or API, results link, engine and commit, hardware, exact options, declared limits) and an entry to submissions/entries.json.
  6. The maintainers verify the results file, re-run open-weights models on the reference GPU, and regenerate this board.

Submission guide on GitHub