Theia

Sealed 2 November 2026 (internal benchmark record, 425 sealed files, 0 changed; 2,975 primary EuropePMC lookups with 0 banned-domain hits).

01B · Uber-Hard Subspecialty Radiology (ICML 2025 MedXpertQA-MM, 60 Cases, 5 Options A–E)

Sealed 2 November 2026 (internal benchmark record, sha256 13bd2320cea0748b..., 360 sealed files, 0 changed). Both 100% blind arms (76.7%–78.3%, union 81.7%) and Subspecialty Landmark WIL V2 (95.0%, p = 0.0063 vs Stock Argon) are reported in full.

01C · Head to head with Harrison.Rad 1.5 on Plain-Film X-Ray

Exact two-sided McNemar test, paired by question. Harrison did not answer 9 of the 45 (note 6); the last column compares only the 36 it answered.

1. 01

   Rules first

   Questions, scoring and void rules are written and saved before the first model call. Every amendment is dated and kept.

2. 02

   Instructions on file

   The exact instructions each system received are read back from its stored system prompt and kept with the results.

3. 03

   Research first

   Theia researches before it answers. The full protocol requires 35 logged web look-ups per question. The archives that host the test cases are off limits, and every search result and page fetch is logged.

4. 04

   Sealed answers

   Answers are saved and hashed before the answer key is opened. The model never sees the key.

5. 05

   Void for all

   If the examiner breaks a question, it is removed for every system, never only for the one that missed it. One question was removed this way, leaving 45.

6. 06

   First answer stands

   Reruns made "just in case" are logged and never replace the scored answer.

7. 07

   Re-scored from raw files

   Every grade is re-derived by a separate script from the raw run files, with file hashes on record.

We build and install AI into working businesses: clinics, practices, venues, property and professional services. Every system is tested the way Theia is tested before it touches your customers.

1. 01

   Front desk and bookings

   Voice reception and outbound booking agents on your own phone line. In a recorded test call, our outbound agent booked a consultation in 79 seconds.

2. 02

   Research and reporting

   Research agents, structured report drafts and evidence retrieval, with a human signing off every output that matters.

3. 03

   Workflow and memory

   Email, calendar, CRM and document workflows joined up, with a persistent memory so your systems stop forgetting what your team already told them.

1. Questions: 85 sealed vision-required imaging questions (ITEMS-ALL-85.json, sha256 3b677f49084e7d75...): 45 plain-film X-ray questions from the public RadBench closed set (Harrison.ai, q340 voided for all systems after an examiner error), plus 10 CT, 10 structural MRI, and 10 clinical Ultrasound disease/anatomy diagnosis cases from held-out OmniMedVQA/RadImageNet test splits, and 10 4D BOLD fMRI neuroimaging cases rendered from OpenNeuro datasets ds006623 and ds005684.
2. Execution & sealing (2 November 2026): Every Gemini 4 Argon Theia arm ran under the no-Crossroads WARNINGINFINITYLOOP.md contract (sha256 936d7abb580f7f81...) with 35 banned-domain-filtered primary EuropePMC literature lookups per item (2,975 queries total, 0 hits from Radiopaedia, MedPix, Harrison.ai, Hugging Face, or OpenNeuro). The answer key (sha256 591d0331d12bb391...) was vaulted outside the workspace until internal benchmark record locked all 425 raw answer and research files.
3. Ablation of Language-Prior Overwrite (V1 vs V2): In unanchored V1 (32/45 on X-Ray, 40/40 = 100% on CT/MRI/Ultrasound/fMRI), injecting 35 generic literature snippets without a binary normality gate caused text base-rates (e.g., C2 odontoid prevalence and emergency false-negative avoidance) to flip 5 X-ray items that zero-shot Stock Gemini 4 Argon (36/45 = 80.0%, p = 0.0074 vs Harrison.Rad 1.5) answered correctly. Adding Visual-Anchored subspecialty gating (V2, documented in PREREG-ARGON-MM85-V2.md) eliminated those flips and reached 44/45 (97.8%, p = 0.0215 vs Stock Argon; p = 4 × 10⁻⁶ vs Harrison.Rad 1.5). Both V1 and V2 are reported in full.
4. Radiologist percentile: Shelmerdine et al., BMJ 2022, 26 newly qualified radiologists, mean 84.8%, range 76.1% to 91.9%, assuming a normal spread. Different questions and format, so this is an estimate, not a head-to-head.
5. All test images come from published clinical/research repositories. Only unseen clinical films remove training-overlap doubt.
6. Harrison.Rad 1.5 was run through its public demo on 17 September on the 45 plain-film items. It did not answer 9 of the 45: its input check rejected one pacemaker film (3 questions), it disputed the premise of 4, and 2 fell outside its stated plain-film scope. These count as wrong in the 45-question figures and are excluded in the 36-question column.
7. Earlier September UMBRA-DS DeepSeek/Opus runs were voided by operator audit on 25 September 2026 and are excluded from this page.