Emmanuel Hernandez
Repair scores for three models, with the T5 violation type where they break
All work

06 Aug 2026 → now

LLM Code-Switching Benchmark

Spanish and English bilinguals mix languages inside a single sentence constantly, and which mixtures sound right is governed by grammar rather than taste. Most evaluations prompt a model with some Spanglish and grade the output on feel, which says nothing about whether the model represents the rule being tested. This grades against the Matrix Language Frame model from linguistics, which makes specific predictions about which language controls the grammar of a mixed sentence, so a violation becomes a gradeable error instead of an awkward phrase.

Version 1.0 is 402 items, hash pinned so any edit fails the suite, balanced across three violation types and four task types. Three local open-weight models each ran the full set. One of them timed out on every item of the language identification task, so that row is unusable and the writeup says so rather than quietly dropping it. The suite is 177 tests at 96% statement coverage. It is the only thing I have built that uses both halves of my major.

Role
Solo
Stack
Python, pydantic, pytest, Ollama
Scale
402 items, 3 models, 177 tests at 96% coverage
Status
Local only, v1.0 tagged

Screens

Output from the committed v1.0 runs and a live test run.

  1. 01

    Repair scores

    Two of the three models repair the easy violation types well and collapse on T5, a bare English verb in a Spanish frame. The third breaks differently, which is why the finding is scoped to two.

    Repair scores

  2. 02

    The runs

    All three models completed 402 items. One lost a single task entirely to timeouts, and a second is degraded across all four. Errored records are counted, never silently dropped.

    The runs

  3. 03

    Tests

    177 tests at 96% statement coverage, with eight of the fourteen modules at 100%.

    Tests

What it does

Grammaticality judgment
Pick the licensed member of a minimal pair that differs only in one violation the theory predicts. The one task every model does well.
Repair
Fix a violating sentence with the smallest possible change. This is where the models actually separate from each other.
Matrix language identification
Name which language is running the grammar and cite the evidence for it. Every model scores below chance here.
Reproducible by construction
Every response is cached by a content hash of prompt, model and seed, so rescoring a changed rule costs zero model calls and every published number regenerates from committed data.

How it's built

Items
402 items, each carrying a source, one violation type from a mutually exclusive taxonomy, a difficulty, and the linguistic justification for its gold answer. The set has a recorded SHA-256 that a test pins, so any edit fails the suite.
Taxonomy
Three violation types, cut down from a planned four. Two candidates could not be populated with enough real items for a held-out split to mean anything, so they were dropped and the reasoning recorded rather than smoothed over.
Scoring
Each task is rescaled against its own floor and tasks are never averaged together. Every reported number carries a 95% bootstrap interval from 10,000 resamples.
Inference
Local open-weight models through Ollama. No API keys and nothing leaves the machine, which meant the harness had to handle GPU out-of-memory, hung models and contention per item rather than failing the run.