encoder-eval-harness

A precision-fair harness for comparing encoder models by fine-tuning.

Fine-tune any set of encoder models on a 17-task suite (5 multilingual + 8 GLUE + 4 SuperGLUE) and report selection-free avg@5 ± std, so only the model weights vary between two cells of the table.

Models:

• XLM-R {base, large, XL},
• mDeBERTa-v3-base,
• mGTE-MLM-base,
• EuroBERT {210M, 610M, 2.1B},
• ModernBERT {base, large}, and
• the LFM2.5 encoder family (Encoder-{230M, 350M}, Embedding-350M, ColBERT-350M).

Tasks (17):

Bucket	Tasks
Multilingual (5)	XNLI · PAWS-X · Amazon Reviews-Multi · MASSIVE-Intent · SeaHorse
GLUE (8)	CoLA · SST-2 · MRPC · STS-B · QQP · MNLI · QNLI · RTE
SuperGLUE (4)	BoolQ · CB · WiC · WSC

For each multilingual task the train splits of all languages are concatenated and shuffled, fine-tuned once, then evaluated per-language and averaged over the Euro language set. All datasets are pulled from public Hugging Face repos.

---

Installation

Requires Python ≥3.11, a CUDA or ROCm GPU, and uv (or plain pip).

make install          # uv venv .venv + editable install
# or:  pip install -e .

The transformers version is pinned to 4.56.2 so dependency drift does not become an uncontrolled evaluation variable. See EVALUATION_METHODOLOGY.md §5.

ROCm (AMD GPUs)

By default, pip install -e . and make install install a CUDA build of PyTorch, which will not detect an AMD GPU. On ROCm, install the matching PyTorch wheel first, then install the harness on top. Pip sees that torch>=2.4 is already installed, so it leaves your ROCm build in place:

python -m venv .venv && source .venv/bin/activate      # Python >=3.11
# pick the wheel index for YOUR ROCm version from https://pytorch.org/get-started/locally/
pip install --index-url https://download.pytorch.org/whl/rocm6.4 torch
pip install -e .                                        # installs transformers==4.56.2 etc.; torch untouched

Confirm the GPU is visible before running a sweep:

python -c "import torch; print(torch.__version__, torch.version.hip, torch.cuda.is_available())"
# e.g.  2.9.1+rocm6.4  6.4.xxxxx  True

After that, make smoke-paws, the sweep, and summarize work exactly the same as on CUDA.

Usage

Run a single fine-tune

A cell is a single fine-tune of one model on one task at one learning rate. It writes its result to a JSON file.

python -m encoder_eval.run \
  --model-path FacebookAI/xlm-roberta-base \
  --task paws_x --mode auto \
  --lr 3.59e-05 --seed 45 --bsz 32 --max-steps 10000 --max-seq-len 256 \
  --early-stop --patience 3 --epoch-cap 20 \
  --out-json results/xlm-r_paws_lr06_s45.json

To check that your install works, run make smoke-paws. It runs a fast, 1000-step version of the command above.

Run a sweep

The protocol runs in two phases. First, a selection phase sweeps the learning-rate grid using dev scores only. Then a reporting phase runs fresh seeds at the selected learning rate. Each phase writes its own run_all.sh.

# 1. selection sweep — 10 LRs x 3 selection seeds (42-44), dev only
python -m encoder_eval.sweep --phase select \
    --out-dir runs/select --results-dir results/select --mode shell
bash runs/select/run_all.sh

# 2. pick the winning LR per (model, task) by seed-averaged dev mean
python -m encoder_eval.select --results-dir results/select --out selection.json

# 3. reporting sweep — 5 fresh seeds (45-49) at the selected LR
python -m encoder_eval.sweep --phase report --selection selection.json \
    --out-dir runs/report --results-dir results/report --mode shell
bash runs/report/run_all.sh

Both phases apply the shared recipe explicitly (--weight-decay 0.1 --adam-beta2 0.95, early stopping with patience 3, epoch cap 20). Both are idempotent: a cell whose result JSON already exists is skipped, so re-running just fills in the gaps. Result filenames include the seed ({model}_{task}_lr{key}_s{seed}.json), so seeds never overwrite each other.

Use --models and --tasks to control what runs. By default the sweep uses every model in the registry and the 5 multilingual tasks. To add GLUE or SuperGLUE tasks, name them, for example --tasks glue_sst2,glue_mnli,sg_boolq.

The full 14-model, 17-task protocol is about 7,100 selection jobs (10 learning rates × 3 seeds) plus about 1,200 reporting jobs (5 seeds). Every job is an independent, single-GPU process, so you can spread them across whatever scheduler you have.

Running on a cluster? Use --mode slurm --slurm-template <your-template>. This writes a launch.sh with one sbatch call per cell, based on a template you provide. The harness does not ship any scheduler config of its own. Your template only needs to activate your environment and run one cell from the exported variables MODEL, TASK, LR, OUT_JSON, and so on. For the exact contract, see the emit_slurm docstring in encoder_eval/sweep.py.

Build the table

python -m encoder_eval.summarize --results-dir results/report \
    --selection selection.json --format markdown        # or json / csv

summarize.py averages the reporting seeds for each cell and reports mean ± std. It does not pick a best run, because the learning rate was already fixed in step 2 using separate seeds. Each task uses its standard metric: MCC for CoLA, F1 for MRPC and QQP, macro-F1 for CB, Spearman for STS-B and SeaHorse, and accuracy otherwise. Scores come from each task's report split, which is the labeled test split for the multilingual tasks and dev (marked *) for GLUE and SuperGLUE. Nothing is typed by hand.

By default, summarize.py refuses to print an incomplete table. A cell is treated as an error rather than silently averaged if it is missing reporting seeds, sits at a learning rate other than the selected one, or was produced without the protocol flags. Pass --allow-partial to print anyway; the footnote then states the real seed count instead of claiming avg@5.

---

Methodology

Every design choice isolates the model as the only variable, controlling for seed, learning rate, and precision. Full protocol and reproduction pitfalls in EVALUATION_METHODOLOGY.md.

1. Uniform precision. Every model is loaded with fp32 master weights and bf16 autocast. Loading a model directly in bf16 is not allowed, because that would compare number formats instead of models.
2. One shared optimizer recipe. Every model uses the same AdamW settings ($\beta_2$=0.95, wd=0.1, 10% linear warmup, bsz 32), from the EuroBERT card, applied to all.
3. LR by seed-averaged dev mean. For each model and task, the harness tries 10 learning rates from 1e-5 to 1e-4 across 3 seeds and picks the one with the best average.
4. Scores from fresh held-out seeds. The reported score is the mean ± std over 5 seeds that never touched selection.
5. Bounded, early-stopped training. max_steps = min(10 000, N epochs), patience 3, load_best_model_at_end.
6. Report split. Multilingual tasks report the labeled test split. GLUE/SuperGLUE report dev (their test labels are hidden).
7. Atomic, idempotent I/O — results written via tmp + os.replace; existing cells skipped, unparseable files ignored.

---

Results

This source repository intentionally contains no unpublished organizational result artifacts or pre-release performance claims. Run the documented sweep and use encoder_eval.summarize to generate a comparison table from your own result files.

---

Extending

Adding a model

You can add a model by adding a row into MODEL_REGISTRY in encoder_eval/sweep.py:

"my-encoder-base": ("org/my-encoder-base", "auto"),          # standard HF *ForSequenceClassification
"my-encoder-body": ("org/my-encoder-body", "liquid-bidir"),  # encoder body + pooled head

• auto — standard AutoModelForSequenceClassification (+ trust_remote_code). Works for any HF encoder that exposes a sequence-classification head.
• liquid-bidir — load the encoder body via AutoModel and attach a fresh mean-pool + linear classifier head. Use this for bidirectional encoders that ship only a backbone (e.g. the LFM2.5 encoder family).

Any architecture-specific setup is handled for you in encoder_eval/loaders.py.

Adding a task

Add a loader to encoder_eval/tasks.py returning (train_dict, per_lang_val, per_lang_test, num_labels, problem), then register it in LOADERS and TASK_KNOBS. The problem value is either single_label_classification or regression.

---

Provenance and reproducibility

Every result JSON records what it took to produce it: the harness version and git SHA, transformers / datasets / torch versions, CUDA-or-HIP build, GPU name, the full recipe (weight decay, β₂, warmup, effective batch, precision), and — importantly — the resolved model revision (the model's exact commit hash on the Hub, not the floating tag you asked for). Two JSONs can therefore be checked for comparability instead of assumed comparable.

Known limits:

• Only transformers is version-pinned; the other dependencies are ranges and the repo ships no lockfile. For a citable run, freeze your environment (uv pip freeze > requirements.lock) and keep it alongside the results.

• Dataset revisions default to unpinned (the loaders track each public HF dataset's current version). Pin them for a citable run — whatever is in effect is recorded under provenance.dataset_revisions:

  export ENCODER_EVAL_DATASET_REVISIONS='{"nyu-mll/glue": "<sha>", "aps/super_glue": "<sha>"}'

  (or edit DATASET_REVISIONS in encoder_eval/tasks.py).

• Results are only comparable across runs produced on the same stack. Compare provenance between two result JSONs before putting their numbers in one table.

trust_remote_code

Both loader modes pass trust_remote_code=True, which executes model-authored Python from the Hub. That is required for several encoders in the registry (EuroBERT, mGTE, the LFM2.5 family) whose architectures are not in transformers. Consequences:

• Only evaluate models you are willing to run arbitrary code from.
• A floating revision means the code you execute can change between runs. Pin a specific commit when it matters — --model-revision <sha> (accepted by encoder_eval.run) — and check the recorded provenance.model_revision in the result JSON to see exactly what ran.

Limitations

• GLUE/SuperGLUE report dev (their test labels are hidden); the fresh-seed rule removes selection coupling, but dev is still the report split for those 12 tasks (see Methodology point 6).
• WSC is retained for completeness but is uninformative — no model beats the majority class under this recipe.
• Result JSONs and generated launch scripts are git-ignored (results*/, runs*/); they are run artifacts, not source.
• Exact model paths are omitted from result JSON by default because local paths and private repository names can be sensitive. Use --record-model-path only for appropriately restricted outputs.

Documentation

• AGENTS.md — driving the harness from a coding agent

Security

See SECURITY.md for credential-handling and private-reporting guidance. The repository includes an optional detect-secrets pre-commit hook.

License

Apache-2.0. See LICENSE.

Citation

@article{liquidAI2026Encoders,
  author = {Liquid AI},
  title = {LFM2.5-Encoders: Fast at Long Context, Even on CPU},
  journal = {Liquid AI Blog},
  year = {2026},
  note = {www.liquid.ai/blog/lfm2-5-encoders},
}