Legal citation NER: The tokenizer bug that faked a regression
I trained RuBERT-family NER models on 300K auto-labelled spans to extend aqlex's rule-based citation extractor, and caught an evaluation bug that made a good model look bad.
Context
aqlex is our AI legal platform for Kyrgyzstan, where every answer must cite a real article. Its rule-based citation extractor finds references to legal acts in text, and the NER models were trained to extend it.
What happened
An AI-run training and evaluation of a new DistilRuBERT model concluded it was a regression. The drop looked too big to believe, so before accepting it I had the agent check input coverage instead of the scores.
The tokenizer’s overflow windows were silently dropping text: 25% of training tokens, including citations at the end of long chunks, never reached the model.
The fix
Fixing inference alone, with the same weights, raised gold F1 from 0.71 to 0.84. We rebuilt tokenization to build windows explicitly over the whole chunk.
Lesson
When a result surprises you, check the inputs before you trust the metric.