ymoslem/ModernBERT-base-TeleMath-router-qe-classifier-binary-10ep-lr2e-05-k1-e2b-5runs

Quality estimator for the Stage 2 cascade of CRE-Router on TeleMath, for Gemma4-E2B (without reasoning). No-clustering ablation (k = 1) under the E2EL budget of 25 s: gates every question Stage 1 sends to Gemma4-E2B; rejected answers escalate to Gemma4-26B.

Given an efficient model's answer, it predicts whether to accept it (label 1) or escalate the query to a stronger model (label 0). An answer is escalated when the probability of acceptance is below 0.5.

Input format

question [SEP] full_output [SEP] num_tokens

full_output is truncated to its last 1,000 words before tokenisation, and the tokeniser runs at max_length=4096.

Training

ModernBERT-base, up to 10 epochs, lr 2e-5, batch size 64, max_length 4096, class-weighted cross-entropy. The training set is the model's own answers on the 299 TeleMath training questions, five sampled runs each (1,495 rows: 634 accept, 861 escalate), labelled by task correctness. Training stops early, with patience 4, and keeps the checkpoint with the best macro-F1 on one run of the 201 TeleMath test questions (70 accept, 131 escalate); the kept checkpoint is epoch 10 of 10, macro-F1 0.770.

Intended use

Deployed only where Stage 1 routes to Gemma4-E2B. It is trained on that model's own outputs and does not transfer to another model's outputs.