fix: Remediate 210 glitch tokens in tokenizer.json
#55
by gkielian - opened
Glitch Token Remediation — Tokenizer Patch
Summary
This PR patches tokenizer.json to remediate 210 glitch tokens identified
by embedding vector analysis. Glitch tokens are vocabulary entries whose embeddings
have collapsed to near-identical representations, causing unpredictable model behavior
when they appear in input.
No model weights are modified — this is a tokenizer-only fix.
Technique: BPE Merge Pruning
Merge pruning (210 tokens): Removes the BPE merge rules that
produce multi-character glitch tokens. Every vocabulary entry and token ID stays intact,
but the flagged entries become unreachable — text that would have matched is now tokenized as
well-trained constituent subwords instead.
239 merges were removed for 210 merge-pruned tokens because some glitch tokens have more than one BPE merge path, and every path must be removed.
Before / after examples: google/gemma-4-12B-it
Merge prune example: adiganik, token ID 194961
Prompt: Please repeat the following string exactly, with nothing else: "adiganik"
| Tokens for the string | Model output | |
|---|---|---|
| Before | ['adiganik'] → [194961] |
تحويلي ❌ |
| After | ['adigan', 'ik'] → [57195, 727] |
adiganik ✅ |
Full-coverage check (every flagged token)
Beyond the examples above, all 203 flagged tokens that the original tokenizer emits in
prompt context were exercised with the same repeat-after-me prompt, using the original and the
patched tokenizer on identical weights (greedy decoding).
| Group | Tokens exercised | Exact repeat (original) | Exact repeat (patched) | Fixed | Regressed |
|---|---|---|---|---|---|
| Merge-pruned | 203 | 0 / 203 | 154 / 203 | 154 | 0 |
- Regressions (worked before, broken after): 0; none of the remaining imperfect replies contain a flagged token ID.
- 5 flagged tokens are not exercised because the
original tokenizer never produces them from text in prompt context
(the patch is a no-op for those inputs). - Ordinary prompts: 8 / 8 identical input IDs and
8 / 8 identical replies with both tokenizers.
⚠️ Note for Agentic Systems
Many remediated glitch tokens are shared across Gemma model families (Gemma 1, 2, 3, 4).
In agentic pipelines, care should be taken that these token strings are not inadvertently
injected into prompts for unpatched models. We recommend applying this patch consistently
across all Gemma models used in a pipeline.
What is preserved
- ✅
len(tokenizer)— unchanged (262,144) - ✅ All token IDs — stable, no re-indexing
- ✅ Chat template — identical to original
- ✅
tokenizer_config.json— identical to original - ✅ All special tokens and
added_tokens— unchanged - ✅ Normal text tokenization — verified via benchmarks
- ✅ Zero model weight changes
Diff summary
| Original | Patched | Delta | |
|---|---|---|---|
| Vocab size | 262,144 | 262,144 | 0 |
| Merges | 514,906 | 514,667 | −239 |
Disclosed side effects
- Downstream compound tokens (3): 3 non-target vocabulary entries whose sole BPE merge path went through a pruned glitch token now decompose into constituent subwords instead of emitting a single token ID. QA inference evaluation (
What is "<token>"?, original vs. patched tokenizer on identical weights) confirmed 0 behavioral regressions across these compound tokens (and eliminated garbled/hallucinated replies on compounds built from collapsed subwords). - Inert consuming merges retained (5): 5 surviving merge rules reference a pruned glitch token as an input operand. Because that operand can never be produced, these merges are inert at runtime (retaining them keeps the diff minimal with identical behavior).
- Already-shadowed target merges (4): 4 flagged target tokens (
▁YYYY,▁yyyy,▁diffformul,▁::::::::) were already shadowed on their isolated surface strings by higher-priority BPE merges in the original tokenizer; removing their merge entries eliminates any residual boundary/substring path and is a no-op on normal text.
Click to expand all 3 downstream compound tokens
| ID | Token | Patched subword decomposition |
|---|---|---|
| 88273 | kitopssynth |
['kit', 'opss', 'ynth'] |
| 131116 | ▁StarSXmlClass |
['▁Star', 'SXml', 'Class'] |
| 207760 | sadurdupoetry |
['sad', 'urdu', 'poetry'] |
Removed merges (239)
Click to expand the full list of 239 removed merges
[".\"+\"|", "\"+\""] → ."+"|"+"
["AllFilesIn", "MkvDir"] → AllFilesInMkvDir
["CBC", "Pksc"] → CBCPksc
["CYCL", "ONEDB"] → CYCLONEDB
["Case", "Missense"] → CaseMissense
["Case", "PTV"] → CasePTV
["Collin", "ToDO"] → CollinToDO
["Consec", "HtIdx"] → ConsecHtIdx
["Control", "PTV"] → ControlPTV
["Custom", "Glare"] → CustomGlare
["CustomGlare", "Def"] → CustomGlareDef
["DT", "MakeRectInCM"] → DTMakeRectInCM
["Days", "GE"] → DaysGE
["Denovo", "Mis"] → DenovoMis
["ELEASE", "STR"] → ELEASESTR
["ENEMY", "PLACE"] → ENEMYPLACE
["Fcm", "Php"] → FcmPhp
["Foldout", "GC"] → FoldoutGC
["GTBase", "Alert"] → GTBaseAlert
["Get", "ParaPkg"] → GetParaPkg
["GoObject", "AllRef"] → GoObjectAllRef
["Go", "PrintError"] → GoPrintError
["Go", "RawContext"] → GoRawContext
["Go", "SrvGroupIndex"] → GoSrvGroupIndex
["HDR", "Offsets"] → HDROffsets
["HDROff", "sets"] → HDROffsets
["H", "IMQTTRVM"] → HIMQTTRVM
["IMQTTR", "VM"] → IMQTTRVM
["KAKKIA", "INEN"] → KAKKIAINEN
["Lua", "ToJavaResult"] → LuaToJavaResult
["MakeRect", "InCM"] → MakeRectInCM
["Malay", "Marks"] → MalayMarks
["Mkv", "Dir"] → MkvDir
["MyShop", "name"] → MyShopname
["Ordinate", "Tuple"] → OrdinateTuple
["PAF", "Struct"] → PAFStruct
["RefIn", "SrvGroup"] → RefInSrvGroup
["Ret", "JpegLen"] → RetJpegLen
["SAFER", "ELEASESTR"] → SAFERELEASESTR
["SRP", "BinBuf"] → SRPBinBuf
["SRPCore", "Shell"] → SRPCoreShell
["SRPInterface", "Item"] → SRPInterfaceItem
["SRP", "ParaPkg"] → SRPParaPkg
["SRPS", "XML"] → SRPSXML
["SRP", "SrvGroup"] → SRPSrvGroup
["SRPS", "rvGroup"] → SRPSrvGroup
["Same", "WordInterval"] → SameWordInterval
["Simulator", "Flood"] → SimulatorFlood
["SrvGroup", "Index"] → SrvGroupIndex
["StarBinBuf", "Body"] → StarBinBufBody
["StarObject", "Body"] → StarObjectBody
["StarParaPkg", "Body"] → StarParaPkgBody
["StarSXml", "Body"] → StarSXmlBody
["StarService", "Body"] → StarServiceBody
["StarSrvGroup", "Body"] → StarSrvGroupBody
["StructOf", "Object"] → StructOfObject
["TP", "ASDW"] → TPASDW
["Term", "ObjectDefer"] → TermObjectDefer
["Transc", "Hist"] → TranscHist
["U", "imcoords"] → Uimcoords
["VSLU", "ATYPE"] → VSLUATYPE
["WMN", "MDA"] → WMNMDA
["WeakTable", "Mutex"] → WeakTableMutex
["WithSize", "InCM"] → WithSizeInCM
["Ziel", "Panel"] → ZielPanel
["addKill", "Bonus"] → addKillBonus
["addKill", "Penalty"] → addKillPenalty
["adig", "anik"] → adiganik
["adigan", "ik"] → adiganik
["ak", "arantadhatu"] → akarantadhatu
["angolo", "Tocco"] → angoloTocco
["arant", "adhatu"] → arantadhatu
["arantad", "hatu"] → arantadhatu
["atthavid", "u"] → atthavidu
["attup", "adani"] → attupadani
["attu", "vasena"] → attuvasena
["blusas", "Fem"] → blusasFem
["cal", "counterx"] → calcounterx
["cal", "countery"] → calcountery
["calleft", "counter"] → calleftcounter
["callow", "counter"] → callowcounter
["calright", "counter"] → calrightcounter
["calup", "counter"] → calupcounter
["capture", "cpu"] → capturecpu
["ch", "ccgi"] → chccgi
["check", "katore"] → checkkatore
["co", "OrdinateTuple"] → coOrdinateTuple
["colour", "CodeDict"] → colourCodeDict
["df", "sonic"] → dfsonic
["dfs", "onic"] → dfsonic
["done", "ProcessAvg"] → doneProcessAvg
["draw", "counterx"] → drawcounterx
["draw", "countery"] → drawcountery
["drawingCode", "hint"] → drawingCodehint
["dw", "RetJpegLen"] → dwRetJpegLen
["eco", "expr"] → ecoexpr
["edLeft", "Shape"] → edLeftShape
["edRight", "Shape"] → edRightShape
["eltat", "double"] → eltatdouble
["faulse", "Ans"] → faulseAns
["foe", "Place"] → foePlace
["get", "HDRProcessor"] → getHDRProcessor
["get", "starcore"] → getstarcore
["getstarcore", "data"] → getstarcoredata
["gmzy", "h"] → gmzyh
["grafo", "Existe"] → grafoExiste
["him", "qttrvm"] → himqttrvm
["ineed", "follower"] → ineedfollower
["inertia", "Seq"] → inertiaSeq
["int", "Fragmentation"] → intFragmentation
["jols", "endev"] → jolsendev
["js", "bpmOb"] → jsbpmOb
["last", "DamageTook"] → lastDamageTook
["local", "instrList"] → localinstrList
["m", "BlitzID"] → mBlitzID
["mark", "UpdateChoice"] → markUpdateChoice
["match", "StudentNo"] → matchStudentNo
["ops", "synth"] → opssynth
["opss", "ynth"] → opssynth
["pJ", "PEGBuf"] → pJPEGBuf
["partial", "owner"] → partialowner
["pmm", "Imp"] → pmmImp
["pos", "Tocco"] → posTocco
["qttr", "vm"] → qttrvm
["respArray", "All"] → respArrayAll
["sad", "urdu"] → sadurdu
["sal", "expr"] → salexpr
["selectTable", "X"] → selectTableX
["selectTable", "Y"] → selectTableY
["sor", "finaly"] → sorfinaly
["squarePos", "Vecchio"] → squarePosVecchio
["start", "ZielPanel"] → startZielPanel
["tcp", "UniqueID"] → tcpUniqueID
["testGet", "Popup"] → testGetPopup
["time", "PlusEvents"] → timePlusEvents
["tochy", "odikwa"] → tochyodikwa
["total", "BlockFit"] → totalBlockFit
["waved", "eltatdouble"] → wavedeltatdouble
["wired", "Elems"] → wiredElems
["వెట్", "స్కీ"] → వెట్స్కీ
["▁", "::::::::"] → ▁::::::::
["▁::", "::::::"] → ▁::::::::
["▁Archers", "Unit"] → ▁ArchersUnit
["▁Atr", "ayam"] → ▁Atrayam
["▁Atra", "yam"] → ▁Atrayam
["▁C", "AdxRtList"] → ▁CAdxRtList
["▁CC", "BUNDLE"] → ▁CCBUNDLE
["▁Control", "Missense"] → ▁ControlMissense
["▁", "ControlPTV"] → ▁ControlPTV
["▁Control", "PTV"] → ▁ControlPTV
["▁DT", "MakeRect"] → ▁DTMakeRect
["▁", "FuncParamNum"] → ▁FuncParamNum
["▁Func", "ParamNum"] → ▁FuncParamNum
["▁GoObject", "To"] → ▁GoObjectTo
["▁", "GoRawContext"] → ▁GoRawContext
["▁Go", "RawContext"] → ▁GoRawContext
["▁Go", "SRP"] → ▁GoSRP
["▁", "GoSrvGroupIndex"] → ▁GoSrvGroupIndex
["▁Go", "SrvGroupIndex"] → ▁GoSrvGroupIndex
["▁", "HIMQTTRVM"] → ▁HIMQTTRVM
["▁H", "IMQTTRVM"] → ▁HIMQTTRVM
["▁HRM", "Client"] → ▁HRMClient
["▁LG", "AGEmoji"] → ▁LGAGEmoji
["▁Lua", "ToGoObject"] → ▁LuaToGoObject
["▁", "LuaToJavaResult"] → ▁LuaToJavaResult
["▁Lua", "ToJavaResult"] → ▁LuaToJavaResult
["▁ND", "IndexArray"] → ▁NDIndexArray
["▁", "NewParaPkg"] → ▁NewParaPkg
["▁New", "ParaPkg"] → ▁NewParaPkg
["▁", "PafHandle"] → ▁PafHandle
["▁Paf", "Handle"] → ▁PafHandle
["▁RUTARE", "AL"] → ▁RUTAREAL
["▁RUTARE", "L"] → ▁RUTAREL
["▁RUTAREL", "ATIV"] → ▁RUTARELATIV
["▁Ref", "ToGoObject"] → ▁RefToGoObject
["▁SIINFE", "KLC"] → ▁SIINFEKLC
["▁SIINFEKL", "C"] → ▁SIINFEKLC
["▁SRP", "Go"] → ▁SRPGo
["▁SRPGo", "Get"] → ▁SRPGoGet
["▁SRPGoGet", "Str"] → ▁SRPGoGetStr
["▁SRPGo", "SetStr"] → ▁SRPGoSetStr
["▁Spring", "ObjectID"] → ▁SpringObjectID
["▁SrvGroup", "Class"] → ▁SrvGroupClass
["▁", "StarSXml"] → ▁StarSXml
["▁Star", "SXml"] → ▁StarSXml
["▁", "TPASDW"] → ▁TPASDW
["▁TP", "ASDW"] → ▁TPASDW
["▁", "TermObjectDefer"] → ▁TermObjectDefer
["▁Term", "ObjectDefer"] → ▁TermObjectDefer
["▁TestAvg", "Callback"] → ▁TestAvgCallback
["▁V", "uttab"] → ▁Vuttab
["▁", "YYYY"] → ▁YYYY
["▁YY", "YY"] → ▁YYYY
["▁add", "ConfigureArg"] → ▁addConfigureArg
["▁add", "SBOM"] → ▁addSBOM
["▁atth", "udd"] → ▁atthudd
["▁bhuv", "adigane"] → ▁bhuvadigane
["▁check", "HDROffsets"] → ▁checkHDROffsets
["▁coi", "Alarm"] → ▁coiAlarm
["▁", "cytyle"] → ▁cytyle
["▁cy", "tyle"] → ▁cytyle
["▁cyt", "yle"] → ▁cytyle
["▁dSample", "Height"] → ▁dSampleHeight
["▁dSample", "Width"] → ▁dSampleWidth
["▁dhat", "ussa"] → ▁dhatussa
["▁dhatu", "ssa"] → ▁dhatussa
["▁diff", "formul"] → ▁diffformul
["▁ditt", "iyam"] → ▁dittiyam
["▁", "doneProcessAvg"] → ▁doneProcessAvg
["▁done", "ProcessAvg"] → ▁doneProcessAvg
["▁", "ecoexpr"] → ▁ecoexpr
["▁eco", "expr"] → ▁ecoexpr
["▁evam", "adisu"] → ▁evamadisu
["▁hetuk", "att"] → ▁hetukatt
["▁icc", "adini"] → ▁iccadini
["▁iccad", "ini"] → ▁iccadini
["▁jaû", "nes"] → ▁jaûnes
["▁jinfo", "Home"] → ▁jinfoHome
["▁jinfo", "Next"] → ▁jinfoNext
["▁m", "SwisTrackCore"] → ▁mSwisTrackCore
["▁", "matchStudentNo"] → ▁matchStudentNo
["▁match", "StudentNo"] → ▁matchStudentNo
["▁mdl", "MeshVD"] → ▁mdlMeshVD
["▁min", "Goto"] → ▁minGoto
["▁morphOutput", "Comp"] → ▁morphOutputComp
["▁o", "LetterLocation"] → ▁oLetterLocation
["▁sc", "StudentVector"] → ▁scStudentVector
["▁student", "LVector"] → ▁studentLVector
["▁subTest", "Avg"] → ▁subTestAvg
["▁subTest", "HDR"] → ▁subTestHDR
["▁subTest", "Panorama"] → ▁subTestPanorama
["▁suddhak", "att"] → ▁suddhakatt
["▁total", "BlockUsed"] → ▁totalBlockUsed
["▁totalBlockUsed", "A"] → ▁totalBlockUsedA
["▁", "yyyy"] → ▁yyyy
["▁yy", "yy"] → ▁yyyy
["▁চিদা", "ভ"] → ▁চিদাভ
["▁బ్లా", "వెట్స్కీ"] → ▁బ్లావెట్స్కీ
["加入", "参数向量中"] → 加入参数向量中
Files changed
tokenizer.json— BPE merge rules pruned (indent=2formatting preserved for clean line-by-line diffs)
How to verify
from transformers import AutoTokenizer
orig = AutoTokenizer.from_pretrained("google/gemma-4-12B-it")
patched = AutoTokenizer.from_pretrained("google/gemma-4-12B-it", revision="refs/pr/55")
# Vocab size and normal text tokenization are unchanged
assert len(orig) == len(patched) == 262144
text = "Hello world! 你好世界 🎉 def foo(): return 42"
assert orig(text)["input_ids"] == patched(text)["input_ids"]
# Glitch token `adiganik` (ID 194961) is no longer produced
assert 194961 in orig("adiganik")["input_ids"]
assert 194961 not in patched("adiganik")["input_ids"]
Algorithm Details
For full details on the glitch token collection algorithm and remediation techniques,
reach out to @gkielian .
Related
This is part of a series of tokenizer fixes across all Gemma model repositories
(Gemma 1, 2, 3, 4, and MedGemma — 18 models total).
dougreid changed pull request status to merged