tokenizers
tokenizer
Ansh-256k / config.json
shaantastic24's picture
Create config.json
4d04c01 verified
raw
history blame contribute delete
271 Bytes
Invalid JSON: No number after minus sign in JSONat line 1, column 2
---
license: apache-2.0
library_name: tokenizers
tags:
- tokenizer
language:
- hi
- as
- mr
- gu
- pa
- en
- or
- te
- ta
- ml
- kn
- bn
- sd
- ur
- ne
- ks
- sa
- gom
- mai
- mni
- brx
- doi
- sat
Vocab_size:
2,56,000
---