byt5-base-dv

Pretrained from scratch on Dhivei (language of the Maldives) with ByT5, Google's new byte-level tokenizer strategy.

Use byt5-dv for now; this is less accurate

Corpus: Sofwath's Dhivehi corpus https://github.com/Sofwath/DhivehiDatasets

Pretraining Notebook: https://colab.research.google.com/drive/1ERIZ1PyHn-yN_jo7dTQeODn22vrt-d1d?usp=sharing

Fine-tuning Demo

On Dhivehi news classification task

https://colab.research.google.com/drive/11u5SafR4bKICmArgDl6KQ9vqfYtDpyWp?usp=sharing

Issues

There was an issue with the vocabulary size, final layer, and/or accuracy on fine-tuning.