# Repetitions after pre-training T5X

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frepetitions-after-pre-training-t5x%2F92840%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 18, 2024, 8:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frepetitions-after-pre-training-t5x%2F92840 "2024-06-18T20:27:27Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![wildcard00](https://avatars.discourse-cdn.com/v4/letter/w/9dc877/32.png) [@wildcard00](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwildcard00)%5C
**Post date:** [June 18, 2024, 8:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frepetitions-after-pre-training-t5x%2F92840%2F1 "2024-06-18T20:27:27Z")

</div>

Hello, I am pre-training T5\_1\_1 using the t5x pretraining script to translate to Japanese on a large corpus of text. After training, I tried to translate a simple “Hello”, but it ends up repeating the “Hello” in Japanese several times in escape unicode sequences. The number of times it repeats is equivalent to the number of task feature lengths I have defined.

1. Is there a setting I can tweak to reduce the number of repetitions similar to CTranslate2?
2. In my preprocessor for the training task, I add the EOS tokens automatically as follows:

```auto
    preprocessors=[
        seqio.preprocessors.tokenize,
        seqio.preprocessors.append_eos_after_trim,
    ],

```

1. Any tips on how to reduce repetitions?