# T5 user defined loss function

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-user-defined-loss-function%2F566%3E%5C
**Category:** Beginners\
**Created:** [August 3, 2020, 12:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-user-defined-loss-function%2F566 "2020-08-03T00:20:23Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![mengyahu](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@mengyahu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmengyahu)%5C
**Post date:** [August 3, 2020, 3:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-user-defined-loss-function%2F566%2F4 "2020-08-03T15:26:18Z")

</div>

Thanks, @valhalla Suraj! This is very helpful!

Could you help me understand the difference of ‘forward’ and ‘\_step’ in your [example code](https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb):

```
def forward(
      self, input_ids, attention_mask=None, decoder_input_ids=None, decoder_attention_mask=None, lm_labels=None
  ):
    return self.model(
        input_ids,
        attention_mask=attention_mask,
        decoder_input_ids=decoder_input_ids,
        decoder_attention_mask=decoder_attention_mask,
        lm_labels=lm_labels,
    )

  def _step(self, batch):
    lm_labels = batch["target_ids"]
    lm_labels[lm_labels[:, :] == self.tokenizer.pad_token_id] = -100

    outputs = self(
        input_ids=batch["source_ids"],
        attention_mask=batch["source_mask"],
        lm_labels=lm_labels,
        decoder_attention_mask=batch['target_mask']
    )

    loss = outputs[0]

    return loss

```

My understanding is ‘**self(xxxxxx)**’ in the ‘ **\_step** ’ is running the ‘ **forward** ’ function defined above and the ‘ **self.model** (xxxxxx)’ in the ‘ **forward** ’ function above is running the ‘ **forward** ’ function of T5ForConditionalGeneration.from\_pretrained(hparams.model\_name\_or\_path).

so to define my own loss function, I need to define it in the ‘ **\_step** ’ like:

```
def _step(self, batch):
        labels = batch["target_ids"]
        labels[labels[:, :] == self.tokenizer.pad_token_id] = -100

        outputs = self(
            input_ids=batch["source_ids"],
            attention_mask=batch["source_mask"],
            labels=labels, 
            decoder_attention_mask=batch['target_mask']
        ) 

        loss1 = outputs[0]

       beam_outputs = self.generate( xxxxxx )
       loss2 = my_metrics(beam_outputs)
       loss = loss1+loss2            
       return loss

```

Here I use self.generate( xxxxxx ) rather than self.model.generate(xx) because self.model is that pretrained model in the input, right?

Thanks!!!

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-user-defined-loss-function%2F566)._