# Finetune BERT for information extraction

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-bert-for-information-extraction%2F18748%3E%5C
**Category:** Beginners\
**Created:** [June 6, 2022, 3:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-bert-for-information-extraction%2F18748 "2022-06-06T03:44:21Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![erfangc](https://avatars.discourse-cdn.com/v4/letter/e/a587f6/32.png) [@erfangc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ferfangc)%5C
**Post date:** [June 6, 2022, 3:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-bert-for-information-extraction%2F18748%2F1 "2022-06-06T03:44:21Z")

</div>

Hello,

What is the best way to finetune/pre-train a model using BERT/T5 or BART (or something else) that would convert text to extracted JSON:

`My name is John` → `{ name: John }`

Assuming my data comes in the form of:

```auto
# train.tsv
text,extraction
name: Jill,name:Jill
My name is Jack,name:Jack

```

So far I’ve tried:

```auto
model_checkpoint = "google/mt5-small"
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

# ... skipping for brevity

args = Seq2SeqTrainingArguments(...)

data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)

trainer = Seq2SeqTrainer(
    model,
    args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["test"],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

trainer.train()

```

Is it a good idea to use an existing Seq2Seq model? Or is there a way to transfer the weights of the Transformer layers and discard the heads?

The reason I ask is b/c - while the Encoder weights are valuable … I feel like the Decoder weights or LMHeads are actually counterproductive - since they’ve been pre-trained on a natural language and not JSON (or any structured language) … and so my intuition would be to to re-initialize them randomly and train their weights from scratch