# Should I use BertModel or BertModelForLM?
**URL:**
**Author:**  [@anon44965279](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon44965279)%5C
**Post date:** [February 10, 2022, 11:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-use-bertmodel-or-bertmodelforlm%2F14574%2F1 "2022-02-10T11:08:36Z")
Hi, I would want to know what class should I use to extract word embeddings.
Given a sentence, I just want extract contextualised word embeddings.
However, I don’t really get how can I do it. In particular, I want to sum up the last four hidden layers of BERT. What’s the difference between hidden\_states from BertForMaskedLM and hidden\_states from BertModel? Thank you in advance.
```auto
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForMaskedLM.from_pretrained("bert-base-uncased")
inputs = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
outputs = model(**inputs, output_hidden_states=True)
outputs.hidden_states
```
or
```auto
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")
inputs = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
outputs = model(**inputs, output_hidden_states=True)
outputs.hidden_states
```
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [February 10, 2022, 2:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-use-bertmodel-or-bertmodelforlm%2F14574%2F2 "2022-02-10T14:58:41Z")
Both will return the same thing. `BertForMaskedLM` just adds a language modeling head on top of `BertModel`. The hidden states (which are the outputs of each layer of the Transformer encoder + the initial embeddings) are identical. As you don’t need to perform masked language modeling, you can just use `BertModel`, like so:
```auto
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")
inputs = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
outputs = model(**inputs, output_hidden_states=True)
contextualized_vector = outputs.last_hidden_state
```
This contextualized vector contains the last hidden states of the tokens of your input sentence. It’s a tensor of shape (batch\_size, number of tokens, hidden size).
---
**Author:**  [@anon44965279](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon44965279)%5C
**Post date:** [February 10, 2022, 3:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-use-bertmodel-or-bertmodelforlm%2F14574%2F3 "2022-02-10T15:02:42Z")
Ok, thank you. What about BertConfig? You’re right. However, if I use BertConfig the outputs are different
```auto
from transformers import AutoConfig, AutoModelForMaskedLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
config = AutoConfig.from_pretrained('bert-base-uncased', output_hidden_states=True)
model = AutoModelForMaskedLM.from_config(config)
print(model(**tokenizer(sent, return_tensors="pt"), output_hidden_states=True).hidden_states)
```