# Finetune xlm roberta base(overfitting ,any solution )
**URL:**
**Author:**  [@afiqlol](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fafiqlol)%5C
**Post date:** [December 22, 2023, 2:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-xlm-roberta-base-overfitting-any-solution%2F66679%2F1 "2023-12-22T02:40:50Z")
import pandas as pd
from sklearn.model\_selection import train\_test\_split
import torch
from torch.utils.data import Dataset, DataLoader
from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments,DataCollatorWithPadding
# Load your dataset
df = pd.read\_excel(‘dataset\_final.xlsx’)
# Split the dataset into training and testing sets
train\_df, test\_df = train\_test\_split(df, test\_size=0.2, random\_state=42)
# Create a PyTorch Dataset
class CustomDataset(Dataset):
def **init** (self, encodings, labels):
self.encodings = encodings
self.labels = labels
```
def __getitem__ (self, idx):
item = {key: val[idx] for key, val in self.encodings.items()}
item['labels'] = torch.tensor(self.labels[idx])
return item
def __len__ (self):
return len(self.labels)
```
tokenizer = AutoTokenizer.from\_pretrained(“xlm-roberta-base”)
data\_collator = DataCollatorWithPadding(tokenizer=tokenizer)
train\_encodings = tokenizer(list(train\_df[‘text’]), truncation=True, padding=True, return\_tensors=‘pt’)
test\_encodings = tokenizer(list(test\_df[‘text’]), truncation=True, padding=True, return\_tensors=‘pt’)
# Prepare the labels
train\_labels = list(train\_df[‘label’])
test\_labels = list(test\_df[‘label’])
# Create DataLoader objects
train\_dataset = CustomDataset(train\_encodings, train\_labels)
test\_dataset = CustomDataset(test\_encodings, test\_labels)
import evaluate
accuracy = evaluate.load(“accuracy”)
import numpy as np
def compute\_metrics(eval\_pred):
predictions, labels = eval\_pred
predictions = np.argmax(predictions, axis=1)
return accuracy.compute(predictions=predictions, references=labels)
id2label = {0: “negative”, 1: “neutral”,2:“positive”}
label2id = {“negative”: 0, “neutral”: 1,“positive”:2}
model = AutoModelForSequenceClassification.from\_pretrained(
“xlm-roberta-base”,hidden\_dropout\_prob=0.3, attention\_probs\_dropout\_prob=0.25, num\_labels=3, id2label=id2label, label2id=label2id)
from transformers import EarlyStoppingCallback
training\_args = TrainingArguments(
output\_dir=“testing\_finetune”,
learning\_rate=2e-5,
per\_device\_train\_batch\_size=16,
per\_device\_eval\_batch\_size=16,
num\_train\_epochs=3.0,
warmup\_steps=600,
weight\_decay=0.1,
evaluation\_strategy=“epoch”,
save\_strategy=“epoch”,
load\_best\_model\_at\_end=True,
push\_to\_hub=True,
)
# Create and configure your Trainer
trainer = Trainer(
model=model,
args=training\_args,
train\_dataset=train\_dataset,
eval\_dataset=test\_dataset,
tokenizer=tokenizer,
data\_collator=data\_collator,
compute\_metrics=compute\_metrics,
)
trainer.train()
trainer.push\_to\_hub()
this is my code ,can someone help me? my result is always over fitting . thank you
---
**Author:**  [@OxxoCodes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOxxoCodes)%5C
**Post date:** [December 22, 2023, 3:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-xlm-roberta-base-overfitting-any-solution%2F66679%2F2 "2023-12-22T03:54:25Z")
Can we get some additional information? You’re training for 3 epochs with 600 warmup steps with a batch size of 16, so you’re doing warmup on 600x16=9600 sequences - is your dataset sufficiently large for this?
Also, when is it overfitting? Is it already overfit after the first epoch? Or only with the final model? If you can provide train/test loss curves or eval info per epoch that would be a big help in identifying the problem.
If it’s already overfitting after the first epoch, that could be an indication that perhaps your train and test splits aren’t representative of one another. Did you shuffle the dataset properly before creating these splits? Maybe the train set has a bunch of duplicates or near-duplicates, making it harder for the model to generalize? Also, what other hyperparameters have you tried? Did those improve/worsen your results?
---
**Author:**  [@afiqlol](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fafiqlol)%5C
**Post date:** [December 26, 2023, 4:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-xlm-roberta-base-overfitting-any-solution%2F66679%2F3 "2023-12-26T04:29:48Z")
for now i am using 3 epoch with no warmup ,for right now ,and my train dataset is 29,306 and my test is 7,328
training\_args = TrainingArguments(
output\_dir=“TESTING-ROBERTA”,
learning\_rate=2e-5,
per\_device\_train\_batch\_size=16,
per\_device\_eval\_batch\_size=16,
num\_train\_epochs=3,
weight\_decay=0.2,
evaluation\_strategy=“epoch”,
save\_strategy=“epoch”,
load\_best\_model\_at\_end=True,
logging\_steps=100,
save\_total\_limit=3,
push\_to\_hub=True,
)
trainer = Trainer(
model=model,
args=training\_args,
train\_dataset=tokenized\_malay[“train”],
eval\_dataset=tokenized\_malay[“test”],
tokenizer=tokenizer,
data\_collator=data\_collator,
compute\_metrics=compute\_metrics,
)
trainer.train()
this is my new trainner argumment
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
| --- | --- | --- | --- | --- |
| 0.6335 | 1.0 | 1832 | 0.6639 | 0.7250 |
| 0.5122 | 2.0 | 3664 | 0.6612 | 0.7404 |
| 0.3887 | 3.0 | 5496 | 0.7192 | 0.7389 |
this is my new result
---
**Author:**  [@afiqlol](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fafiqlol)%5C
**Post date:** [December 26, 2023, 4:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-xlm-roberta-base-overfitting-any-solution%2F66679%2F4 "2023-12-26T04:30:59Z")
i did shuffle the dataset and split into test\_size =0.2