AlexStamp/roberta-base-lora-yelp-ratings

RoBERTa-base + LoRA PEFT for Yelp Star Rating Classification

This repository contains a Low-Rank Adaptation (LoRA) adapter trained on top of FacebookAI/roberta-base for predicting 5-class Yelp customer review ratings (1 to 5 stars).

The goal of this experiment was to test Parameter-Efficient Fine-Tuning (PEFT) against traditional Full Fine-Tuning: can we match full-parameter performance while updating under 1% of the network?

🎯 Motivation & Key Results: Full Fine-Tuning vs. LoRA

Full fine-tuning requires updating all ~125 Million parameters of roberta-base. Using LoRA, we froze $99.3\mathrm{\%}99.3\backslash \%$ of the base model weights and only trained lightweight low-rank matrices inserted into the query/value attention layers, along with the classification head.

Fine-Tuning Strategy	Trainable Parameters	Macro F1-Score	Off-by-1 Accuracy ( $\le 1\backslash le\; 1$ star error)
Full Fine-Tuning	125,533,445 (100%)	0.631	0.980
LoRA PEFT (Ours)	887,813 (~0.7%) 🟢	0.633 🟢	0.982

Key Takeaways

1. Superior Regularization: LoRA achieved a slightly higher Macro F1 score (0.633 vs. 0.631 on the test set) than full fine-tuning. Freezing 99% of the backbone acted as a strong regularizer, preventing overfitting on noisy text.
2. Storage Efficiency: Instead of storing/downloading a monolithic ~500 MB model checkpoint, the LoRA adapter weights consume less than 5 MB.
3. Hyperparameter Tuning: While full fine-tuning used $\text{LR}=2\times {10}^{-5}\backslash text\{LR\}\; =\; 2\; \backslash times\; 10^\{-5\}$, LoRA required a higher learning rate ( $\text{LR}=3\times {10}^{-4}\backslash text\{LR\}\; =\; 3\; \backslash times\; 10^\{-4\}$ ) to allow the low-rank bottleneck layers to adapt efficiently.

---

Training and evaluation data

The model was fine-tuned on a 10,000-review subset of the Yelp review data associated with the RecSys2013: Yelp Business Rating Prediction Kaggle competition.

The original competition training data contains approximately 230,000 reviews. A 10,000-review subset was used for this project to enable rapid experimentation and iteration on modest computational resources.

The subset contains review text and the corresponding 1–5 star rating. The labels were mapped to integer class IDs from 0 to 4 for model training. For this experimental project, the 10k subset provides a practical dataset size for investigating Transformer fine-tuning and parameter-efficient adaptation.

Data Split

The dataset was divided using stratified sampling:

• Training: 8,000 reviews (80%)
• Validation: 1,000 reviews (10%)
• Test: 1,000 reviews (10%)

Stratification was used to preserve the star-rating distribution across the splits. No explicit class rebalancing or oversampling was performed.

---

🛠️ LoRA Configuration & Hyperparameters

• Base Model: FacebookAI/roberta-base
• Task: 5-Class Multi-Class Text Classification (num_labels=5)
• PEFT Method: LoRA (peft library)
• Target Modules: Attention projection layers (query, value) + modules_to_save=["classifier"]
• LoRA Rank ( $rr$ ): 8
• LoRA Alpha ( $\alpha \backslash alpha$ ): 16 ( $\text{scaling}=2.0\backslash text\{scaling\}\; =\; 2.0$ )
• LoRA Dropout: 0.1
• Learning Rate: 3e-4
• Batch Size: 16 (Train & Eval)
• Epochs: 3
• Optimizer / Precision: AdamW (fp16=True)

---

Training procedure

Training Details

Training was performed using the Hugging Face Transformers Trainer API with the PyTorch backend.

• Hardware: NVIDIA T4 GPU (Google Colab)
• Framework: Hugging Face Transformers
• PEFT framework: Hugging Face PEFT
• Epochs: 3
• Learning rate: 3e-4
• Training batch size: 16
• Evaluation batch size: 16
• Weight decay: 0.01
• Precision: FP16
• Random seed: 42
• Logging: Epoch-based
• Model selection: Best validation macro-F1

The higher learning rate used for LoRA reflects the different optimization regime of parameter-efficient fine-tuning compared with full model fine-tuning.

Training results

Training Loss	Epoch	Step	Validation Loss	Validation Accuracy	Validation F1 Macro	Validation Precision Macro	Validation Recall Macro	Validation MAE
0.9765	1.0	500	0.8275	0.637	0.5836	0.6124	0.6119	0.41
0.8179	2.0	1000	0.7886	0.653	0.6139	0.6316	0.6136	0.381
0.7616	3.0	1500	0.7786	0.663	0.6279	0.6341	0.6311	0.368

Note: See comment in the Validation Behaviour section below.

Framework versions

• PEFT 0.20.0
• Transformers 5.15.0
• Pytorch 2.11.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2

---

Evaluation

The model was evaluated on the held-out test set using:

Accuracy	Macro F1	Macro Precision	Macro Recall	ROC-AUC	PR-AUC	MAE	MSE	Off-by-1 Accuracy	Off-by-1 Macro-F1

ROC-AUC and PR-AUC are calculated using a one-vs-rest formulation with macro averaging across the five classes. Because star ratings are ordinal, MAE, MSE, RMSE and off-by-1 metrics provide additional information about the magnitude of prediction errors.

Test Set Results

Metric	RoBERTa-LoRA	Full fine-tuning
Accuracy	0.650	0.639
Macro F1	0.633	0.631
Macro Precision	0.637	0.634
Macro Recall	0.635	0.630
ROC-AUC	0.905	0.904
PR-AUC	0.682	0.682
MAE	0.372	0.387
MSE	0.424	0.451
RMSE	0.651	0.672
Off-by-1 Accuracy	0.982	0.980
Off-by-1 Macro-F1	0.975	0.973

Comparison with Full Fine-Tuning

The central purpose of this experiment was to compare LoRA with conventional full fine-tuning of the same roberta-base model. Both models used the same:

• Dataset
• Train/validation/test split
• Base architecture
• Tokenization
• Number of epochs
• Evaluation procedure

The main difference was the fine-tuning strategy. The results are remarkably close despite LoRA updating only a small fraction of the parameters of the pretrained model.

Interestingly, LoRA achieved slightly better results than full fine-tuning on the reported evaluation metrics, including Macro F1, ROC-AUC, MSE and RMSE. However, the differences are small and the test set contains only 1,000 reviews. Therefore, these results should be interpreted as evidence of comparable performance in this experiment, rather than as evidence that LoRA is inherently superior to full fine-tuning.

Confusion Matrix

The confusion matrix below shows the distribution of predictions across the five star-rating classes.

---

Key Takeaway

The main finding of this experiment is that LoRA achieved essentially the same predictive performance as full fine-tuning while training only a small fraction of the pretrained model parameters.

This is particularly interesting given the relatively small dataset used in the experiment.

The results suggest that, for this text-classification task, updating the full RoBERTa model was not necessary to obtain strong performance. A lightweight low-rank adaptation was sufficient to reach comparable results.

The slightly better LoRA results should not be overinterpreted given the size of the test set, but they demonstrate the practical potential of parameter-efficient fine-tuning.

---

Validation Behaviour

An interesting difference was observed during training. For the full fine-tuning experiment, validation loss began increasing after the first epoch despite continued reductions in training loss.

For the LoRA experiment, validation loss continued to decrease throughout all three epochs:

Epoch	Training Loss	Validation Loss	Validation Accuracy	Validation Macro F1
1	0.9765	0.8275	0.637	0.5836
2	0.8179	0.7886	0.653	0.6139
3	0.7616	0.7786	0.663	0.6279

This behaviour is consistent with LoRA providing a degree of regularization in this particular experiment, although further experiments would be required to establish whether this behaviour generalizes.

---

🚀 How to Load and Run Inference

Because this repository stores adapter weights, you must load the base FacebookAI/roberta-base model first and attach the LoRA adapter.

import numpy as np
import torch
from peft import PeftModel, PeftConfig
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# 1. Path to this adapter repository on the Hub
peft_repo_id = "AlexStamp/roberta-base-lora-yelp-ratings"

# 2. Extract base model metadata
config = PeftConfig.from_pretrained(peft_repo_id)
base_model_name = config.base_model_name_or_path  # "FacebookAI/roberta-base"

id2label = {0: "1 Star", 1: "2 Stars", 2: "3 Stars", 3: "4 Stars", 4: "5 Stars"}
label2id = {v: k for k, v in id2label.items()}

# 3. Load Tokenizer & Base Model
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForSequenceClassification.from_pretrained(
    base_model_name,
    num_labels=5,
    id2label=id2label,
    label2id=label2id
)

# 4. Attach LoRA Adapter
model = PeftModel.from_pretrained(base_model, peft_repo_id)
model.eval()

# 5. Run Prediction
inputs = tokenizer("The food was incredible, but we had to wait 45 minutes for a table.", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
    pred = logits.argmax(-1).item()

print("Predicted Rating:", model.config.id2label[pred])

---

Limitations

• The dataset contains only 10,000 reviews, substantially smaller than the original Yelp competition training dataset.
• The held-out test set contains only 1,000 reviews, so small differences between the two fine-tuning strategies should not be overinterpreted.
• The star-rating classes are imbalanced.
• No explicit class rebalancing or oversampling was performed.
• The model predicts ratings from review text alone and does not use user, business or historical interaction information.
• The model is trained on Yelp review data and may not generalize to other domains.
• The experiment compares a single LoRA configuration and a single full-fine-tuning configuration rather than performing exhaustive hyperparameter optimization.
• The experiment is intended for educational and experimental purposes rather than production deployment.

---

Intended Use

This model is intended for:

• Experimentation with parameter-efficient fine-tuning
• Learning and demonstrating LoRA
• Comparing LoRA with conventional Transformer fine-tuning
• Supervised NLP text classification
• Exploring the relationship between model capacity, trainable parameters and predictive performance

It should not be considered a production-grade Yelp rating prediction system.

---

Libraries

Main libraries used:

• transformers — model, tokenizer and Trainer API
• peft — LoRA / parameter-efficient fine-tuning
• datasets — dataset preparation and splitting
• scikit-learn — additional evaluation metrics and confusion matrix
• PyTorch — underlying deep learning framework

---

Future Work

Possible extensions include:

• Testing additional LoRA configurations and ranks
• Comparing LoRA with other parameter-efficient fine-tuning methods
• Evaluating on a larger subset of the Yelp dataset
• Investigating the effect of class imbalance
• Measuring training time and GPU memory usage more systematically
• Comparing the number of trainable parameters and storage requirements of LoRA and full fine-tuning

---

Citation

If you use this model or project, please refer to the original RoBERTa paper and the original Yelp/RecSys2013 dataset source.

RoBERTa:
Liu et al., A Robustly Optimized BERT Pretraining Approach, 2019.

Dataset:
Yelp / RecSys2013: Yelp Business Rating Prediction, Kaggle.

---

Model Repository

This repository contains the LoRA adapter, tokenizer, configuration and associated model-card documentation for the Yelp review classification experiment.

A conventionally fine-tuned RoBERTa model trained using the same dataset and evaluation procedure is available separately for comparison.