vasiledraguta/deberta-v3-base-fever-verifier-20k-finetuned

DeBERTa v3 Base FEVER Verifier, 20k Fine-Tuned

This is a microsoft/deberta-v3-base sequence-classification model fine-tuned as a FEVER-style verifier. It predicts one of:

• SUPPORTED
• REFUTED
• NOT_ENOUGH_INFO

The model is intended to be used inside a retrieval-augmented fact-checking pipeline. It expects the input to contain a claim plus retrieved evidence passages, not just the claim alone.

Training Data

The verifier was trained from FEVER train only. FEVER labelled_dev was not used for training.

Dataset build summary:

• Source dataset: FEVER train
• Retrieval setup: title retrieval + dense retrieval, with noisy retrieved evidence passages
• Requested build size: 20,000 unique FEVER train claims
• Final training rows: 14,508
• Held-out train-derived verifier dev rows: 2,000
• Max passages in training examples: 3
• Base model: microsoft/deberta-v3-base
• Fine-tuning epochs: 2
• Max length: 384

Metrics

Verifier dev split from the generated FEVER train examples:

Metric	Value
Accuracy	0.819
Macro F1	0.7947
Eval loss	0.5757

Full RAG pipeline evaluation on 500 FEVER labelled_dev claims with train overlaps excluded:

Pipeline	Accuracy	Macro F1	Citation hallucination
Baseline RAG + NLI synthesis	0.618	0.597	0.0%
Calibrated trained verifier	0.704	0.701	0.0%

Input Format

Use the same format used during fine-tuning:

Claim: <claim>

Evidence:
[1] Source: <page title>
<retrieved passage text>

[2] Source: <page title>
<retrieved passage text>

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "vasiledraguta/deberta-v3-base-fever-verifier-20k-finetuned"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

text = """Claim: Lionel Messi has won the FIFA World Cup.

Evidence:
[1] Source: Lionel Messi
Lionel Messi captained Argentina to victory in the 2022 FIFA World Cup."""

inputs = tokenizer(
    text,
    truncation=True,
    max_length=384,
    return_tensors="pt",
)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]

label_id = int(probs.argmax())
label = model.config.id2label[label_id]
confidence = float(probs[label_id])

print(label, confidence)

Intended Use

This model is for educational and experimental fact-checking pipelines that retrieve evidence first and then classify whether the retrieved evidence supports, refutes, or does not contain enough information for the claim.

It should not be treated as a standalone truth oracle. Accuracy depends heavily on retrieval quality and evidence coverage.

Limitations

• The model only classifies the claim/evidence pair; it does not retrieve evidence by itself.
• It was trained on retrieved FEVER evidence, so out-of-domain claims may behave differently.
• NOT_ENOUGH_INFO remains the hardest class in the full pipeline.
• The hosted model does not include the project retrieval index or FEVER data.