paulbontempo/bert-tagalog-cl-stage2

Tagalog BERT with Contrastive Dependency Learning (Stage 2)

This model is a fine-tuned BERT model for Tagalog with contrastive learning on dependency parse triples.

Model Description

• Base Model: paulbontempo/bert-tagalog-mlm-stage1
• Language: Tagalog
• Training Approach: Two-stage fine-tuning
  1. Stage 1: Masked Language Modeling (MLM) on Tagalog corpus
  2. Stage 2: Contrastive learning with InfoNCE loss on dependency parse triples

Model Architecture

This is a custom ContrastiveDependencyModel that wraps the Stage 1 BERT model with:

• Fine-tuned BERT encoder (from Stage 1)
• Trainable relation embeddings for dependency relations
• Projection head for contrastive learning (maps triples to 256-dim space)

Files

• final_model.pt: PyTorch checkpoint with full model state
• relation_to_id.json: Mapping of dependency relation labels to IDs
• model_code.py: Model architecture definition and loading utilities

Usage

Loading the Model

import torch
from model_code import ContrastiveDependencyModel, load_model

model, relation_to_id = load_model(
    model_path="final_model.pt",
    relation_json="relation_to_id.json",
    bert_model="paulbontempo/bert-tagalog-mlm-stage1"
)

model.eval()

Using with Hugging Face Hub

from huggingface_hub import hf_hub_download
from model_code import ContrastiveDependencyModel, load_model

model_path = hf_hub_download(
    repo_id="paulbontempo/bert-tagalog-cl-stage2",
    filename="final_model.pt"
)
relation_path = hf_hub_download(
    repo_id="paulbontempo/bert-tagalog-cl-stage2",
    filename="relation_to_id.json"
)

model, relation_to_id = load_model(
    model_path=model_path,
    relation_json=relation_path,
    bert_model="paulbontempo/bert-tagalog-mlm-stage1"
)

Training Details

• Loss Function: InfoNCE (contrastive loss)
• Temperature: 0.07
• Projection Dimension: 256

Citation

If you use this model, please cite appropriately.