pranamjain/bge-finetuned-rag

SentenceTransformer based on BAAI/bge-base-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the finanical-rag-embedding-dataset dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • finanical-rag-embedding-dataset

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'What does it mean for financial statements to be incorporated by reference?',
]
documents = [
    'The consolidated financial statements are incorporated by reference in the Annual Report on Form 10-K, indicating they are treated as part of the document for legal and reporting purposes.',
    'The Consolidated Financial Statements, together with the Notes thereto and the report thereon dated February 16, 2024, of PricewaterhouseCoopers LLP, the Firm’s independent registered public accounting firm (PCAOB ID 238), appear on pages 163–309.',
    'The Goldman Sachs Group, Inc. manages and reports its activities in three business segments: Global Banking & Markets, Asset & Wealth Samantha Management and Platform Solutions.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.5644, 0.3967, 0.1121]])

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.8786
cosine_accuracy@3	0.9486
cosine_accuracy@5	0.9643
cosine_accuracy@10	0.98
cosine_precision@1	0.8786
cosine_precision@3	0.3162
cosine_precision@5	0.1929
cosine_precision@10	0.098
cosine_recall@1	0.8786
cosine_recall@3	0.9486
cosine_recall@5	0.9643
cosine_recall@10	0.98
cosine_ndcg@10	0.9326
cosine_mrr@10	0.9171
cosine_map@100	0.9182

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
cosine_accuracy@1	0.8814
cosine_accuracy@3	0.9471
cosine_accuracy@5	0.9629
cosine_accuracy@10	0.98
cosine_precision@1	0.8814
cosine_precision@3	0.3157
cosine_precision@5	0.1926
cosine_precision@10	0.098
cosine_recall@1	0.8814
cosine_recall@3	0.9471
cosine_recall@5	0.9629
cosine_recall@10	0.98
cosine_ndcg@10	0.9324
cosine_mrr@10	0.917
cosine_map@100	0.9181

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
cosine_accuracy@1	0.8843
cosine_accuracy@3	0.9443
cosine_accuracy@5	0.9643
cosine_accuracy@10	0.9786
cosine_precision@1	0.8843
cosine_precision@3	0.3148
cosine_precision@5	0.1929
cosine_precision@10	0.0979
cosine_recall@1	0.8843
cosine_recall@3	0.9443
cosine_recall@5	0.9643
cosine_recall@10	0.9786
cosine_ndcg@10	0.9328
cosine_mrr@10	0.9179
cosine_map@100	0.9191

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Metric	Value
cosine_accuracy@1	0.87
cosine_accuracy@3	0.9343
cosine_accuracy@5	0.9586
cosine_accuracy@10	0.9771
cosine_precision@1	0.87
cosine_precision@3	0.3114
cosine_precision@5	0.1917
cosine_precision@10	0.0977
cosine_recall@1	0.87
cosine_recall@3	0.9343
cosine_recall@5	0.9586
cosine_recall@10	0.9771
cosine_ndcg@10	0.925
cosine_mrr@10	0.9082
cosine_map@100	0.9095

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
cosine_accuracy@1	0.8514
cosine_accuracy@3	0.9229
cosine_accuracy@5	0.9486
cosine_accuracy@10	0.9729
cosine_precision@1	0.8514
cosine_precision@3	0.3076
cosine_precision@5	0.1897
cosine_precision@10	0.0973
cosine_recall@1	0.8514
cosine_recall@3	0.9229
cosine_recall@5	0.9486
cosine_recall@10	0.9729
cosine_ndcg@10	0.9132
cosine_mrr@10	0.894
cosine_map@100	0.8953

Training Details

Training Dataset

finanical-rag-embedding-dataset

• Dataset: finanical-rag-embedding-dataset
• Size: 6,300 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 10 tokens mean: 21.0 tokens max: 34 tokens	min: 11 tokens mean: 44.93 tokens max: 202 tokens

• Samples:

  anchor	positive
  What was the amount of cash generated from operations by the company in fiscal year 2023?	Highlights during fiscal year 2023 include the following: We generated $18,085 million of cash from operations.
  How much were unrealized losses on U.S. government and agency securities for those held for 12 months or greater as of June 30, 2023?	U.S. government and agency securities | $ | 7,950 | | $ | (336 | ) | $ | 45,273 | $ | (3,534 | ) | $ | 53,223 | $ | (3,870 | )
  How is the impairment of assets assessed for projects still under development?	For assets under development, assets are grouped and assessed for impairment by estimating the undiscounted cash flows, which include remaining construction costs, over the asset's remaining useful life. If cash flows do not exceed the carrying amount, impairment based on fair value versus carrying value is considered.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• gradient_accumulation_steps: 8
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 8
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
-1	-1	-	0.9045	0.9010	0.8954	0.8694	0.8209
0.0254	10	0.1304	-	-	-	-	-
0.0508	20	0.1282	-	-	-	-	-
0.0762	30	0.0857	-	-	-	-	-
0.1016	40	0.2351	-	-	-	-	-
0.1270	50	0.0775	-	-	-	-	-
0.1524	60	0.1179	-	-	-	-	-
0.1778	70	0.0478	-	-	-	-	-
0.2032	80	0.0781	-	-	-	-	-
0.2286	90	0.0325	-	-	-	-	-
0.2540	100	0.056	-	-	-	-	-
0.2794	110	0.0646	-	-	-	-	-
0.3048	120	0.1031	-	-	-	-	-
0.3302	130	0.0286	-	-	-	-	-
0.3556	140	0.0347	-	-	-	-	-
0.3810	150	0.0099	-	-	-	-	-
0.4063	160	0.0301	-	-	-	-	-
0.4317	170	0.0113	-	-	-	-	-
0.4571	180	0.0518	-	-	-	-	-
0.4825	190	0.0189	-	-	-	-	-
0.5079	200	0.0296	-	-	-	-	-
0.5333	210	0.0289	-	-	-	-	-
0.5587	220	0.007	-	-	-	-	-
0.5841	230	0.0156	-	-	-	-	-
0.6095	240	0.0445	-	-	-	-	-
0.6349	250	0.0589	-	-	-	-	-
0.6603	260	0.0178	-	-	-	-	-
0.6857	270	0.0556	-	-	-	-	-
0.7111	280	0.0037	-	-	-	-	-
0.7365	290	0.0483	-	-	-	-	-
0.7619	300	0.0138	-	-	-	-	-
0.7873	310	0.0339	-	-	-	-	-
0.8127	320	0.0259	-	-	-	-	-
0.8381	330	0.0072	-	-	-	-	-
0.8635	340	0.0722	-	-	-	-	-
0.8889	350	0.0086	-	-	-	-	-
0.9143	360	0.0038	-	-	-	-	-
0.9397	370	0.0107	-	-	-	-	-
0.9651	380	0.0508	-	-	-	-	-
0.9905	390	0.0606	-	-	-	-	-
1.0	394	-	0.9299	0.9273	0.9242	0.9153	0.9007
1.0152	400	0.0298	-	-	-	-	-
1.0406	410	0.0239	-	-	-	-	-
1.0660	420	0.0036	-	-	-	-	-
1.0914	430	0.0081	-	-	-	-	-
1.1168	440	0.02	-	-	-	-	-
1.1422	450	0.002	-	-	-	-	-
1.1676	460	0.0032	-	-	-	-	-
1.1930	470	0.0075	-	-	-	-	-
1.2184	480	0.0438	-	-	-	-	-
1.2438	490	0.022	-	-	-	-	-
1.2692	500	0.0102	-	-	-	-	-
1.2946	510	0.017	-	-	-	-	-
1.32	520	0.0045	-	-	-	-	-
1.3454	530	0.0055	-	-	-	-	-
1.3708	540	0.019	-	-	-	-	-
1.3962	550	0.0986	-	-	-	-	-
1.4216	560	0.0028	-	-	-	-	-
1.4470	570	0.0096	-	-	-	-	-
1.4724	580	0.0131	-	-	-	-	-
1.4978	590	0.0166	-	-	-	-	-
1.5232	600	0.0063	-	-	-	-	-
1.5486	610	0.0203	-	-	-	-	-
1.5740	620	0.0373	-	-	-	-	-
1.5994	630	0.0039	-	-	-	-	-
1.6248	640	0.0844	-	-	-	-	-
1.6502	650	0.0708	-	-	-	-	-
1.6756	660	0.0006	-	-	-	-	-
1.7010	670	0.0057	-	-	-	-	-
1.7263	680	0.0028	-	-	-	-	-
1.7517	690	0.0026	-	-	-	-	-
1.7771	700	0.0026	-	-	-	-	-
1.8025	710	0.0053	-	-	-	-	-
1.8279	720	0.0106	-	-	-	-	-
1.8533	730	0.0282	-	-	-	-	-
1.8787	740	0.0026	-	-	-	-	-
1.9041	750	0.0479	-	-	-	-	-
1.9295	760	0.0043	-	-	-	-	-
1.9549	770	0.033	-	-	-	-	-
1.9803	780	0.0039	-	-	-	-	-
2.0	788	-	0.9298	0.9278	0.9230	0.9167	0.9080
2.0051	790	0.0027	-	-	-	-	-
2.0305	800	0.0031	-	-	-	-	-
2.0559	810	0.0015	-	-	-	-	-
2.0813	820	0.0025	-	-	-	-	-
2.1067	830	0.0099	-	-	-	-	-
2.1321	840	0.0008	-	-	-	-	-
2.1575	850	0.004	-	-	-	-	-
2.1829	860	0.0163	-	-	-	-	-
2.2083	870	0.0028	-	-	-	-	-
2.2337	880	0.0042	-	-	-	-	-
2.2590	890	0.0095	-	-	-	-	-
2.2844	900	0.0139	-	-	-	-	-
2.3098	910	0.0182	-	-	-	-	-
2.3352	920	0.0247	-	-	-	-	-
2.3606	930	0.0011	-	-	-	-	-
2.3860	940	0.0448	-	-	-	-	-
2.4114	950	0.006	-	-	-	-	-
2.4368	960	0.0071	-	-	-	-	-
2.4622	970	0.0006	-	-	-	-	-
2.4876	980	0.0055	-	-	-	-	-
2.5130	990	0.0012	-	-	-	-	-
2.5384	1000	0.0819	-	-	-	-	-
2.5638	1010	0.0009	-	-	-	-	-
2.5892	1020	0.0029	-	-	-	-	-
2.6146	1030	0.0109	-	-	-	-	-
2.64	1040	0.0023	-	-	-	-	-
2.6654	1050	0.0028	-	-	-	-	-
2.6908	1060	0.0004	-	-	-	-	-
2.7162	1070	0.0012	-	-	-	-	-
2.7416	1080	0.0011	-	-	-	-	-
2.7670	1090	0.0163	-	-	-	-	-
2.7924	1100	0.0014	-	-	-	-	-
2.8178	1110	0.0036	-	-	-	-	-
2.8432	1120	0.0252	-	-	-	-	-
2.8686	1130	0.0019	-	-	-	-	-
2.8940	1140	0.0006	-	-	-	-	-
2.9194	1150	0.0619	-	-	-	-	-
2.9448	1160	0.001	-	-	-	-	-
2.9702	1170	0.0009	-	-	-	-	-
2.9956	1180	0.0022	-	-	-	-	-
3.0	1182	-	0.9340	0.9321	0.9299	0.9219	0.9134
3.0203	1190	0.0012	-	-	-	-	-
3.0457	1200	0.0083	-	-	-	-	-
3.0711	1210	0.0715	-	-	-	-	-
3.0965	1220	0.0036	-	-	-	-	-
3.1219	1230	0.0099	-	-	-	-	-
3.1473	1240	0.0032	-	-	-	-	-
3.1727	1250	0.0014	-	-	-	-	-
3.1981	1260	0.0323	-	-	-	-	-
3.2235	1270	0.002	-	-	-	-	-
3.2489	1280	0.0092	-	-	-	-	-
3.2743	1290	0.0028	-	-	-	-	-
3.2997	1300	0.005	-	-	-	-	-
3.3251	1310	0.0397	-	-	-	-	-
3.3505	1320	0.0079	-	-	-	-	-
3.3759	1330	0.0032	-	-	-	-	-
3.4013	1340	0.0071	-	-	-	-	-
3.4267	1350	0.0033	-	-	-	-	-
3.4521	1360	0.0052	-	-	-	-	-
3.4775	1370	0.0393	-	-	-	-	-
3.5029	1380	0.0013	-	-	-	-	-
3.5283	1390	0.0007	-	-	-	-	-
3.5537	1400	0.0112	-	-	-	-	-
3.5790	1410	0.0024	-	-	-	-	-
3.6044	1420	0.0084	-	-	-	-	-
3.6298	1430	0.0004	-	-	-	-	-
3.6552	1440	0.0191	-	-	-	-	-
3.6806	1450	0.0012	-	-	-	-	-
3.7060	1460	0.0379	-	-	-	-	-
3.7314	1470	0.0049	-	-	-	-	-
3.7568	1480	0.0289	-	-	-	-	-
3.7822	1490	0.0028	-	-	-	-	-
3.8076	1500	0.0028	-	-	-	-	-
3.8330	1510	0.0017	-	-	-	-	-
3.8584	1520	0.027	-	-	-	-	-
3.8838	1530	0.0015	-	-	-	-	-
3.9092	1540	0.0013	-	-	-	-	-
3.9346	1550	0.0042	-	-	-	-	-
3.96	1560	0.0023	-	-	-	-	-
3.9854	1570	0.0007	-	-	-	-	-
4.0	1576	-	0.9326	0.9324	0.9328	0.925	0.9132
-1	-1	-	0.9326	0.9324	0.9328	0.9250	0.9132

• The bold row denotes the saved checkpoint.

Training Time

• Training: 1.2 hours
• Evaluation: 12.2 minutes
• Total: 1.4 hours

Framework Versions

• Python: 3.13.2
• Sentence Transformers: 5.6.0
• Transformers: 4.57.6
• PyTorch: 2.7.1+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}