IoannisKat1/bge-m3-new2

bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Who should communicate a personal data breach to the data subject?',
    'The controller should communicate to the data subject a personal data breach, without undue delay, where that personal data breach is likely to result in a high risk to the rights and freedoms of the natural person in order to allow him or her to take the necessary precautions. The communication should describe the nature of the personal data breach as well as recommendations for the natural person concerned to mitigate potential adverse effects. Such communications to data subjects should be made as soon as reasonably feasible and in close cooperation with the supervisory authority, respecting guidance provided by it or by other relevant authorities such as law-enforcement authorities. For example, the need to mitigate an immediate risk of damage would call for prompt communication with data subjects whereas the need to implement appropriate measures against continuing or similar personal data breaches may justify more time for communication.',
    "1.In the absence of an adequacy decision pursuant to Article 45(3), or of appropriate safeguards pursuant to Article 46, including binding corporate rules, a transfer or a set of transfers of personal data to a third country or an international organisation shall take place only on one of the following conditions: (a)  the data subject has explicitly consented to the proposed transfer, after having been informed of the possible risks of such transfers for the data subject due to the absence of an adequacy decision and appropriate safeguards; (b)  the transfer is necessary for the performance of a contract between the data subject and the controller or the implementation of pre-contractual measures taken at the data subject's request; (c)  the transfer is necessary for the conclusion or performance of a contract concluded in the interest of the data subject between the controller and another natural or legal person; (d)  the transfer is necessary for important reasons of public interest; (e)  the transfer is necessary for the establishment, exercise or defence of legal claims; (f)  the transfer is necessary in order to protect the vital interests of the data subject or of other persons, where the data subject is physically or legally incapable of giving consent; (g)  the transfer is made from a register which according to Union or Member State law is intended to provide information to the public and which is open to consultation either by the public in general or by any person who can demonstrate a legitimate interest, but only to the extent that the conditions laid down by Union or Member State law for consultation are fulfilled in the particular case. Where a transfer could not be based on a provision in Article 45 or 46, including the provisions on binding corporate rules, and none of the derogations for a specific situation referred to in the first subparagraph of this paragraph is applicable, a transfer to a third country or an international organisation may take place only if the transfer is not repetitive, concerns only a limited number of data subjects, is necessary for the purposes of compelling legitimate interests pursued by the controller which are not overridden by the interests or rights and freedoms of the data subject, and the controller has assessed all the circumstances surrounding the data transfer and has on the basis of that assessment provided suitable safeguards with regard to the protection of personal data. The controller shall inform the supervisory authority of the transfer. The controller shall, in addition to providing the information referred to in Articles 13 and 14, inform the data subject of the transfer and on the compelling legitimate interests pursued.\n2.A transfer pursuant to point (g) of the first subparagraph of paragraph 1 shall not involve the entirety of the personal data or entire categories of the personal data contained in the register. Where the register is intended for consultation by persons having a legitimate interest, the transfer shall be made only at the request of those persons or if they are to be the recipients. 4.5.2016 L 119/64  \n3.Points (a), (b) and (c) of the first subparagraph of paragraph 1 and the second subparagraph thereof shall not apply to activities carried out by public authorities in the exercise of their public powers.\n4.The public interest referred to in point (d) of the first subparagraph of paragraph 1 shall be recognised in Union law or in the law of the Member State to which the controller is subject.\n5.In the absence of an adequacy decision, Union or Member State law may, for important reasons of public interest, expressly set limits to the transfer of specific categories of personal data to a third country or an international organisation. Member States shall notify such provisions to the Commission.\n6.The controller or processor shall document the assessment as well as the suitable safeguards referred to in the second subparagraph of paragraph 1 of this Article in the records referred to in Article 30.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7041, 0.2945],
#         [0.7041, 1.0000, 0.3575],
#         [0.2945, 0.3575, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.5051
cosine_accuracy@3	0.5379
cosine_accuracy@5	0.5783
cosine_accuracy@10	0.6237
cosine_precision@1	0.5051
cosine_precision@3	0.4832
cosine_precision@5	0.4495
cosine_precision@10	0.3889
cosine_recall@1	0.1083
cosine_recall@3	0.2625
cosine_recall@5	0.3474
cosine_recall@10	0.4711
cosine_ndcg@10	0.5624
cosine_mrr@10	0.5318
cosine_map@100	0.6073

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
cosine_accuracy@1	0.4924
cosine_accuracy@3	0.5328
cosine_accuracy@5	0.5707
cosine_accuracy@10	0.6212
cosine_precision@1	0.4924
cosine_precision@3	0.4747
cosine_precision@5	0.4439
cosine_precision@10	0.3828
cosine_recall@1	0.105
cosine_recall@3	0.257
cosine_recall@5	0.3459
cosine_recall@10	0.472
cosine_ndcg@10	0.5541
cosine_mrr@10	0.5217
cosine_map@100	0.6007

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
cosine_accuracy@1	0.4621
cosine_accuracy@3	0.5
cosine_accuracy@5	0.548
cosine_accuracy@10	0.5783
cosine_precision@1	0.4621
cosine_precision@3	0.4461
cosine_precision@5	0.4227
cosine_precision@10	0.3644
cosine_recall@1	0.0973
cosine_recall@3	0.238
cosine_recall@5	0.3246
cosine_recall@10	0.4473
cosine_ndcg@10	0.5236
cosine_mrr@10	0.4904
cosine_map@100	0.5706

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Metric	Value
cosine_accuracy@1	0.4596
cosine_accuracy@3	0.4823
cosine_accuracy@5	0.5202
cosine_accuracy@10	0.5758
cosine_precision@1	0.4596
cosine_precision@3	0.4377
cosine_precision@5	0.4071
cosine_precision@10	0.3545
cosine_recall@1	0.097
cosine_recall@3	0.2362
cosine_recall@5	0.314
cosine_recall@10	0.4391
cosine_ndcg@10	0.5128
cosine_mrr@10	0.4834
cosine_map@100	0.5606

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
cosine_accuracy@1	0.3662
cosine_accuracy@3	0.4015
cosine_accuracy@5	0.4571
cosine_accuracy@10	0.5126
cosine_precision@1	0.3662
cosine_precision@3	0.3544
cosine_precision@5	0.3369
cosine_precision@10	0.3048
cosine_recall@1	0.0789
cosine_recall@3	0.1968
cosine_recall@5	0.2664
cosine_recall@10	0.3808
cosine_ndcg@10	0.4353
cosine_mrr@10	0.3988
cosine_map@100	0.4852

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,580 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 8 tokens mean: 17.23 tokens max: 37 tokens	min: 27 tokens mean: 739.0 tokens max: 2824 tokens

• Samples:

  anchor	positive
  How has technology affected the collection and sharing of personal data?	Rapid technological developments and globalisation have brought new challenges for the protection of personal data. The scale of the collection and sharing of personal data has increased significantly. Technology allows both private companies and public authorities to make use of personal data on an unprecedented scale in order to pursue their activities. Natural persons increasingly make personal information available publicly and globally. Technology has transformed both the economy and social life, and should further facilitate the free flow of personal data within the Union and the transfer to third countries and international organisations, while ensuring a high level of the protection of personal data.
  When was the Official Journal of the European Union published?	In order to create incentives to apply pseudonymisation when processing personal data, measures of pseudonymisation should, whilst allowing general analysis, be possible within the same controller when that controller has taken technical and organisational measures necessary to ensure, for the processing concerned, that this Regulation is implemented, and that additional information for attributing the personal data to a specific data subject is kept separately. The controller processing the personal data should indicate the authorised persons within the same controller. 4.5.2016 L 119/5 Official Journal of the European Union EN
  What is the meaning of 'processing' in the context of personal data?	1) 'personal data' means any information relating to an identified or identifiable natural person ('data subject'); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person; (2) ‘processing’ means any operation or set of operations which is performed on personal data or on sets of personal data, whether or not by automated means, such as collection, recording, organisation, structuring, storage, adaptation or alteration, retrieval, consultation, use, disclosure by transmission, dissemination or otherwise making available, alignment or combination, restriction, erasure or destruction; (3) ‘restriction of processing’ means the marking of stored personal data with the aim of limiting their processin...

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• gradient_accumulation_steps: 4
• learning_rate: 3e-05
• num_train_epochs: 20
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 4
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 3e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 20
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
-1	-1	-	0.4571	0.4375	0.4008	0.3170	0.2563
0.2020	10	21.3767	-	-	-	-	-
0.4040	20	19.2573	-	-	-	-	-
0.6061	30	18.6623	-	-	-	-	-
0.8081	40	18.2772	-	-	-	-	-
1.0	50	16.0375	0.4596	0.4511	0.4154	0.3519	0.2872
1.2020	60	17.5302	-	-	-	-	-
1.4040	70	17.0313	-	-	-	-	-
1.6061	80	14.3965	-	-	-	-	-
1.8081	90	14.3407	-	-	-	-	-
2.0	100	12.6464	0.4835	0.4805	0.4585	0.4101	0.3288
2.2020	110	12.1612	-	-	-	-	-
2.4040	120	12.1747	-	-	-	-	-
2.6061	130	10.8968	-	-	-	-	-
2.8081	140	11.3075	-	-	-	-	-
3.0	150	9.3359	0.5314	0.5141	0.4915	0.4568	0.3777
3.2020	160	8.9723	-	-	-	-	-
3.4040	170	9.4735	-	-	-	-	-
3.6061	180	9.3438	-	-	-	-	-
3.8081	190	9.643	-	-	-	-	-
4.0	200	8.3103	0.5374	0.5426	0.5113	0.4848	0.4180
4.2020	210	7.3441	-	-	-	-	-
4.4040	220	7.8091	-	-	-	-	-
4.6061	230	7.1368	-	-	-	-	-
4.8081	240	7.7728	-	-	-	-	-
5.0	250	8.0139	0.5495	0.5443	0.5049	0.4923	0.4108
5.2020	260	5.6836	-	-	-	-	-
5.4040	270	6.0556	-	-	-	-	-
5.6061	280	7.1601	-	-	-	-	-
5.8081	290	7.8578	-	-	-	-	-
6.0	300	5.7557	0.5700	0.5564	0.5235	0.5106	0.4292
6.2020	310	5.7718	-	-	-	-	-
6.4040	320	5.1293	-	-	-	-	-
6.6061	330	6.7643	-	-	-	-	-
6.8081	340	5.1791	-	-	-	-	-
7.0	350	5.4592	0.5624	0.5541	0.5236	0.5128	0.4353
7.2020	360	5.0135	-	-	-	-	-
7.4040	370	4.9931	-	-	-	-	-
7.6061	380	5.1182	-	-	-	-	-
7.8081	390	4.5808	-	-	-	-	-
8.0	400	4.6313	0.5716	0.5549	0.5362	0.4893	0.4189
8.2020	410	5.0518	-	-	-	-	-
8.4040	420	4.7227	-	-	-	-	-
8.6061	430	3.5472	-	-	-	-	-
8.8081	440	5.4843	-	-	-	-	-
9.0	450	4.6137	0.5738	0.5589	0.5320	0.4849	0.4286
-1	-1	-	0.5624	0.5541	0.5236	0.5128	0.4353

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.1.0
• Transformers: 4.51.3
• PyTorch: 2.8.0+cu126
• Accelerate: 1.10.1
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}