| --- |
| license: mit |
| language: en |
| tags: |
| - bert |
| - cloze |
| - distractor |
| - generation |
| datasets: |
| - dgen |
| widget: |
| - text: "The only known planet with large amounts of water is [MASK]. [SEP] earth" |
| - text: "The products of photosynthesis are glucose and [MASK] else. [SEP] oxygen" |
| --- |
| |
| # cdgp-csg-scibert-dgen |
|
|
| ## Model description |
|
|
| This model is a Candidate Set Generator in **"CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model", Findings of EMNLP 2022**. |
|
|
| Its input are stem and answer, and output is candidate set of distractors. It is fine-tuned by [**DGen**](https://github.com/DRSY/DGen) dataset based on [**allenai/scibert_scivocab_uncased**](https://huggingface.co/allenai/scibert_scivocab_uncased) model. |
|
|
| For more details, you can see our **paper** or [**GitHub**](https://github.com/AndyChiangSH/CDGP). |
|
|
| ## How to use? |
|
|
| 1. Download model by hugging face transformers. |
| ```python |
| from transformers import BertTokenizer, BertForMaskedLM, pipeline |
| |
| tokenizer = BertTokenizer.from_pretrained("AndyChiang/cdgp-csg-scibert-dgen") |
| csg_model = BertForMaskedLM.from_pretrained("AndyChiang/cdgp-csg-scibert-dgen") |
| ``` |
|
|
| 2. Create a unmasker. |
| ```python |
| unmasker = pipeline("fill-mask", tokenizer=tokenizer, model=csg_model, top_k=10) |
| ``` |
|
|
| 3. Use the unmasker to generate the candidate set of distractors. |
| ```python |
| sent = "The only known planet with large amounts of water is [MASK]. [SEP] earth" |
| cs = unmasker(sent) |
| print(cs) |
| ``` |
|
|
| ## Dataset |
|
|
| This model is fine-tuned by [DGen](https://github.com/DRSY/DGen) dataset, which covers multiple domains including science, vocabulary, common sense and trivia. It is compiled from a wide variety of datasets including SciQ, MCQL, AI2 Science Questions, etc. The detail of DGen dataset is shown below. |
|
|
| | DGen dataset | Train | Valid | Test | Total | |
| | ----------------------- | ----- | ----- | ---- | ----- | |
| | **Number of questions** | 2321 | 300 | 259 | 2880 | |
|
|
| You can also use the [dataset](https://huggingface.co/datasets/AndyChiang/dgen) we have already cleaned. |
|
|
| ## Training |
|
|
| We use a special way to fine-tune model, which is called **"Answer-Relating Fine-Tune"**. More details are in our paper. |
|
|
| ### Training hyperparameters |
|
|
| The following hyperparameters were used during training: |
|
|
| - Pre-train language model: [allenai/scibert_scivocab_uncased](https://huggingface.co/allenai/scibert_scivocab_uncased) |
| - Optimizer: adam |
| - Learning rate: 0.0001 |
| - Max length of input: 64 |
| - Batch size: 64 |
| - Epoch: 1 |
| - Device: NVIDIA® Tesla T4 in Google Colab |
|
|
| ## Testing |
|
|
| The evaluations of this model as a Candidate Set Generator in CDGP is as follows: |
|
|
| | P@1 | F1@3 | MRR | NDCG@10 | |
| | ----- | ----- | ----- | ------- | |
| | 13.13 | 12.23 | 25.12 | 34.17 | |
|
|
| ## Other models |
|
|
| ### Candidate Set Generator |
|
|
| | Models | CLOTH | DGen | |
| | ----------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | |
| | **BERT** | [cdgp-csg-bert-cloth](https://huggingface.co/AndyChiang/cdgp-csg-bert-cloth) | [cdgp-csg-bert-dgen](https://huggingface.co/AndyChiang/cdgp-csg-bert-dgen) | |
| | **SciBERT** | [cdgp-csg-scibert-cloth](https://huggingface.co/AndyChiang/cdgp-csg-scibert-cloth) | [*cdgp-csg-scibert-dgen*](https://huggingface.co/AndyChiang/cdgp-csg-scibert-dgen) | |
| | **RoBERTa** | [cdgp-csg-roberta-cloth](https://huggingface.co/AndyChiang/cdgp-csg-roberta-cloth) | [cdgp-csg-roberta-dgen](https://huggingface.co/AndyChiang/cdgp-csg-roberta-dgen) | |
| | **BART** | [cdgp-csg-bart-cloth](https://huggingface.co/AndyChiang/cdgp-csg-bart-cloth) | [cdgp-csg-bart-dgen](https://huggingface.co/AndyChiang/cdgp-csg-bart-dgen) | |
|
|
| ### Distractor Selector |
|
|
| **fastText**: [cdgp-ds-fasttext](https://huggingface.co/AndyChiang/cdgp-ds-fasttext) |
|
|
|
|
| ## Citation |
|
|
| None |
