Update README with model loading code
Browse files
README.md
CHANGED
|
@@ -13,12 +13,30 @@ Part of the [LAPVQA collection](https://huggingface.co/collections/dmusingu/lapv
|
|
| 13 |
|
| 14 |
## Description
|
| 15 |
|
| 16 |
-
RRG decoder trained on
|
| 17 |
([`lapvqa-pretrain-captioning`](https://huggingface.co/dmusingu/lapvqa-pretrain-captioning)).
|
| 18 |
-
|
| 19 |
|
| 20 |
-
##
|
| 21 |
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
## Description
|
| 15 |
|
| 16 |
+
RRG decoder trained on the frozen **LAPVQA captioning-pretrained encoder**
|
| 17 |
([`lapvqa-pretrain-captioning`](https://huggingface.co/dmusingu/lapvqa-pretrain-captioning)).
|
| 18 |
+
Checkpoint format: `{state_dict, vis_dim, d_model, num_layers, nhead, encoder, epoch, val_bleu4}`.
|
| 19 |
|
| 20 |
+
## Loading
|
| 21 |
|
| 22 |
+
```python
|
| 23 |
+
import torch
|
| 24 |
+
import tiktoken
|
| 25 |
+
from lapvqa.rrg.heads import ReportGenerationHead
|
| 26 |
+
|
| 27 |
+
ckpt = torch.load("pretrain-captioning.pt", map_location="cpu")
|
| 28 |
+
head = ReportGenerationHead(
|
| 29 |
+
vis_dim = ckpt["vis_dim"],
|
| 30 |
+
d_model = ckpt["d_model"],
|
| 31 |
+
num_layers = ckpt["num_layers"],
|
| 32 |
+
nhead = ckpt["nhead"],
|
| 33 |
+
)
|
| 34 |
+
head.load_state_dict(ckpt["state_dict"])
|
| 35 |
+
head.eval()
|
| 36 |
+
|
| 37 |
+
enc = tiktoken.get_encoding("gpt2")
|
| 38 |
+
bos_id = eos_id = enc.eot_token
|
| 39 |
+
# pair with encoder_final.pt from lapvqa-pretrain-captioning
|
| 40 |
+
token_ids = head.generate(vis_tokens, bos_id=bos_id, eos_id=eos_id)
|
| 41 |
+
reports = [enc.decode(ids) for ids in token_ids]
|
| 42 |
+
```
|