---
license: mit
datasets:
- liuganghuggingface/demodiff_downstream
tags:
- chemistry
- biology
---

### Model Configuration

| Parameter | Value | Description |
|------------|--------|-------------|
| **context_length** | 150 | Maximum sequence length for the input context. |
| **depth** | 24 | Number of transformer layers. |
| **diffusion_steps** | 500 | Number of diffusion steps during training. |
| **hidden_size** | 1280 | Hidden dimension size in the transformer. |
| **mlp_ratio** | 4 | Expansion ratio in the MLP block. |
| **num_heads** | 16 | Number of attention heads. |
| **task_name** | `pretrain` | Task type for model training. |
| **tokenizer_name** | `pretrain` | Tokenizer used for model input. |
| **vocab_ring_len** | 300 | Length of the circular vocabulary window. |
| **vocab_size** | 3000 | Total vocabulary size. |