Text Generation
Safetensors
English
beam
mixture-of-experts
reasoning
code
tool-use
conversational
Eval Results
Instructions to use reflection/Beam-1.0-501B-A23B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Inference
- HuggingChat
- Notebooks
- Google Colab
- Kaggle
Add a Beam evaluation guide and link it from the model card (#20)
Browse files- Add Beam evaluation guide to the model card (d76effa94ccbf143f479b4ba99305000d5a7bc37)
- Link self-hosting instructions from the connection section (c020e0c0543f64f24395f05704f6c0275df8e5c3)
- Tighten eval guide bullets and remove repeated information (e5b654f87fada621060b7c26260fe17e1085e9d9)
- Move the eval guide into EVALUATION.md and link from the model card (5ad00ce7783ec3c7770d39b26047043d7e0eaf3f)
- Make the evaluation guide self-contained (a9d6c2abe588a35720440cfb7c738902d24114d0)
Co-authored-by: Tekur <cptekur@users.noreply.huggingface.co>
- EVALUATION.md +77 -0
- README.md +4 -0
EVALUATION.md
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Evaluating Beam
|
| 2 |
+
|
| 3 |
+
- Beam is Reflection's open-weight text model for coding, reasoning and tool use, with a `1M`-token context window.
|
| 4 |
+
- Self-hosted weights are available in `BF16`, `FP8` and `NVFP4`. Compare against the [reference scores](https://huggingface.co/reflection/Beam-1.0-501B-A23B#benchmark-results) for the precision you test.
|
| 5 |
+
|
| 6 |
+
## Connect to Beam
|
| 7 |
+
|
| 8 |
+
- Create a Reflection API key at [platform.reflection.ai](https://platform.reflection.ai). Set your Chat Completions base URL to `https://api.reflection.ai/openai/v1` ([API quickstart](https://developers.reflection.ai/quickstart)).
|
| 9 |
+
- To self-host Beam, follow the [deployment guide](https://developers.reflection.ai/deployment-overview) in the developer docs.
|
| 10 |
+
- Check that your outgoing request contains `"model": "Beam-1.0-501B-A23B"`.
|
| 11 |
+
- Before a formal eval, ask your Reflection contact to increase your organization's quota and rate limits.
|
| 12 |
+
- Include your registered email and organization, eval workload, expected token usage, request rate, duration and any zero data retention requirement.
|
| 13 |
+
- Check your limits in [Billing](https://platform.reflection.ai/billing) and confirm the increase before starting. Ask your contact to confirm daily allowance, concurrency and the per-request output cap.
|
| 14 |
+
|
| 15 |
+
## Set up the model
|
| 16 |
+
|
| 17 |
+
- Use `max` for `reasoning_effort`. Beam reaches comparable solution quality to leading open models using approximately `3x` fewer reasoning tokens.
|
| 18 |
+
- Start with these model settings:
|
| 19 |
+
|
| 20 |
+
```json
|
| 21 |
+
{
|
| 22 |
+
"model": "Beam-1.0-501B-A23B",
|
| 23 |
+
"reasoning_effort": "max",
|
| 24 |
+
"temperature": 0.7,
|
| 25 |
+
"top_p": 0.99,
|
| 26 |
+
"max_completion_tokens": 32768
|
| 27 |
+
}
|
| 28 |
+
```
|
| 29 |
+
|
| 30 |
+
- Suggested output budgets per API call, including reasoning:
|
| 31 |
+
- `32768`: start here for general evals.
|
| 32 |
+
- `65536`: start here for SWE-bench. Raise to `131072` if `finish_reason` is `length`.
|
| 33 |
+
- `16384`: use for short tasks only after checking that answers finish without being cut off. Keep `reasoning_effort` at `max`.
|
| 34 |
+
- Stay within Beam's `131072`-token output limit. Input plus output budget must fit the context window; `null` uses the configured default.
|
| 35 |
+
- Save the model ID, prompts, settings and test software version. If hosting Beam yourself, also save the weights, tokenizer, chat template and server settings.
|
| 36 |
+
|
| 37 |
+
## Check your endpoint with Beam Verifier
|
| 38 |
+
|
| 39 |
+
- Use it for self-hosted or third-party Beam endpoints to catch deployment issues before the full eval.
|
| 40 |
+
- `beam-verifier verify` checks API behavior, including tool calls, streaming and multi-turn conversations.
|
| 41 |
+
- `beam-verifier eval` runs GPQA-Diamond and AIME. Use its benchmark presets when comparing with reference scores.
|
| 42 |
+
- [Setup and usage](https://developers.reflection.ai/beam-verifier) · [GitHub](https://github.com/reflection-oss/beam-verifier)
|
| 43 |
+
|
| 44 |
+
## Agent and conversation setup
|
| 45 |
+
|
| 46 |
+
- For tool-based evals, start with [Mirror](https://developers.reflection.ai/mirror-overview). Beam has been validated with Mirror, Pi, OpenCode and Hermes.
|
| 47 |
+
- Your agent or API client must preserve `reasoning_content` and any `tool_calls` unchanged in prior assistant messages. It must return one `tool` message per call, with `tool_call_id` matching the call's `id`.
|
| 48 |
+
|
| 49 |
+
## Errors and debugging
|
| 50 |
+
|
| 51 |
+
- Log request IDs when debugging failures. If `finish_reason` is `length`, the output was cut off; track service failures separately from wrong answers.
|
| 52 |
+
- Follow rate limits and wait as directed by `Retry-After`. Do not retry when `x-should-retry` is `false`.
|
| 53 |
+
- For debugging, share the failing request, settings and request ID with Reflection.
|
| 54 |
+
|
| 55 |
+
## Example: [SWE-bench Verified](https://www.swebench.com/SWE-bench/guides/datasets/)
|
| 56 |
+
|
| 57 |
+
- Run [Mirror](https://developers.reflection.ai/mirror-overview) on each issue from its original repository commit, using the `bash`-only eval setup. Set `max_completion_tokens` to `65536`. Keep reference fixes and grading tests hidden.
|
| 58 |
+
- Save the generated patches in `predictions.jsonl` using the [official prediction format](https://www.swebench.com/SWE-bench/guides/evaluation/).
|
| 59 |
+
- On the evaluation machine, start [Docker Engine](https://www.swebench.com/SWE-bench/guides/docker_setup/) (Linux) or [Docker Desktop](https://www.swebench.com/SWE-bench/guides/docker_setup/) (macOS/Windows). With `swebench` installed, run the command below; the harness creates the task containers.
|
| 60 |
+
|
| 61 |
+
```bash
|
| 62 |
+
python -m swebench.harness.run_evaluation \
|
| 63 |
+
--dataset_name SWE-bench/SWE-bench_Verified \
|
| 64 |
+
--predictions_path predictions.jsonl \
|
| 65 |
+
--max_workers 4 \
|
| 66 |
+
--run_id beam-verified-001
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
- Report the percentage of issues fixed with one attempt per issue. Include failed or missing runs.
|
| 70 |
+
- Use a new `run_id` whenever the patches change.
|
| 71 |
+
|
| 72 |
+
## Developer references
|
| 73 |
+
|
| 74 |
+
- [API quickstart](https://developers.reflection.ai/quickstart) and [models](https://developers.reflection.ai/models)
|
| 75 |
+
- [Generation settings](https://developers.reflection.ai/text-generation) and [reasoning](https://developers.reflection.ai/reasoning)
|
| 76 |
+
- [Tool calling](https://developers.reflection.ai/tool-calling)
|
| 77 |
+
- [Rate limits](https://developers.reflection.ai/rate-limits) and [errors](https://developers.reflection.ai/errors)
|
README.md
CHANGED
|
@@ -472,6 +472,10 @@ These models were evaluated against a large collection of different datasets and
|
|
| 472 |
</table>
|
| 473 |
</div>
|
| 474 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 475 |
## Model Data
|
| 476 |
|
| 477 |
Beam is pretrained on a diverse variety of textual data from publicly available sources and data obtained through commercial partner agreements. The corpus includes general web content, source code, technical documentation, reference materials, and educational and scientific documents. Mid-training uses data derived from curated real-world examples to develop reasoning, tool-use, and long-context capabilities, including structured code repositories, long-horizon tasks, and long-form documents. Reinforcement learning uses environments spanning coding, agentic tasks, and STEM, built primarily through synthetic pipelines and supplemented by proprietary vendor data and open-source sources.
|
|
|
|
| 472 |
</table>
|
| 473 |
</div>
|
| 474 |
|
| 475 |
+
## Evaluating Beam
|
| 476 |
+
|
| 477 |
+
See the [evaluation guide](./EVALUATION.md) for recommended settings, Beam Verifier and a SWE-bench Verified example.
|
| 478 |
+
|
| 479 |
## Model Data
|
| 480 |
|
| 481 |
Beam is pretrained on a diverse variety of textual data from publicly available sources and data obtained through commercial partner agreements. The corpus includes general web content, source code, technical documentation, reference materials, and educational and scientific documents. Mid-training uses data derived from curated real-world examples to develop reasoning, tool-use, and long-context capabilities, including structured code repositories, long-horizon tasks, and long-form documents. Reinforcement learning uses environments spanning coding, agentic tasks, and STEM, built primarily through synthetic pipelines and supplemented by proprietary vendor data and open-source sources.
|