Text Generation
Safetensors
English
beam
mixture-of-experts
reasoning
code
tool-use
conversational
Eval Results
Instructions to use reflection/Beam-1.0-501B-A23B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Inference
- HuggingChat
- Notebooks
- Google Colab
- Kaggle
Tighten eval guide bullets and remove repeated information
Browse files
README.md
CHANGED
|
@@ -477,15 +477,9 @@ These models were evaluated against a large collection of different datasets and
|
|
| 477 |
### Connect to Beam
|
| 478 |
|
| 479 |
- Create a Reflection API key at [platform.reflection.ai](https://platform.reflection.ai). Connect your Chat Completions client using the [API quickstart](https://developers.reflection.ai/quickstart).
|
| 480 |
-
|
| 481 |
-
- For further documentation: [`https://developers.reflection.ai`](https://developers.reflection.ai).
|
| 482 |
-
|
| 483 |
- To self-host Beam, follow the [deployment guide](https://developers.reflection.ai/deployment-overview) in the developer docs.
|
| 484 |
-
|
| 485 |
- Check that your outgoing request contains `"model": "Beam-1.0-501B-A23B"`.
|
| 486 |
-
|
| 487 |
- Before a formal eval, ask your Reflection contact to increase your organization's quota and rate limits.
|
| 488 |
-
|
| 489 |
- Include your registered email and organization, eval workload, expected token usage, request rate, duration and any zero data retention requirement.
|
| 490 |
- Check your limits in [Billing](https://platform.reflection.ai/billing) and confirm the increase before starting. Ask your contact to confirm daily allowance, concurrency and the per-request output cap.
|
| 491 |
|
|
@@ -503,49 +497,35 @@ These models were evaluated against a large collection of different datasets and
|
|
| 503 |
}
|
| 504 |
```
|
| 505 |
|
| 506 |
-
- Even at `max`, Beam generates fewer tokens than other models at `medium` or `high`.
|
| 507 |
-
|
| 508 |
- Suggested output budgets per API call, including reasoning:
|
| 509 |
-
|
| 510 |
- `32768`: start here for general evals.
|
| 511 |
- `65536`: start here for SWE-bench. Raise to `131072` if `finish_reason` is `length`.
|
| 512 |
- `16384`: use for short tasks only after checking that answers finish without being cut off. Keep `reasoning_effort` at `max`.
|
| 513 |
- Stay within Beam's `131072`-token output limit. Input plus output budget must fit the context window; `null` uses the configured default.
|
| 514 |
-
|
| 515 |
- Save the model ID, prompts, settings and test software version. If hosting Beam yourself, also save the weights, tokenizer, chat template and server settings.
|
| 516 |
|
| 517 |
### Check your endpoint with Beam Verifier
|
| 518 |
|
| 519 |
- Use it for self-hosted or third-party Beam endpoints to catch deployment issues before the full eval.
|
| 520 |
-
|
| 521 |
- `beam-verifier verify` checks API behavior, including tool calls, streaming and multi-turn conversations.
|
| 522 |
-
|
| 523 |
- `beam-verifier eval` runs GPQA-Diamond and AIME. Use its benchmark presets when comparing with reference scores.
|
| 524 |
-
|
| 525 |
- [Setup and usage](https://developers.reflection.ai/beam-verifier) · [GitHub](https://github.com/reflection-oss/beam-verifier)
|
| 526 |
|
| 527 |
### Agent and conversation setup
|
| 528 |
|
| 529 |
-
- For tool-based evals, start with [Mirror](https://developers.reflection.ai/mirror-overview).
|
| 530 |
-
|
| 531 |
-
- Beam has been validated with [Mirror](https://developers.reflection.ai/mirror-overview), Pi, OpenCode and Hermes.
|
| 532 |
-
|
| 533 |
- Your agent or API client must preserve `reasoning_content` and any `tool_calls` unchanged in prior assistant messages. It must return one `tool` message per call, with `tool_call_id` matching the call's `id`.
|
| 534 |
|
| 535 |
### Errors and debugging
|
| 536 |
|
| 537 |
- Log request IDs when debugging failures. If `finish_reason` is `length`, the output was cut off; track service failures separately from wrong answers.
|
| 538 |
-
|
| 539 |
- Follow rate limits and wait as directed by `Retry-After`. Do not retry when `x-should-retry` is `false`.
|
| 540 |
-
|
| 541 |
- For debugging, share the failing request, settings and request ID with Reflection.
|
| 542 |
|
| 543 |
### Example: [SWE-bench Verified](https://www.swebench.com/SWE-bench/guides/datasets/)
|
| 544 |
|
| 545 |
- Run [Mirror](https://developers.reflection.ai/mirror-overview) on each issue from its original repository commit, using the `bash`-only eval setup. Set `max_completion_tokens` to `65536`. Keep reference fixes and grading tests hidden.
|
| 546 |
-
|
| 547 |
- Save the generated patches in `predictions.jsonl` using the [official prediction format](https://www.swebench.com/SWE-bench/guides/evaluation/).
|
| 548 |
-
|
| 549 |
- On the evaluation machine, start [Docker Engine](https://www.swebench.com/SWE-bench/guides/docker_setup/) (Linux) or [Docker Desktop](https://www.swebench.com/SWE-bench/guides/docker_setup/) (macOS/Windows). With `swebench` installed, run the command below; the harness creates the task containers.
|
| 550 |
|
| 551 |
```bash
|
|
@@ -557,17 +537,13 @@ python -m swebench.harness.run_evaluation \
|
|
| 557 |
```
|
| 558 |
|
| 559 |
- Report the percentage of issues fixed with one attempt per issue. Include failed or missing runs.
|
| 560 |
-
|
| 561 |
- Use a new `run_id` whenever the patches change.
|
| 562 |
|
| 563 |
### Developer references
|
| 564 |
|
| 565 |
- [API quickstart](https://developers.reflection.ai/quickstart) and [models](https://developers.reflection.ai/models)
|
| 566 |
-
|
| 567 |
- [Generation settings](https://developers.reflection.ai/text-generation) and [reasoning](https://developers.reflection.ai/reasoning)
|
| 568 |
-
|
| 569 |
- [Tool calling](https://developers.reflection.ai/tool-calling)
|
| 570 |
-
|
| 571 |
- [Rate limits](https://developers.reflection.ai/rate-limits) and [errors](https://developers.reflection.ai/errors)
|
| 572 |
|
| 573 |
## Model Data
|
|
|
|
| 477 |
### Connect to Beam
|
| 478 |
|
| 479 |
- Create a Reflection API key at [platform.reflection.ai](https://platform.reflection.ai). Connect your Chat Completions client using the [API quickstart](https://developers.reflection.ai/quickstart).
|
|
|
|
|
|
|
|
|
|
| 480 |
- To self-host Beam, follow the [deployment guide](https://developers.reflection.ai/deployment-overview) in the developer docs.
|
|
|
|
| 481 |
- Check that your outgoing request contains `"model": "Beam-1.0-501B-A23B"`.
|
|
|
|
| 482 |
- Before a formal eval, ask your Reflection contact to increase your organization's quota and rate limits.
|
|
|
|
| 483 |
- Include your registered email and organization, eval workload, expected token usage, request rate, duration and any zero data retention requirement.
|
| 484 |
- Check your limits in [Billing](https://platform.reflection.ai/billing) and confirm the increase before starting. Ask your contact to confirm daily allowance, concurrency and the per-request output cap.
|
| 485 |
|
|
|
|
| 497 |
}
|
| 498 |
```
|
| 499 |
|
|
|
|
|
|
|
| 500 |
- Suggested output budgets per API call, including reasoning:
|
|
|
|
| 501 |
- `32768`: start here for general evals.
|
| 502 |
- `65536`: start here for SWE-bench. Raise to `131072` if `finish_reason` is `length`.
|
| 503 |
- `16384`: use for short tasks only after checking that answers finish without being cut off. Keep `reasoning_effort` at `max`.
|
| 504 |
- Stay within Beam's `131072`-token output limit. Input plus output budget must fit the context window; `null` uses the configured default.
|
|
|
|
| 505 |
- Save the model ID, prompts, settings and test software version. If hosting Beam yourself, also save the weights, tokenizer, chat template and server settings.
|
| 506 |
|
| 507 |
### Check your endpoint with Beam Verifier
|
| 508 |
|
| 509 |
- Use it for self-hosted or third-party Beam endpoints to catch deployment issues before the full eval.
|
|
|
|
| 510 |
- `beam-verifier verify` checks API behavior, including tool calls, streaming and multi-turn conversations.
|
|
|
|
| 511 |
- `beam-verifier eval` runs GPQA-Diamond and AIME. Use its benchmark presets when comparing with reference scores.
|
|
|
|
| 512 |
- [Setup and usage](https://developers.reflection.ai/beam-verifier) · [GitHub](https://github.com/reflection-oss/beam-verifier)
|
| 513 |
|
| 514 |
### Agent and conversation setup
|
| 515 |
|
| 516 |
+
- For tool-based evals, start with [Mirror](https://developers.reflection.ai/mirror-overview). Beam has been validated with Mirror, Pi, OpenCode and Hermes.
|
|
|
|
|
|
|
|
|
|
| 517 |
- Your agent or API client must preserve `reasoning_content` and any `tool_calls` unchanged in prior assistant messages. It must return one `tool` message per call, with `tool_call_id` matching the call's `id`.
|
| 518 |
|
| 519 |
### Errors and debugging
|
| 520 |
|
| 521 |
- Log request IDs when debugging failures. If `finish_reason` is `length`, the output was cut off; track service failures separately from wrong answers.
|
|
|
|
| 522 |
- Follow rate limits and wait as directed by `Retry-After`. Do not retry when `x-should-retry` is `false`.
|
|
|
|
| 523 |
- For debugging, share the failing request, settings and request ID with Reflection.
|
| 524 |
|
| 525 |
### Example: [SWE-bench Verified](https://www.swebench.com/SWE-bench/guides/datasets/)
|
| 526 |
|
| 527 |
- Run [Mirror](https://developers.reflection.ai/mirror-overview) on each issue from its original repository commit, using the `bash`-only eval setup. Set `max_completion_tokens` to `65536`. Keep reference fixes and grading tests hidden.
|
|
|
|
| 528 |
- Save the generated patches in `predictions.jsonl` using the [official prediction format](https://www.swebench.com/SWE-bench/guides/evaluation/).
|
|
|
|
| 529 |
- On the evaluation machine, start [Docker Engine](https://www.swebench.com/SWE-bench/guides/docker_setup/) (Linux) or [Docker Desktop](https://www.swebench.com/SWE-bench/guides/docker_setup/) (macOS/Windows). With `swebench` installed, run the command below; the harness creates the task containers.
|
| 530 |
|
| 531 |
```bash
|
|
|
|
| 537 |
```
|
| 538 |
|
| 539 |
- Report the percentage of issues fixed with one attempt per issue. Include failed or missing runs.
|
|
|
|
| 540 |
- Use a new `run_id` whenever the patches change.
|
| 541 |
|
| 542 |
### Developer references
|
| 543 |
|
| 544 |
- [API quickstart](https://developers.reflection.ai/quickstart) and [models](https://developers.reflection.ai/models)
|
|
|
|
| 545 |
- [Generation settings](https://developers.reflection.ai/text-generation) and [reasoning](https://developers.reflection.ai/reasoning)
|
|
|
|
| 546 |
- [Tool calling](https://developers.reflection.ai/tool-calling)
|
|
|
|
| 547 |
- [Rate limits](https://developers.reflection.ai/rate-limits) and [errors](https://developers.reflection.ai/errors)
|
| 548 |
|
| 549 |
## Model Data
|