cptekur commited on
Commit
e5b654f
·
verified ·
1 Parent(s): c020e0c

Tighten eval guide bullets and remove repeated information

Browse files
Files changed (1) hide show
  1. README.md +1 -25
README.md CHANGED
@@ -477,15 +477,9 @@ These models were evaluated against a large collection of different datasets and
477
  ### Connect to Beam
478
 
479
  - Create a Reflection API key at [platform.reflection.ai](https://platform.reflection.ai). Connect your Chat Completions client using the [API quickstart](https://developers.reflection.ai/quickstart).
480
-
481
- - For further documentation: [`https://developers.reflection.ai`](https://developers.reflection.ai).
482
-
483
  - To self-host Beam, follow the [deployment guide](https://developers.reflection.ai/deployment-overview) in the developer docs.
484
-
485
  - Check that your outgoing request contains `"model": "Beam-1.0-501B-A23B"`.
486
-
487
  - Before a formal eval, ask your Reflection contact to increase your organization's quota and rate limits.
488
-
489
  - Include your registered email and organization, eval workload, expected token usage, request rate, duration and any zero data retention requirement.
490
  - Check your limits in [Billing](https://platform.reflection.ai/billing) and confirm the increase before starting. Ask your contact to confirm daily allowance, concurrency and the per-request output cap.
491
 
@@ -503,49 +497,35 @@ These models were evaluated against a large collection of different datasets and
503
  }
504
  ```
505
 
506
- - Even at `max`, Beam generates fewer tokens than other models at `medium` or `high`.
507
-
508
  - Suggested output budgets per API call, including reasoning:
509
-
510
  - `32768`: start here for general evals.
511
  - `65536`: start here for SWE-bench. Raise to `131072` if `finish_reason` is `length`.
512
  - `16384`: use for short tasks only after checking that answers finish without being cut off. Keep `reasoning_effort` at `max`.
513
  - Stay within Beam's `131072`-token output limit. Input plus output budget must fit the context window; `null` uses the configured default.
514
-
515
  - Save the model ID, prompts, settings and test software version. If hosting Beam yourself, also save the weights, tokenizer, chat template and server settings.
516
 
517
  ### Check your endpoint with Beam Verifier
518
 
519
  - Use it for self-hosted or third-party Beam endpoints to catch deployment issues before the full eval.
520
-
521
  - `beam-verifier verify` checks API behavior, including tool calls, streaming and multi-turn conversations.
522
-
523
  - `beam-verifier eval` runs GPQA-Diamond and AIME. Use its benchmark presets when comparing with reference scores.
524
-
525
  - [Setup and usage](https://developers.reflection.ai/beam-verifier) · [GitHub](https://github.com/reflection-oss/beam-verifier)
526
 
527
  ### Agent and conversation setup
528
 
529
- - For tool-based evals, start with [Mirror](https://developers.reflection.ai/mirror-overview).
530
-
531
- - Beam has been validated with [Mirror](https://developers.reflection.ai/mirror-overview), Pi, OpenCode and Hermes.
532
-
533
  - Your agent or API client must preserve `reasoning_content` and any `tool_calls` unchanged in prior assistant messages. It must return one `tool` message per call, with `tool_call_id` matching the call's `id`.
534
 
535
  ### Errors and debugging
536
 
537
  - Log request IDs when debugging failures. If `finish_reason` is `length`, the output was cut off; track service failures separately from wrong answers.
538
-
539
  - Follow rate limits and wait as directed by `Retry-After`. Do not retry when `x-should-retry` is `false`.
540
-
541
  - For debugging, share the failing request, settings and request ID with Reflection.
542
 
543
  ### Example: [SWE-bench Verified](https://www.swebench.com/SWE-bench/guides/datasets/)
544
 
545
  - Run [Mirror](https://developers.reflection.ai/mirror-overview) on each issue from its original repository commit, using the `bash`-only eval setup. Set `max_completion_tokens` to `65536`. Keep reference fixes and grading tests hidden.
546
-
547
  - Save the generated patches in `predictions.jsonl` using the [official prediction format](https://www.swebench.com/SWE-bench/guides/evaluation/).
548
-
549
  - On the evaluation machine, start [Docker Engine](https://www.swebench.com/SWE-bench/guides/docker_setup/) (Linux) or [Docker Desktop](https://www.swebench.com/SWE-bench/guides/docker_setup/) (macOS/Windows). With `swebench` installed, run the command below; the harness creates the task containers.
550
 
551
  ```bash
@@ -557,17 +537,13 @@ python -m swebench.harness.run_evaluation \
557
  ```
558
 
559
  - Report the percentage of issues fixed with one attempt per issue. Include failed or missing runs.
560
-
561
  - Use a new `run_id` whenever the patches change.
562
 
563
  ### Developer references
564
 
565
  - [API quickstart](https://developers.reflection.ai/quickstart) and [models](https://developers.reflection.ai/models)
566
-
567
  - [Generation settings](https://developers.reflection.ai/text-generation) and [reasoning](https://developers.reflection.ai/reasoning)
568
-
569
  - [Tool calling](https://developers.reflection.ai/tool-calling)
570
-
571
  - [Rate limits](https://developers.reflection.ai/rate-limits) and [errors](https://developers.reflection.ai/errors)
572
 
573
  ## Model Data
 
477
  ### Connect to Beam
478
 
479
  - Create a Reflection API key at [platform.reflection.ai](https://platform.reflection.ai). Connect your Chat Completions client using the [API quickstart](https://developers.reflection.ai/quickstart).
 
 
 
480
  - To self-host Beam, follow the [deployment guide](https://developers.reflection.ai/deployment-overview) in the developer docs.
 
481
  - Check that your outgoing request contains `"model": "Beam-1.0-501B-A23B"`.
 
482
  - Before a formal eval, ask your Reflection contact to increase your organization's quota and rate limits.
 
483
  - Include your registered email and organization, eval workload, expected token usage, request rate, duration and any zero data retention requirement.
484
  - Check your limits in [Billing](https://platform.reflection.ai/billing) and confirm the increase before starting. Ask your contact to confirm daily allowance, concurrency and the per-request output cap.
485
 
 
497
  }
498
  ```
499
 
 
 
500
  - Suggested output budgets per API call, including reasoning:
 
501
  - `32768`: start here for general evals.
502
  - `65536`: start here for SWE-bench. Raise to `131072` if `finish_reason` is `length`.
503
  - `16384`: use for short tasks only after checking that answers finish without being cut off. Keep `reasoning_effort` at `max`.
504
  - Stay within Beam's `131072`-token output limit. Input plus output budget must fit the context window; `null` uses the configured default.
 
505
  - Save the model ID, prompts, settings and test software version. If hosting Beam yourself, also save the weights, tokenizer, chat template and server settings.
506
 
507
  ### Check your endpoint with Beam Verifier
508
 
509
  - Use it for self-hosted or third-party Beam endpoints to catch deployment issues before the full eval.
 
510
  - `beam-verifier verify` checks API behavior, including tool calls, streaming and multi-turn conversations.
 
511
  - `beam-verifier eval` runs GPQA-Diamond and AIME. Use its benchmark presets when comparing with reference scores.
 
512
  - [Setup and usage](https://developers.reflection.ai/beam-verifier) · [GitHub](https://github.com/reflection-oss/beam-verifier)
513
 
514
  ### Agent and conversation setup
515
 
516
+ - For tool-based evals, start with [Mirror](https://developers.reflection.ai/mirror-overview). Beam has been validated with Mirror, Pi, OpenCode and Hermes.
 
 
 
517
  - Your agent or API client must preserve `reasoning_content` and any `tool_calls` unchanged in prior assistant messages. It must return one `tool` message per call, with `tool_call_id` matching the call's `id`.
518
 
519
  ### Errors and debugging
520
 
521
  - Log request IDs when debugging failures. If `finish_reason` is `length`, the output was cut off; track service failures separately from wrong answers.
 
522
  - Follow rate limits and wait as directed by `Retry-After`. Do not retry when `x-should-retry` is `false`.
 
523
  - For debugging, share the failing request, settings and request ID with Reflection.
524
 
525
  ### Example: [SWE-bench Verified](https://www.swebench.com/SWE-bench/guides/datasets/)
526
 
527
  - Run [Mirror](https://developers.reflection.ai/mirror-overview) on each issue from its original repository commit, using the `bash`-only eval setup. Set `max_completion_tokens` to `65536`. Keep reference fixes and grading tests hidden.
 
528
  - Save the generated patches in `predictions.jsonl` using the [official prediction format](https://www.swebench.com/SWE-bench/guides/evaluation/).
 
529
  - On the evaluation machine, start [Docker Engine](https://www.swebench.com/SWE-bench/guides/docker_setup/) (Linux) or [Docker Desktop](https://www.swebench.com/SWE-bench/guides/docker_setup/) (macOS/Windows). With `swebench` installed, run the command below; the harness creates the task containers.
530
 
531
  ```bash
 
537
  ```
538
 
539
  - Report the percentage of issues fixed with one attempt per issue. Include failed or missing runs.
 
540
  - Use a new `run_id` whenever the patches change.
541
 
542
  ### Developer references
543
 
544
  - [API quickstart](https://developers.reflection.ai/quickstart) and [models](https://developers.reflection.ai/models)
 
545
  - [Generation settings](https://developers.reflection.ai/text-generation) and [reasoning](https://developers.reflection.ai/reasoning)
 
546
  - [Tool calling](https://developers.reflection.ai/tool-calling)
 
547
  - [Rate limits](https://developers.reflection.ai/rate-limits) and [errors](https://developers.reflection.ai/errors)
548
 
549
  ## Model Data