Decision Index 0.2.1 with adaptive thinking on all ten Knowledge & Reasoning benchmarks: 62.48 (MMLU-Pro 84.6, BBH 92.0); latency with and without speculative decoding

README.md

@@ -28,19 +28,19 @@ model-index:
     results:
       - task:
           type: text-classification
-          name: Jev Decision Index 0.2.1 with adaptive thinking (own scoring; see the model card for the run composition)
+          name: Jev Decision Index 0.2.1 with adaptive thinking (own scoring; Knowledge & Reasoning adaptive, other areas System 1)
         dataset:
           type: decision-index
           name: Decision Index suite 0.2 (edition 0.2.1)
         metrics:
           - type: decision_index
             name: Decision Index (balanced skill)
-            value: 61.15
+            value: 62.48
 ---
 
 # autotrust/GEV-26B-Decide
 
-### A decision model with adaptive thinking, on Gemma-4-26B-A4B-it (26 B parameters, ≈ 4 B active per token) — Decision Index 0.2.1: **61.15**
+### A decision model with adaptive thinking, on Gemma-4-26B-A4B-it (26 B parameters, ≈ 4 B active per token) — Decision Index 0.2.1: **62.48**
 
 **GEV-26B-Decide answers typed questions with a calibrated probability for every option, and thinks only when it needs
 to.** System 1 decides in one forward pass (about 45 ms). When its leading option is uncertain, System 2 (the same
@@ -89,8 +89,8 @@ the always-think gain while thinking on 48 % of the questions. Thinking length:
 
 ### Decision Index, Knowledge & Reasoning
 
-The area's benchmarks run through the server (`POST /v1/decide`, `thinking: "auto"`, threshold 0.8, budget 8,192 thinking
-tokens), scored with the Decision Index kit. Eight of the ten benchmarks were run to completion:
+All ten benchmarks of the area (33,856 scored requests) ran through the server (`POST /v1/decide`, `thinking: "auto"`,
+threshold 0.8, budget 8,192 thinking tokens) and were scored with the Decision Index kit:
 
 | benchmark | chance | System 1 | **adaptive** | DI skill, System 1 → adaptive | questions that thought |
 |---|---:|---:|---:|---:|---:|
@@ -102,14 +102,13 @@ tokens), scored with the Decision Index kit. Eight of the ten benchmarks were ru
 | MuSR | 37.1 | 67.3 | 67.6 | 0.480 → 0.484 | 51 % |
 | ChessBench | 8.2 | 23.7 | 23.6 | 0.169 → 0.168 | 88 % |
 | HLE | 16.4 | 8.4 | 17.8 | 0.000 → 0.016 | 88 % |
-| MMLU-Pro (first 1,611 of 12,032 questions) | 11.1 | 59.2 | **80.1** | counted with System 1 | 68 % |
-| BBH | 31.0 | 75.2 | not run | counted with System 1 | — |
+| MMLU-Pro | 11.1 | 65.0 | **84.6** | 0.607 → **0.827** | 78 % |
+| BBH | 31.0 | 75.0 | **92.0** | 0.638 → **0.884** | 60 % |
 
 Accuracy in %; "chance" is the kit's random baseline, and the Decision Index skill rescales accuracy so that chance is
-0 (below chance counts as 0). The Knowledge & Reasoning area skill rises from 0.430 to **0.551** with eight of ten
-benchmarks adaptive (MMLU-Pro and BBH counted with System 1).
+0 (below chance counts as 0). The Knowledge & Reasoning area skill rises from 0.429 to **0.602**.
 
-* **Real gains** on science, code and causal reasoning: GPQA Diamond, CRUXEval and CLadder.
+* **Real gains** on science, code, causal and multi-step reasoning: GPQA Diamond, CRUXEval, CLadder, MMLU-Pro and BBH.
 * **HLE only returns to chance.** System 1 is below chance (8.4 % against 16.4 %), and the questions where it is confident
   enough not to think are almost all wrong (1.6 %). Thinking brings HLE to 17.8 %, the level of guessing, so its skill
   stays near 0.
@@ -120,27 +119,31 @@ benchmarks adaptive (MMLU-Pro and BBH counted with System 1).
 
 ### Latency
 
-Thinking costs time. On the eight completed benchmarks, 59 % of the requests thought at least once. Thinking length:
-median 8,192 tokens (the budget) on GPQA, HLE and ChessBench, about 6,400 on CLadder, 900 on GSM8K. Estimated
-single-request latency on one idle B200 (System 1 ≈ 45 ms, thinking ≈ 250 tokens per second): about 0.05 s without
-thinking and up to about 33 s with an 8,192-token thought; median 15 s over these benchmarks. Lower `threshold` or
-`think_budget` to trade accuracy for speed; `thinking: "off"` keeps every decision in one pass.
+Thinking costs time. Over the area's ten benchmarks, 66 % of the requests thought at least once. Thinking length:
+median 8,192 tokens (the budget) on GPQA, HLE and ChessBench, about 5,400 on MMLU-Pro, 3,300 on BBH and 900 on GSM8K.
+Estimated single-request latency on one idle B200 (System 1 ≈ 45 ms, thinking ≈ 250 tokens per second): about 0.05 s
+without thinking and up to about 33 s with an 8,192-token thought; median 13.4 s over the area (90th percentile 33 s).
+With speculative decoding (below) thinking runs at about 438 tokens per second: median 7.7 s, 90th percentile 19 s.
+Lower `threshold` or `think_budget` to trade accuracy for speed; `thinking: "off"` keeps every decision in one pass.
+
+MMLU-Pro and BBH (except six questions) ran with speculative decoding, which does not change the output distribution.
+Six BBH questions with 18 options hit a vLLM error in the speculative-decoding read-out path and were answered by the same
+server without speculative decoding.
 
 ## Decision Index
 
 | | Decision Index 0.2.1 (balanced skill) | balanced raw | breadth skill |
 |---|---:|---:|---:|
-| **autotrust/GEV-26B-Decide, adaptive thinking** | **61.15** | 69.62 | 60.60 |
+| **autotrust/GEV-26B-Decide, adaptive thinking** | **62.48** | 70.66 | 62.00 |
 | TypeSafe Jev 1.13 (board) | 57.91 | — | — |
 
 | area (skill) | Knowledge & Reasoning | Language | Retrieval & Classification | Tools & Automation | Arts & Taste |
 |---|---:|---:|---:|---:|---:|
-| GEV-26B-Decide, adaptive thinking | **0.551** | 0.636 | 0.679 | 0.697 | 0.415 |
+| GEV-26B-Decide, adaptive thinking | **0.602** | 0.636 | 0.679 | 0.697 | 0.415 |
 
 How the score was computed (our scoring with the kit's `score --edition 0.2.1`, not a board entry):
 
-* **Knowledge & Reasoning:** the eight benchmarks above that were run to completion with adaptive thinking; MMLU-Pro and
-  BBH with System 1.
+* **Knowledge & Reasoning:** all ten benchmarks with adaptive thinking (table above).
 * **Other four areas:** the complete System 1 run of these weights (all 150,759 requests, 0 errors), results in
   [`autotrust/jev-decision-index-results`](https://huggingface.co/datasets/autotrust/jev-decision-index-results)
   (`runs/jev-gemma4-26b-a4b`, the weights' previous name). Adaptive thinking was not run on them.
@@ -265,6 +268,14 @@ requests.post("http://localhost:8000/v1/chat/completions", json={
 * Adaptive thinking adds nothing when it does not think, and about 1 second per 250 thinking tokens when it does
   (see [Latency](#latency)).
 
+**Faster thinking with speculative decoding.** Adding
+`--speculative-config '{"model": "google/gemma-4-26B-A4B-it-assistant", "num_speculative_tokens": 4}'` to `serve.sh`
+(Google's 0.9 GB draft model for this backbone) speeds up System 2 about 1.8–1.9× on one B200: 247 → 438 tokens per
+second for a single request, 7,072 → 13,505 tokens per second at 128 concurrent requests (mean acceptance length 3.5–3.7
+of 4). System 1 decisions are unchanged, but System 1 throughput at high concurrency drops (257 → 140 decisions per second
+with 64 clients), and in this vLLM build the System 2 answer read-out fails for questions with more than 16 options. Use it
+when you mostly think, and serve System 1-heavy traffic without it.
+
 If you call `/v1/completions` for System 1 yourself, pass `top_k: 0` and `top_p: 1.0`: the model's generation config sets
 `top_k=64` and `top_p=0.95`, which vLLM applies as request defaults and which would truncate the returned probabilities.
 
@@ -335,8 +346,8 @@ were chosen on data outside the Decision Index suite.
   chess or expert-level HLE questions, and can slightly lower accuracy on commonsense questions (CommonsenseQA 86.7 → 85.0).
 * Thinking is slow on hard inputs: on expert-level and chess questions it often uses the full 8,192-token budget, and
   the adaptive mode does not meet the Decision Index latency limit on the Knowledge & Reasoning benchmarks.
-* The Decision Index score above combines adaptive runs of eight Knowledge & Reasoning benchmarks with System 1 runs of
-  the rest; it is our own scoring, not a board result.
+* The Decision Index score above combines the adaptive run of the Knowledge & Reasoning area with System 1 runs of the
+  other four areas; it is our own scoring, not a board result.
 * Weaker than JEV-27B on long structured inputs (e.g. the Decision Index's Home appliance simulator and POP909).
 * HLE: below chance with System 1, like every open entry on the board.
 * Decisions over images are zero-shot; contexts above 128K tokens are untested.

reports/adaptive_latency_summary.json

@@ -1,83 +1,102 @@
 {
  "per_benchmark": {
+  "BBH fixed-option tasks": {
+   "requests": 5507,
+   "questions_thinking_pct": 59.7,
+   "think_tokens_median": 3255.0,
+   "budget_hit_pct": 8.2,
+   "est_median_s_no_mtp": 5.27,
+   "est_median_s_mtp": 3.05
+  },
+  "CLadder": {
+   "requests": 5000,
+   "questions_thinking_pct": 51.1,
+   "think_tokens_median": 6431.5,
+   "budget_hit_pct": 33.4,
+   "est_median_s_no_mtp": 6.66,
+   "est_median_s_mtp": 3.85
+  },
+  "CRUXEval": {
+   "requests": 570,
+   "questions_thinking_pct": 41.8,
+   "think_tokens_median": 7143.0,
+   "budget_hit_pct": 41.2,
+   "est_median_s_no_mtp": 0.04,
+   "est_median_s_mtp": 0.05
+  },
+  "ChessBench": {
+   "requests": 5000,
+   "questions_thinking_pct": 87.6,
+   "think_tokens_median": 8192.0,
+   "budget_hit_pct": 75.5,
+   "est_median_s_no_mtp": 32.86,
+   "est_median_s_mtp": 18.8
+  },
   "GPQA Diamond": {
-   "questions": 198,
-   "thinking_pct": 86.9,
+   "requests": 198,
+   "questions_thinking_pct": 86.9,
    "think_tokens_median": 8192.0,
    "budget_hit_pct": 65.7,
-   "est_latency_median_s": 32.86,
-   "est_latency_p90_s": 32.9
+   "est_median_s_no_mtp": 32.86,
+   "est_median_s_mtp": 18.8
   },
   "GSM8K": {
-   "questions": 2638,
-   "thinking_pct": 5.3,
+   "requests": 2638,
+   "questions_thinking_pct": 5.3,
    "think_tokens_median": 888.5,
    "budget_hit_pct": 7.9,
-   "est_latency_median_s": 0.04,
-   "est_latency_p90_s": 0.0
+   "est_median_s_no_mtp": 0.04,
+   "est_median_s_mtp": 0.05
   },
-  "ChessBench": {
-   "questions": 5000,
-   "thinking_pct": 87.6,
+  "HLE": {
+   "requests": 513,
+   "questions_thinking_pct": 87.9,
    "think_tokens_median": 8192.0,
-   "budget_hit_pct": 75.5,
-   "est_latency_median_s": 32.86,
-   "est_latency_p90_s": 32.9
+   "budget_hit_pct": 61.4,
+   "est_median_s_no_mtp": 32.86,
+   "est_median_s_mtp": 18.8
+  },
+  "MMLU-Pro": {
+   "requests": 12024,
+   "questions_thinking_pct": 78.4,
+   "think_tokens_median": 5361.0,
+   "budget_hit_pct": 28.0,
+   "est_median_s_no_mtp": 16.56,
+   "est_median_s_mtp": 9.49
   },
   "MuSR": {
-   "questions": 756,
-   "thinking_pct": 51.2,
+   "requests": 756,
+   "questions_thinking_pct": 51.2,
    "think_tokens_median": 4781.0,
    "budget_hit_pct": 11.4,
-   "est_latency_median_s": 5.41,
-   "est_latency_p90_s": 28.7
+   "est_median_s_no_mtp": 5.41,
+   "est_median_s_mtp": 3.13
   },
   "SATA-Bench": {
-   "questions": 15517,
-   "thinking_pct": 21.1,
+   "requests": 1650,
+   "questions_thinking_pct": 21.1,
    "think_tokens_median": 2600.0,
    "budget_hit_pct": 1.1,
-   "est_latency_median_s": 15.18,
-   "est_latency_p90_s": 60.6
-  },
-  "CRUXEval": {
-   "questions": 570,
-   "thinking_pct": 41.8,
-   "think_tokens_median": 7143.0,
-   "budget_hit_pct": 41.2,
-   "est_latency_median_s": 0.04,
-   "est_latency_p90_s": 32.9
-  },
-  "CLadder": {
-   "questions": 5000,
-   "thinking_pct": 51.1,
-   "think_tokens_median": 6431.5,
-   "budget_hit_pct": 33.4,
-   "est_latency_median_s": 6.66,
-   "est_latency_p90_s": 32.9
-  },
-  "HLE": {
-   "questions": 513,
-   "thinking_pct": 87.9,
-   "think_tokens_median": 8192.0,
-   "budget_hit_pct": 61.4,
-   "est_latency_median_s": 32.86,
-   "est_latency_p90_s": 32.9
+   "est_median_s_no_mtp": 15.18,
+   "est_median_s_mtp": 8.71
   }
  },
- "all_eight": {
-  "requests": 16325,
-  "thinking_pct_questions": 38.4,
-  "requests_with_thinking_pct": 58.7,
-  "est_latency_median_s": 14.96,
-  "est_latency_p90_s": 32.9
+ "all_ten": {
+  "requests": 33856,
+  "questions_thinking_pct": 51.0,
+  "requests_thinking_pct": 65.9,
+  "est_latency_no_mtp": {
+   "median_s": 13.4,
+   "p90_s": 32.9
+  },
+  "est_latency_mtp": {
+   "median_s": 7.7,
+   "p90_s": 18.8
+  }
  },
- "assumptions": "single request on one idle B200: System 1 ~45 ms; thinking ~250 tokens/s; ~50 ms answer read-out",
- "think_budget": 8192,
- "threshold": 0.8,
- "mmlu_pro_partial": {
-  "answered": 1611,
-  "system1": 0.5915580384854128,
-  "adaptive": 0.8013656114214773
+ "assumptions": "single request on one idle B200: System 1 ~45 ms; thinking ~250 tokens/s without and ~438 tokens/s with speculative decoding (gemma-4-26B-A4B-it-assistant); ~50 ms answer read-out",
+ "settings": {
+  "threshold": 0.8,
+  "think_budget": 8192
  }
 }

reports/decision_index_adaptive.json

@@ -1,17 +1,17 @@
 {
  "s1": {
-  "rows_replaced": 16325,
-  "decision_index": 58.03,
+  "rows_replaced": 33864,
+  "decision_index": 58.02,
   "scores": {
-   "balanced_skill": 58.03,
-   "balanced_raw": 67.33,
-   "breadth_skill": 56.96
+   "balanced_skill": 58.02,
+   "balanced_raw": 67.32,
+   "breadth_skill": 56.94
   },
   "areas": [
    {
     "id": "knowledge",
-    "skill": 0.4297,
-    "raw": 0.5475,
+    "skill": 0.4293,
+    "raw": 0.5472,
     "coverage": 1.0,
     "n": 10
    },
@@ -53,23 +53,23 @@
    "CRUXEval": 0.6754385964912281,
    "CLadder": 0.7102,
    "HLE": 0.08383233532934131,
-   "MMLU-Pro": 0.651845079787234,
-   "BBH fixed-option tasks": 0.7517704739422553
+   "MMLU-Pro": 0.6501828457446809,
+   "BBH fixed-option tasks": 0.7504993644452514
   }
  },
  "adaptive": {
-  "rows_replaced": 16325,
-  "decision_index": 61.15,
+  "rows_replaced": 33864,
+  "decision_index": 62.48,
   "scores": {
-   "balanced_skill": 61.15,
-   "balanced_raw": 69.62,
-   "breadth_skill": 60.6
+   "balanced_skill": 62.48,
+   "balanced_raw": 70.66,
+   "breadth_skill": 62.0
   },
   "areas": [
    {
     "id": "knowledge",
-    "skill": 0.5506,
-    "raw": 0.6362,
+    "skill": 0.602,
+    "raw": 0.6765,
     "coverage": 1.0,
     "n": 10
    },
@@ -111,18 +111,20 @@
    "CRUXEval": 0.9070175438596492,
    "CLadder": 0.8662,
    "HLE": 0.17764471057884232,
-   "MMLU-Pro": 0.651845079787234,
-   "BBH fixed-option tasks": 0.7517704739422553
+   "MMLU-Pro": 0.8464095744680851,
+   "BBH fixed-option tasks": 0.9202832758307609
   }
  },
  "completed_with_adaptive_thinking": [
+  "BBH fixed-option tasks",
+  "CLadder",
+  "CRUXEval",
+  "ChessBench",
   "GPQA Diamond",
   "GSM8K",
-  "ChessBench",
+  "HLE",
+  "MMLU-Pro",
   "MuSR",
-  "SATA-Bench",
-  "CRUXEval",
-  "CLadder",
-  "HLE"
+  "SATA-Bench"
  ]
 }