omnilingual-gaia2 / scripts /ingest_run.py
Andrea Caciolai
Widen the schema, match the appendix, resync the prose
811ffc6
Raw History Blame Contribute Delete
47 kB
#!/usr/bin/env python3 # Copyright (c) Meta Platforms, Inc. and affiliates. # All rights reserved. # # This source code is licensed under the license found in the # LICENSE file in the root directory of this source tree. """Turn a `gaia2-runner` output directory into a shard of the rollout archive. This is the whole submission path. Run it once per (agent, language) and it writes one `data/<submission>__<lang>.parquet` -- the archive's write unit, and the only format this leaderboard publishes rollouts in -- into a checkout of the archive dataset, ready to commit and open a pull request against it: python scripts/ingest_run.py OUTPUT_DIR --provider my-org --model my-agent \\ --lang spa_Latn --started-at 2026-03-17 -o ../omnilingual-gaia2-results `-o` is required and is the *other* repository: the archive is published as its own dataset, and nothing here writes into this one. A *submission* is one agent as run by one harness on one date, and it is what the leaderboard treats as an entry: two harnesses driving the same underlying model are two rows, because a Gaia2 score is as much a property of the scaffold as of the model. Its id is `<provider>__<model>__<harness>__<YYYYMMDD>` unless `--submission` names it outright, and the date comes from the run rather than from the clock so that re-ingesting replaces a submission instead of minting a second one. `--harness` is a closed vocabulary -- `Harness` in `omnigaia/catalog.py` -- so that the column it lands in stays groupable instead of accumulating spellings of one scaffold. The id is half the shard's filename and its `submission` column both, so a submission is one string end to end. It is not what the leaderboard shows: that is the `<provider>/<model>` repo id from `AGENTS`, so the id can carry the provenance without the table having to display it. One shard is written and nothing else in the archive is touched, so a submission adds files and rewrites none. Re-running a language replaces that language's shard rather than adding a second one beside it, and ingesting the same run twice is a no-op. The five tables committed under `data/tables/` in this repository are a build product of that archive, so writing a shard makes them stale by exactly one shard. Two things follow, and the ingester does both. *Before* it writes, it checks that the committed tables still derive byte for byte from the archive at `-o` -- `check_against` in `scripts/rederive.py`, the same comparison `rederive.py --check` makes -- and refuses the run if they do not, because ingesting onto an already-drifted archive folds this submission's change and someone else's into one re-derivation and the pull request can no longer say which is which. *After* it writes, it prints the re-derive command; it does not run it, because the tables belong to a different repository and to a different commit than the shard. `--no-rederive-check` skips the check, which is what an archive that is not a checkout of the published dataset -- a scratch one, or a first shard written offline -- needs. Four artifacts are read out of each scenario leaf, and `result.json` is only the first of them. It used to be the only one, on the argument that nothing the leaderboard reports needs the rest -- which is true and is the wrong test: the archive is the source of truth and the run directory it came from is in nobody's repository, so a field not extracted here is a measurement that ceases to exist the moment the tree is deleted. What each of the other three is for: * `events.jsonl` -- turn attribution. The tool calls in `result.json:events` carry no notion of which user turn they belong to, and the conversation boundary is not recoverable from them: it is a running count of `AgentUserInterface.send_message_to_agent` in the runner's own event log. Read from there, each event in the archive's `events` blob gains a `turn` and an `is_agent`, and the rollout gains an `n_turns` scalar so that the common question is answerable without parsing the blob. * `trace.jsonl` -- chain-of-thought, into the `reasoning` column. This is the expensive one: ~1.4 MB per leaf and ~30 GB per submission, so it is streamed a line at a time and never held whole. `--no-reasoning` skips it, which is what a contributor who did not retain traces needs; an absent `trace.jsonl` is a warning and not a failure, for the same reason. * the scenario definition at `result.json:scenario_file` -- `num_oracle_events`. The runner writes a literal 0 for it on the `judged_in_container` path, so the value in `result.json` is not one. The true count is the number of `class_name == "OracleEvent"` entries in the scenario's own definition, which is a property of the scenario rather than of the rollout and is therefore the same for every rollout of it -- so it is read once per scenario file and cached. Layout handled, as written by `gaia2-runner`: pass_at > 1: OUTPUT_DIR/run_<N>/<capability>/[<partition>/]<scenario>/result.json pass_at == 1: OUTPUT_DIR/<capability>/[<partition>/]<scenario>/result.json `events.jsonl` and `trace.jsonl` sit beside `result.json` in the same leaf. The capability is the first path segment below the run directory, skipping a `train|validation|test` partition segment if the dataset root has one. When a run directory is absent the run number is 1. Neither the language nor the judge is recorded in the outputs: `gaia2-runner` persists them only in the TOML that `run_config.json:config_path` points at, and only for the `run-config` command. Both are read from there when available and can be overridden on the command line; `--lang` is required when neither the TOML nor the dataset path reveals it. The same gap is why `--started-at YYYY-MM-DD` is usually required: `run-dataset` sets no `config_path`, so nothing on that path dates the run, and dating it from the clock is refused rather than guessed -- see `submission_id`. `success` is tri-state. A rollout that never produced a judgment (infrastructure error) stays `null` and is excluded from every denominator rather than counted as a model failure. `reward` is never read: it is *absent* on the error path, so defaulting it would silently convert outages into failures. """ from __future__ import annotations import argparse import functools import json import re import sys import tomllib from collections import defaultdict from concurrent.futures import ThreadPoolExecutor from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # `scripts/` too, so that the re-derivation check is the one in `rederive.py` # rather than a second spelling of it here. Same reason `check_submission.py` # imports `judge_matches_published` from this module. sys.path.insert(0, str(Path(__file__).resolve().parent)) from rederive import check_against from omnigaia.aggregate import JUDGE_MODEL, JUDGE_PROMPT_VERSION from omnigaia.archive import ( RESULTS_REPO, shard_file, shard_files, split_stem, write_shard, ) from omnigaia.catalog import Harness, Model from omnigaia.metrics import CAPABILITIES, ENGLISH #: Dataset partition directories that sit between the capability and the #: scenario when a local dataset root has them. Mirrors gaia2-runner's own list. PARTITION_DIRS = frozenset({"train", "validation", "test"}) #: `run_<N>` directories, present only when the run used `pass_at > 1`. RUN_DIR_RE = re.compile(r"^run_(\d+)$") #: FLORES-style language codes, as they appear in dataset paths. LANG_RE = re.compile(r"\b([a-z]{3}_[A-Z][a-z]{3})\b") #: The runner's own event log, beside `result.json` in each scenario leaf. Small #: -- tens of KB -- and the only place a turn boundary is recorded. EVENTS_LOG = "events.jsonl" #: The LLM call log, also beside `result.json`. ~1.4 MB per leaf, which is why it #: is streamed and why reading it is optional. TRACE_LOG = "trace.jsonl" #: The call that hands a user message to the agent, i.e. the start of a turn. #: Turns are counted by occurrences of this and by nothing else: the agent's own #: replies are not turn boundaries, and neither is a tool return. TURN_TRIGGER = "AgentUserInterface.send_message_to_agent" #: The `class_name` an oracle event carries in a scenario definition. ORACLE_CLASS = "OracleEvent" # --------------------------------------------------------------------------- # # events.jsonl -- which turn each tool call belongs to # --------------------------------------------------------------------------- # def event_name(event: dict) -> str: """A logged event's qualified `App.function` name. Assembled rather than read, because the runner spells it two ways depending on which writer produced the line: as an `app`/`fn` pair, which is the shape `result.json:events` also uses, or as one already-qualified `tool_name`. Both are accepted and neither is guessed at beyond that -- an event with neither returns "" and simply never matches `TURN_TRIGGER`, which is the right outcome for a line this does not understand. """ app = event.get("app") or event.get("app_name") or "" fn = event.get("fn") or event.get("function") or event.get("function_name") or "" if app and fn: return f"{app}.{fn}" return str(event.get("tool_name") or event.get("name") or fn or "") def is_agent_event(event: dict) -> bool: """Whether the *agent* made this call, rather than the environment. The runner's own predicate: a write (`w is True`) with no `event_id`. The `event_id` is what a scripted environment event carries, so its absence on a write is what distinguishes something the agent did from something that was done to it. `is True` rather than truthiness is deliberate and is copied from the runner: `w` is tri-state there, and a null `w` is "not recorded" rather than "not a write". """ write = event.get("w", event.get("write")) return write is True and event.get("event_id") is None def read_event_log(leaf: Path) -> list[dict] | None: """Per-event `{turn, is_agent}` for one leaf, in log order. None if absent. A running counter over `TURN_TRIGGER`, which is the only definition of a turn the runner records anywhere. The trigger event itself belongs to the turn it opens, so the counter increments *before* the event is labelled; anything logged before the first user message is turn 0, because a negative turn would be a boundary that did not happen rather than a turn that did. None, not `[]`, when there is no log: "this leaf published no event log" and "this leaf logged no events" are different facts, and only the second one means every event in it is attributable to turn 0. Order is the file's order and is load-bearing: this list is zipped against `result.json:events` positionally, which is the only correspondence the two files have. """ path = leaf / EVENTS_LOG if not path.is_file(): return None attribution: list[dict] = [] turn = -1 try: with path.open() as fh: for line in fh: line = line.strip() if not line: continue event = json.loads(line) if event_name(event) == TURN_TRIGGER: turn += 1 attribution.append( {"turn": max(turn, 0), "is_agent": is_agent_event(event)} ) except (OSError, json.JSONDecodeError, AttributeError) as exc: print(f" unreadable {path}: {exc}", file=sys.stderr) return None return attribution def count_turns(attribution: list[dict] | None) -> int | None: """How many user turns the log recorded. None when there was no log.""" if attribution is None: return None return max((entry["turn"] for entry in attribution), default=-1) + 1 def annotate_events(events: list, attribution: list[dict] | None) -> list: """Add `turn` and `is_agent` to each event of a rollout's trace. In place in the event dicts rather than in a column beside them -- see the `events` field in `omnigaia/archive.py` for why -- so the existing payload keys are untouched and the order is exactly the order that came in. A downstream join depends on both. The two files are matched positionally, because that is the only correspondence they have: `result.json:events` and `events.jsonl` are two renderings of one sequence. A length mismatch therefore means the pairing is not the one assumed, and the answer is to attribute nothing and say so rather than to attribute most of it and be wrong about the tail. """ if not events or not attribution: return events if len(events) != len(attribution): print( f" {len(events)} events but {len(attribution)} logged: not " "attributing turns for this rollout", file=sys.stderr, ) return events return [ {**event, **extra} if isinstance(event, dict) else event for event, extra in zip(events, attribution, strict=True) ] # --------------------------------------------------------------------------- # # trace.jsonl -- chain-of-thought # --------------------------------------------------------------------------- # def is_genuine_user_message(message: dict) -> bool: """Whether a request message is a real user turn rather than a tool return. The two serving shapes disagree about where a tool result goes, and counting them as user messages would inflate every turn number: * vLLM and the OpenAI-compatible endpoints give a tool return its own `role == "tool"`, so every `role == "user"` message is genuine; * Anthropic has no `tool` role and returns results as a `role == "user"` message whose content is a list of blocks with `type == "tool_result"`. So the rule is `role == "user"` minus anything carrying a `tool_result` block. A string content is always genuine -- only the block form can hold a tool result at all. """ if message.get("role") != "user": return False content = message.get("content") if not isinstance(content, list): return True return not any( isinstance(block, dict) and block.get("type") == "tool_result" for block in content ) def turn_of_call(request: dict) -> int: """Which user turn an LLM call served: genuine user messages so far, minus 1. The request carries the whole conversation, so its user messages are a count of the turns opened up to and including this one -- and turns are numbered from 0, hence the minus. Clamped at 0 for a call with no user message at all, which is a system-only preflight rather than turn -1. """ messages = request.get("messages") or [] turns = sum( 1 for m in messages if isinstance(m, dict) and is_genuine_user_message(m) ) return max(turns - 1, 0) def read_reasoning_log(leaf: Path) -> list[dict] | None: """Chain-of-thought for one leaf, as `{seq, turn, reasoning}` records. Streamed a line at a time and never materialised whole: `trace.jsonl` is ~1.4 MB per leaf and ~30 GB per submission, and the records kept out of it are a small fraction of that. Only calls that actually carried reasoning text produce a record, so `[]` is the correct and common answer for a provider that returns none -- three of the seven published submissions emit no reasoning at all, and one more has no retained traces. `seq` numbers the reasoning-bearing calls within the rollout, in trace order, and is not the index of the call in the trace: it is a key for the records that exist rather than a pointer into a file the archive does not hold. None when there is no trace to read, which the caller warns about once. A malformed line is skipped rather than fatal -- a truncated final line is what a killed run leaves behind, and it should not cost the whole rollout. """ path = leaf / TRACE_LOG if not path.is_file(): return None records: list[dict] = [] try: with path.open() as fh: for line in fh: line = line.strip() if not line: continue try: call = json.loads(line) except json.JSONDecodeError: continue if not isinstance(call, dict): continue turn = turn_of_call(call.get("request") or {}) choices = (call.get("raw_response") or {}).get("choices") or [] for choice in choices: if not isinstance(choice, dict): continue text = (choice.get("message") or {}).get("reasoning") if not text: continue records.append( {"seq": len(records), "turn": turn, "reasoning": text} ) except OSError as exc: print(f" unreadable {path}: {exc}", file=sys.stderr) return None return records # --------------------------------------------------------------------------- # # The scenario definition -- num_oracle_events # --------------------------------------------------------------------------- # @functools.cache def count_oracle_events(scenario_file: str) -> int | None: """Oracle events in a scenario definition, or None if it cannot be read. The length of the reference solution, and the value `num_oracle_events` in `result.json` is supposed to hold: the runner writes a literal 0 there on the `judged_in_container` path, so it is read from the scenario instead. A property of the scenario, so every rollout of one scenario has the same answer and the whole tree resolves to as many reads as there are scenarios -- hence the cache, keyed on the path string because that is what `result.json` records. None rather than 0 on any failure. 0 is a real count -- a scenario with no oracle events -- and the whole reason this function exists is that a 0 standing in for "not computed" is indistinguishable from one. """ path = Path(scenario_file) if not scenario_file or not path.is_file(): return None try: scenario = json.loads(path.read_text()) except (OSError, json.JSONDecodeError) as exc: print(f" unreadable scenario {path}: {exc}", file=sys.stderr) return None events = scenario.get("events") or [] return sum( 1 for event in events if isinstance(event, dict) and event.get("class_name") == ORACLE_CLASS ) def find_results(output_dir: Path, workers: int) -> list[Path]: """Every `result.json` under the output dir, in deterministic order. IO-bound over a lot of small directories, so the walk is threaded: on a network filesystem this is where the wall-clock goes. """ tops = sorted(p for p in output_dir.iterdir() if p.is_dir()) found: list[list[Path]] = [] if tops: with ThreadPoolExecutor(max_workers=workers) as pool: found = list(pool.map(lambda top: sorted(top.rglob("result.json")), tops)) paths = [path for group in found for path in group] # A pass@1 run can also drop results directly under the output dir. paths.extend(sorted(output_dir.glob("*/result.json"))) return sorted(set(paths)) def parse_leaf(path: Path, output_dir: Path) -> tuple[int, str] | None: """Recover `(run, capability)` from a result.json's path. Returns None when the path does not look like a scenario leaf, so that stray files never become rollouts. """ parts = path.relative_to(output_dir).parts[:-2] # drop <scenario>/result.json if not parts: return None run = 1 if match := RUN_DIR_RE.match(parts[0]): run = int(match.group(1)) parts = parts[1:] parts = tuple(p for p in parts if p not in PARTITION_DIRS) if not parts: return None return run, parts[0] def read_result( path: Path, output_dir: Path, lang: str, read_reasoning: bool = True ) -> dict | None: """Parse one rollout leaf, or None if it is not one. Handles all four shapes `gaia2-runner` writes: judged, scenario timeout, and the two infrastructure-error shapes, the last of which carries only `scenario_id`, `success: null` and `error`. Every optional key is read with `.get`, and `success` is taken verbatim so that it stays tri-state. The whole leaf is read, not just `result.json`: the sibling `events.jsonl` attributes each tool call to a turn, the sibling `trace.jsonl` yields the chain-of-thought, and `scenario_file` yields the oracle-event count the runner failed to write. The module docstring says why each is worth the read. Every one of the three is optional and absent for some trees, so each returns None and lands as a null rather than blocking the rollout. `read_reasoning=False` skips `trace.jsonl` entirely, which is what `--no-reasoning` is: the one read here that costs ~1.4 MB of IO per leaf, and the one a contributor may simply not have kept. Returns the rollout record plus the two scenario-level fields (`scenario_id`, `capability`) that the bundle carries once rather than per rollout. """ parsed = parse_leaf(path, output_dir) if parsed is None: return None run, capability = parsed leaf = path.parent try: result = json.loads(path.read_text()) except (OSError, json.JSONDecodeError) as exc: print(f" unreadable {path}: {exc}", file=sys.stderr) return None reasons = result.get("failure_reasons") or [] if not reasons and result.get("error"): reasons = [str(result["error"])] attribution = read_event_log(leaf) reasoning = read_reasoning_log(leaf) if read_reasoning else None # The scenario's own count, falling back to what `result.json` recorded only # when the definition cannot be read. The fallback is usually the literal 0 # the runner writes, which is why it is a last resort rather than a default. oracle = count_oracle_events(str(result.get("scenario_file") or "")) if oracle is None: oracle = result.get("num_oracle_events") return { "scenario_id": result.get("scenario_id") or leaf.name, "capability": capability, "had_trace_log": (leaf / TRACE_LOG).is_file(), "rollout": { "lang": lang, "run": run, "success": result.get("success"), "num_agent_events": result.get("num_agent_events"), "num_oracle_events": oracle, "n_turns": count_turns(attribution), "failure_reason": reasons[0] if reasons else "", "agent_response": result.get("agent_response") or "", "judgments": result.get("judgments") or [], "events": annotate_events(result.get("events") or [], attribution), "reasoning": reasoning, }, } # --------------------------------------------------------------------------- # # Run configuration # --------------------------------------------------------------------------- # def read_run_config(output_dir: Path) -> dict: """Merge `run_config.json` with the TOML it points at, if either exists. `run_config.json` lives at the root of each run directory, so a pass@N tree has one per run; they agree on everything read here. The TOML is the only place the judge prompt version and `pass_at` are recorded, and `config_path` is set by `run-config` but not by `run-dataset`. """ candidates = sorted(output_dir.glob("run_*/run_config.json")) candidates.append(output_dir / "run_config.json") config: dict = {} for path in candidates: if path.exists(): try: config = json.loads(path.read_text()) except (OSError, json.JSONDecodeError) as exc: print(f" ignoring {path}: {exc}", file=sys.stderr) continue break if not config: return {} toml_path = config.get("config_path") if toml_path and Path(toml_path).exists(): with open(toml_path, "rb") as fh: config["_toml"] = tomllib.load(fh) return config def infer_lang(config: dict) -> str | None: """Find a FLORES-style code in the dataset path, e.g. `.../spa_Latn/data`.""" toml = config.get("_toml") or {} haystacks = [ str((toml.get("target") or {}).get("dataset_root") or ""), str((toml.get("target") or {}).get("dataset") or ""), str(config.get("dataset") or ""), str(config.get("dataset_cache_dir") or ""), ] for text in haystacks: if match := LANG_RE.search(text): return match.group(1) return None def judge_matches_published(judge_model: str, prompt_version: str) -> bool: """Whether this run's judge is the one every published cell was graded by. `JUDGE_MODEL`/`JUDGE_PROMPT_VERSION` are imported rather than restated, so this cannot drift from the published convention. A `-01`/`-02` suffix is a serving-endpoint replica of that one judge rather than a different judge -- `omnigaia/aggregate.py` collapses them for exactly that reason -- so the comparison collapses them too instead of raising a false alarm. """ replica = re.fullmatch(rf"{re.escape(JUDGE_MODEL)}(-\d+)?", judge_model) return bool(replica) and prompt_version == JUDGE_PROMPT_VERSION def in_catalog(model: str) -> bool: """Whether `omnigaia/catalog.py` has display metadata under this name. `Model` is a closed `StrEnum`, so membership is checked against its values -- `Model(name)` would raise, and a warning is the wrong place to do that. """ return model in {str(member) for member in Model} def describe_run(config: dict) -> dict[str, str]: """The provenance worth reporting to whoever is submitting, as strings.""" toml = config.get("_toml") or {} agent = toml.get("agent") or {} judge = toml.get("judge") or {} run = toml.get("run") or {} return { "model": str(agent.get("model") or config.get("model") or ""), "judge_model": str(judge.get("model") or config.get("judge_model") or ""), "judge_prompt_version": str(judge.get("prompt_version") or ""), "pass_at": str(run.get("pass_at") or config.get("total_runs") or ""), "started_at": str(config.get("started_at") or ""), } # --------------------------------------------------------------------------- # # Bundles # --------------------------------------------------------------------------- # def submission_id(provider: str, model: str, harness: str, started_at: str) -> str: """`<provider>__<model>__<harness>__<YYYYMMDD>`, one submission's directory. Double underscores separate the parts because every part may contain a single one, and model slugs routinely do (`qwen-3.6-35b-a3b`). The date comes from the run, never from the clock: minting it at ingest time would give the same run a new id on every invocation, so re-ingesting would *add* a second submission rather than replace the first and double-count the agent into every denominator. That is also why an undatable run is an error here rather than a fallback to today -- a date that is merely a timestamp of when someone ran this script reads as provenance while being fiction. The seven published at launch have no `started_at` in their bundles and so carry the three-part form; `--submission` is how to name any other id outright. """ day = re.sub(r"[^0-9]", "", started_at)[:8] if len(day) != 8: raise ValueError( f"cannot date this run from started_at={started_at!r}. Pass " "--started-at YYYY-MM-DD, or --submission to name it outright." ) return f"{slug(provider)}__{slug(model)}__{slug(harness)}__{day}" def slug(text: str) -> str: """Lowercase, and everything outside `[a-z0-9.-]` collapsed to `-`.""" return re.sub(r"-{2,}", "-", re.sub(r"[^a-z0-9.-]+", "-", text.lower())).strip("-") def rollout_key(rollout: dict) -> tuple[str, int]: return rollout["lang"], rollout["run"] def submission_langs(root: Path, submission: str) -> set[str]: """Every language this submission already has a shard for. Read from the filenames rather than from any file's contents: the shard *is* the language, so the answer is in the directory listing and no parquet footer has to be opened. """ return { parts[1] for path in shard_files(root) if (parts := split_stem(path.stem)) is not None and parts[0] == submission } def archive_rows( by_scenario: dict[str, dict], provenance: dict, lang: str ) -> list[dict]: """This run's rollouts, as archive rows. The boundary between the ingester's shape and the archive's. Everything the run recorded is carried across -- the two blobs included -- because the archive is the source of truth and the run directory it came from is not part of any repository. What the published tables happen to read is a strictly smaller set, and not this function's concern. The scenario-level and submission-level fields are pushed down onto every row. Columnar encoding stores each once per shard, so the redundancy is free and the row becomes self-describing. """ rows = [] for scenario_id, group in sorted(by_scenario.items()): for r in sorted(group["rollouts"], key=rollout_key): events = r.get("events") or None judgments = r.get("judgments") or None # `[]` collapses to null here as the other blobs do, but the # distinction it collapses is real and worth stating: a trace that was # read and held no reasoning is the ordinary case for a provider that # emits none, and a trace that was never read is `--no-reasoning` or a # tree without one. Both land as null, because the column cannot # carry the difference and the `runs` table is where provenance # belongs. reasoning = r.get("reasoning") or None rows.append( { "scenario_id": scenario_id, "capability": group["capability"], "lang": lang, "run": r["run"], "success": r["success"], "num_agent_events": r["num_agent_events"], "failure_reason": r["failure_reason"], "trace_published": bool(events), "harness": provenance["harness"] or None, "judge_model": provenance["judge_model"] or None, "judge_prompt_version": provenance["judge_prompt_version"] or None, "num_oracle_events": r.get("num_oracle_events"), "n_turns": r.get("n_turns"), # The true length of the stream, written beside the stream # itself. This ingester caps nothing, so the two agree on # every row it writes -- which is exactly why the column is # filled here rather than left for a backfill: the archive's # existing rows came through a sampler that clipped `events` # at 150, and a reader has no way to tell a clipped list from # a short one without this count. Null where nothing was # published, for the same reason `n_turns` is. "n_events": len(events) if events else None, "agent_response": r.get("agent_response") or None, "model": provenance["model"] or None, "started_at": provenance["started_at"] or None, "pass_at": provenance["pass_at"] or None, # Null, not "[]": "none was published" is not "one was # published and was empty". "judgments": json.dumps(judgments, ensure_ascii=False) if judgments else None, "events": json.dumps(events, ensure_ascii=False) if events else None, "reasoning": json.dumps(reasoning, ensure_ascii=False) if reasoning else None, } ) return rows def report_extraction(records: list[dict], no_reasoning: bool) -> None: """Say what the three extra reads found, and warn where they found nothing. A warning rather than a refusal in every case. None of the three artifacts is guaranteed to be in a tree -- an older runner wrote no `events.jsonl`, and a contributor may have deleted 30 GB of `trace.jsonl` long before deciding to submit -- and refusing an otherwise complete run over an optional column would cost the leaderboard the rollouts to gain a field nothing publishes. What it must not do is be silent. A null `reasoning` column is indistinguishable after the write from a provider that emits no reasoning, so the moment to say which one happened is here, while the tree is still in front of whoever ran this. """ total = len(records) turns = sum(1 for r in records if r["rollout"]["n_turns"] is not None) oracle = sum(1 for r in records if r["rollout"]["num_oracle_events"] is not None) print( f" turns: {turns}/{total} rollouts attributed, " f"oracle events: {oracle}/{total} counted from the scenario definition" ) if not turns: print( f" no {EVENTS_LOG} in any leaf, so no tool call is attributed to a " "turn and `n_turns` is null on every row. The trace is still " "published; only the per-turn attribution is missing.", file=sys.stderr, ) if no_reasoning: print(f" not reading {TRACE_LOG} (--no-reasoning); `reasoning` is null") return traced = sum(1 for r in records if r["had_trace_log"]) with_reasoning = sum(1 for r in records if r["rollout"]["reasoning"]) print(f" reasoning: {with_reasoning}/{total} rollouts, {traced} traces read") if not traced: print( f" no {TRACE_LOG} in any leaf, so `reasoning` is null on every row. " "That is expected for a tree whose traces were not retained; pass " "--no-reasoning to skip the lookup and this warning.", file=sys.stderr, ) elif not with_reasoning: print( f" {traced} traces read and no reasoning text in any of them. That " "is the expected result for a provider that returns none -- see the " "`reasoning` field in omnigaia/archive.py -- and a surprise for one " "that does.", file=sys.stderr, ) def rederive_check(out: Path, skip: bool) -> int: """Refuse to ingest onto an archive the committed tables have drifted from. Returns an exit code to return, or 0 to carry on. The check is the *first* thing `main` does -- before the walk, before a shard is written -- because a refusal after the write would be advice about a file that already exists, and because there is no post-write check to be had: writing the shard is itself what makes the tables stale, so a re-derivation check run afterwards would fail every time and mean nothing. What it can protect is the state before that. If the five tables in `data/tables/` already fail to follow from the archive at `out`, then someone committed a shard without re-deriving, or edited a CSV by hand, and the re-derivation this run is about to make necessary would sweep their change up with this one. Re-deriving first separates the two into two commits that each say what they are. Run on `--dry-run` too, and deliberately: a dry run exists to say what the real one would do, and "it would be refused" is part of that. Note that the tables it reads are this repository's own, wherever this script was copied to -- `omnigaia.derive.DATA_DIR` -- while the archive is the one named by `-o`. The pairing is the invariant, and it is the pair that has to be consistent, not either one alone. """ if skip: print(f"not checking data/tables/ against {out} (--no-rederive-check)") return 0 stale = check_against(out) if not stale: print(f"data/tables/ still derives from the archive at {out}") return 0 print( f"the tables committed under data/tables/ do not follow from the " f"archive at {out}:", file=sys.stderr, ) for line in stale: print(line, file=sys.stderr) print( f"\nSo this run is refused rather than written on top of that. If {out} " f"is a checkout of {RESULTS_REPO}, re-derive and commit the tables " f"first, as their own change:\n" f" python scripts/rederive.py --archive {out}\n" "If it is not -- a scratch archive, or a first shard written with no " "published archive to hand -- then there is nothing for the tables to " "follow from and --no-rederive-check is the right answer.", file=sys.stderr, ) return 2 def main() -> int: ap = argparse.ArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter ) ap.add_argument("output_dir", type=Path, help="a gaia2-runner output directory") ap.add_argument( "--model", help="agent id for the rollouts; defaults to the one in the run config", ) ap.add_argument( "--provider", help="who publishes the weights, the `<org>` half of the Hub repo id " "(e.g. moonshotai); part of the submission id", ) ap.add_argument( "--lang", help="FLORES-style code, e.g. spa_Latn; inferred from the dataset path " "when possible", ) ap.add_argument( "--harness", type=Harness, choices=list(Harness), default=Harness.OPENCLAW, help=f"what drove the agent; recorded in every bundle and part of the " f"submission id (default: {Harness.OPENCLAW}). A scaffold not listed here " f"is a pull request adding it to `Harness` in omnigaia/catalog.py", ) ap.add_argument( "--started-at", help="the run's date, when the run config does not record one", ) ap.add_argument( "--submission", help="name the submission outright instead of deriving " "<provider>__<model>__<harness>__<YYYYMMDD>", ) ap.add_argument( "-o", "--out", type=Path, required=True, help="a checkout of the archive dataset " "(facebook/omnilingual-gaia2-results) to write the shard into", ) ap.add_argument( "--dry-run", action="store_true", help="report what would be written without touching any shard", ) ap.add_argument( "--no-rederive-check", action="store_true", help="write even if the tables committed under data/tables/ do not " "already derive from the archive at -o. What a scratch archive, or a " "first shard written with no published archive to hand, needs", ) ap.add_argument( "--no-reasoning", action="store_true", help=f"do not read {TRACE_LOG}, leaving the `reasoning` column null. It " "is ~1.4 MB per scenario leaf and ~30 GB per submission, and a " "contributor who did not retain traces has nothing to read; the rest of " "the ingest is unaffected", ) ap.add_argument( "--keep-all-capabilities", action="store_true", help="keep splits outside the four leaderboard capabilities (e.g. time)", ) ap.add_argument("--workers", type=int, default=32, help="threads for the walk") args = ap.parse_args() if not args.output_dir.is_dir(): print(f"not a directory: {args.output_dir}", file=sys.stderr) return 2 if code := rederive_check(args.out, args.no_rederive_check): return code config = read_run_config(args.output_dir) provenance = describe_run(config) model = args.model or provenance["model"] if not model: print( "no --model given and none in the run config; pass --model explicitly", file=sys.stderr, ) return 2 lang = args.lang or infer_lang(config) if not lang: print( "could not determine the language: pass --lang (e.g. --lang spa_Latn)", file=sys.stderr, ) return 2 started_at = args.started_at or provenance["started_at"] if not args.submission and not args.provider: print( "no --provider given; it is part of the submission id. Pass the " "`<org>` half of the Hub repo id (e.g. --provider moonshotai), or " "--submission to name the submission outright.", file=sys.stderr, ) return 2 try: submission = args.submission or submission_id( args.provider, model, args.harness, started_at ) except ValueError as exc: print(exc, file=sys.stderr) return 2 paths = find_results(args.output_dir, args.workers) print(f"found {len(paths)} result.json files under {args.output_dir}") with ThreadPoolExecutor(max_workers=args.workers) as pool: parsed = pool.map( lambda path: read_result( path, args.output_dir, lang, read_reasoning=not args.no_reasoning ), paths, ) records = [row for row in parsed if row is not None] if not args.keep_all_capabilities: dropped = { r["capability"] for r in records if r["capability"] not in CAPABILITIES } if dropped: print(f" dropping splits outside the leaderboard: {sorted(dropped)}") records = [r for r in records if r["capability"] in CAPABILITIES] if not records: print( "no rollouts parsed; is this a gaia2-runner output tree?", file=sys.stderr ) return 1 by_scenario: dict[str, dict] = defaultdict( lambda: {"capability": "", "rollouts": []} ) for record in records: group = by_scenario[record["scenario_id"]] group["capability"] = record["capability"] group["rollouts"].append(record["rollout"]) graded = sum(1 for r in records if r["rollout"]["success"] is not None) print( f"{submission} / {lang}: {len(records)} rollouts over " f"{len(by_scenario)} scenarios, {graded} graded, {len(records) - graded} " "ungraded (excluded from every denominator)" ) report_extraction(records, no_reasoning=args.no_reasoning) judge, prompts = provenance["judge_model"], provenance["judge_prompt_version"] if judge: print(f" judge: {judge} prompt_version={prompts or 'unset'}") if not judge_matches_published(judge, prompts): print( f" that is not the published judge: this run was graded by " f"{judge} with the {prompts or 'unset'} prompts, not by " f"{JUDGE_MODEL} with the {JUDGE_PROMPT_VERSION} prompts, so its " "cells are not comparable to the published ones. Nothing is " "blocked -- the bundles record the judge and the leaderboard " "shows it, so the entry says so. Re-grade with the published " "judge if you want it read as comparable.", file=sys.stderr, ) else: print( " judge unknown: run_config.json has no config_path (run-dataset does " f"not set one). Say in the PR which judge graded this run -- only runs " f"graded by {JUDGE_MODEL} with the {JUDGE_PROMPT_VERSION} prompts are " "comparable to the published cells." ) provenance_meta = { "submission": submission, "model": provenance["model"] or model, "harness": str(args.harness), "started_at": started_at, "judge_model": provenance["judge_model"], "judge_prompt_version": provenance["judge_prompt_version"], "pass_at": provenance["pass_at"], } rows = archive_rows(by_scenario, provenance_meta, lang) shard = shard_file(args.out, submission, lang) existed = shard.exists() verb = "would write" if args.dry_run else "wrote" if not args.dry_run: # One shard, replaced wholesale. Nothing else in the archive is touched: # the shard is exactly (this submission, this language), so re-ingesting # is idempotent and no other language is rewritten. write_shard(args.out, submission, lang, rows) print( f"{verb} {len(rows):,} rollouts over {len(by_scenario)} scenarios to " f"{shard} ({'replaced' if existed else 'new'})" ) if not args.dry_run: if ENGLISH not in submission_langs(args.out, submission): print( f" {submission} has no {ENGLISH} rollouts yet, so it will not be " "listed: every cell is scored against the submission's own English " "cell. Ingest English and it appears.", file=sys.stderr, ) if not in_catalog(submission): print( f" {submission} is not in the catalog, so it will be listed under " "that raw slug with an empty `Type` column, and the data contract " "will fail on the missing metadata. Add a `Model` member and an " "`AGENTS` row for it in omnigaia/catalog.py -- the " "`<org>/<model>` id and whether the weights are open -- in the " "same pull request, or ask for one there.", file=sys.stderr, ) print( f"Check it: python scripts/check_submission.py {submission} " f"--archive {args.out}" ) print(f"Re-derive: python scripts/rederive.py --archive {args.out}") print( "The tables under data/tables/ derived from the archive as it was a " "moment ago, so they are now stale by exactly this shard -- that is " "what the re-derive above fixes, and what the same check will refuse " "the next ingest over if it is skipped. The two are committed " "together: the shard to the dataset, the re-derived tables to the " "Space. See the README." ) return 0 if __name__ == "__main__": raise SystemExit(main())
#!/usr/bin/env python3
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
"""Turn a `gaia2-runner` output directory into a shard of the rollout archive.
This is the whole submission path. Run it once per (agent, language) and it
writes one `data/<submission>__<lang>.parquet` -- the archive's write unit, and
the only format this leaderboard publishes rollouts in -- into a checkout of the
archive dataset, ready to commit and open a pull request against it:
python scripts/ingest_run.py OUTPUT_DIR --provider my-org --model my-agent \\
--lang spa_Latn --started-at 2026-03-17 -o ../omnilingual-gaia2-results
`-o` is required and is the *other* repository: the archive is published as its
own dataset, and nothing here writes into this one.
A *submission* is one agent as run by one harness on one date, and it is what the
leaderboard treats as an entry: two harnesses driving the same underlying model
are two rows, because a Gaia2 score is as much a property of the scaffold as of
the model. Its id is `<provider>__<model>__<harness>__<YYYYMMDD>` unless
`--submission` names it outright, and the date comes from the run rather than from
the clock so that re-ingesting replaces a submission instead of minting a second
one. `--harness` is a closed vocabulary -- `Harness` in `omnigaia/catalog.py` --
so that the column it lands in stays groupable instead of accumulating spellings
of one scaffold.
The id is half the shard's filename and its `submission` column both, so a
submission is one string end to end. It is not what the leaderboard shows: that
is the `<provider>/<model>` repo id from `AGENTS`, so the id can carry the
provenance without the table having to display it.
One shard is written and nothing else in the archive is touched, so a submission
adds files and rewrites none. Re-running a language replaces that language's
shard rather than adding a second one beside it, and ingesting the same run twice
is a no-op.
The five tables committed under `data/tables/` in this repository are a build
product of that archive, so writing a shard makes them stale by exactly one
shard. Two things follow, and the ingester does both. *Before* it writes, it
checks that the committed tables still derive byte for byte from the archive at
`-o` -- `check_against` in `scripts/rederive.py`, the same comparison
`rederive.py --check` makes -- and refuses the run if they do not, because
ingesting onto an already-drifted archive folds this submission's change and
someone else's into one re-derivation and the pull request can no longer say
which is which. *After* it writes, it prints the re-derive command; it does not
run it, because the tables belong to a different repository and to a different
commit than the shard. `--no-rederive-check` skips the check, which is what an
archive that is not a checkout of the published dataset -- a scratch one, or a
first shard written offline -- needs.
Four artifacts are read out of each scenario leaf, and `result.json` is only the
first of them. It used to be the only one, on the argument that nothing the
leaderboard reports needs the rest -- which is true and is the wrong test: the
archive is the source of truth and the run directory it came from is in nobody's
repository, so a field not extracted here is a measurement that ceases to exist
the moment the tree is deleted. What each of the other three is for:
* `events.jsonl` -- turn attribution. The tool calls in `result.json:events`
carry no notion of which user turn they belong to, and the conversation
boundary is not recoverable from them: it is a running count of
`AgentUserInterface.send_message_to_agent` in the runner's own event log. Read
from there, each event in the archive's `events` blob gains a `turn` and an
`is_agent`, and the rollout gains an `n_turns` scalar so that the common
question is answerable without parsing the blob.
* `trace.jsonl` -- chain-of-thought, into the `reasoning` column. This is the
expensive one: ~1.4 MB per leaf and ~30 GB per submission, so it is streamed a
line at a time and never held whole. `--no-reasoning` skips it, which is what a
contributor who did not retain traces needs; an absent `trace.jsonl` is a
warning and not a failure, for the same reason.
* the scenario definition at `result.json:scenario_file` -- `num_oracle_events`.
The runner writes a literal 0 for it on the `judged_in_container` path, so the
value in `result.json` is not one. The true count is the number of
`class_name == "OracleEvent"` entries in the scenario's own definition, which
is a property of the scenario rather than of the rollout and is therefore the
same for every rollout of it -- so it is read once per scenario file and cached.
Layout handled, as written by `gaia2-runner`:
pass_at > 1: OUTPUT_DIR/run_<N>/<capability>/[<partition>/]<scenario>/result.json
pass_at == 1: OUTPUT_DIR/<capability>/[<partition>/]<scenario>/result.json
`events.jsonl` and `trace.jsonl` sit beside `result.json` in the same leaf.
The capability is the first path segment below the run directory, skipping a
`train|validation|test` partition segment if the dataset root has one. When a run
directory is absent the run number is 1.
Neither the language nor the judge is recorded in the outputs: `gaia2-runner`
persists them only in the TOML that `run_config.json:config_path` points at, and
only for the `run-config` command. Both are read from there when available and
can be overridden on the command line; `--lang` is required when neither the TOML
nor the dataset path reveals it. The same gap is why `--started-at YYYY-MM-DD` is
usually required: `run-dataset` sets no `config_path`, so nothing on that path
dates the run, and dating it from the clock is refused rather than guessed --