Spaces:
Running
Running
Download scripts/ingest_run.py from facebook/omnilingual-gaia2: direct link, hf CLI and curl.
- Browser
- Download file 47 kB
-
https://huggingface.co/spaces/facebook/omnilingual-gaia2/resolve/main/scripts/ingest_run.py
- Command line
-
hf download hf://spaces/facebook/omnilingual-gaia2/scripts/ingest_run.py
-
curl -L -o ingest_run.py https://huggingface.co/spaces/facebook/omnilingual-gaia2/resolve/main/scripts/ingest_run.py
47 kB
#!/usr/bin/env python3
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
"""Turn a `gaia2-runner` output directory into a shard of the rollout archive.
This is the whole submission path. Run it once per (agent, language) and it
writes one `data/<submission>__<lang>.parquet` -- the archive's write unit, and
the only format this leaderboard publishes rollouts in -- into a checkout of the
archive dataset, ready to commit and open a pull request against it:
python scripts/ingest_run.py OUTPUT_DIR --provider my-org --model my-agent \\
--lang spa_Latn --started-at 2026-03-17 -o ../omnilingual-gaia2-results
`-o` is required and is the *other* repository: the archive is published as its
own dataset, and nothing here writes into this one.
A *submission* is one agent as run by one harness on one date, and it is what the
leaderboard treats as an entry: two harnesses driving the same underlying model
are two rows, because a Gaia2 score is as much a property of the scaffold as of
the model. Its id is `<provider>__<model>__<harness>__<YYYYMMDD>` unless
`--submission` names it outright, and the date comes from the run rather than from
the clock so that re-ingesting replaces a submission instead of minting a second
one. `--harness` is a closed vocabulary -- `Harness` in `omnigaia/catalog.py` --
so that the column it lands in stays groupable instead of accumulating spellings
of one scaffold.
The id is half the shard's filename and its `submission` column both, so a
submission is one string end to end. It is not what the leaderboard shows: that
is the `<provider>/<model>` repo id from `AGENTS`, so the id can carry the
provenance without the table having to display it.
One shard is written and nothing else in the archive is touched, so a submission
adds files and rewrites none. Re-running a language replaces that language's
shard rather than adding a second one beside it, and ingesting the same run twice
is a no-op.
The five tables committed under `data/tables/` in this repository are a build
product of that archive, so writing a shard makes them stale by exactly one
shard. Two things follow, and the ingester does both. *Before* it writes, it
checks that the committed tables still derive byte for byte from the archive at
`-o` -- `check_against` in `scripts/rederive.py`, the same comparison
`rederive.py --check` makes -- and refuses the run if they do not, because
ingesting onto an already-drifted archive folds this submission's change and
someone else's into one re-derivation and the pull request can no longer say
which is which. *After* it writes, it prints the re-derive command; it does not
run it, because the tables belong to a different repository and to a different
commit than the shard. `--no-rederive-check` skips the check, which is what an
archive that is not a checkout of the published dataset -- a scratch one, or a
first shard written offline -- needs.
Four artifacts are read out of each scenario leaf, and `result.json` is only the
first of them. It used to be the only one, on the argument that nothing the
leaderboard reports needs the rest -- which is true and is the wrong test: the
archive is the source of truth and the run directory it came from is in nobody's
repository, so a field not extracted here is a measurement that ceases to exist
the moment the tree is deleted. What each of the other three is for:
* `events.jsonl` -- turn attribution. The tool calls in `result.json:events`
carry no notion of which user turn they belong to, and the conversation
boundary is not recoverable from them: it is a running count of
`AgentUserInterface.send_message_to_agent` in the runner's own event log. Read
from there, each event in the archive's `events` blob gains a `turn` and an
`is_agent`, and the rollout gains an `n_turns` scalar so that the common
question is answerable without parsing the blob.
* `trace.jsonl` -- chain-of-thought, into the `reasoning` column. This is the
expensive one: ~1.4 MB per leaf and ~30 GB per submission, so it is streamed a
line at a time and never held whole. `--no-reasoning` skips it, which is what a
contributor who did not retain traces needs; an absent `trace.jsonl` is a
warning and not a failure, for the same reason.
* the scenario definition at `result.json:scenario_file` -- `num_oracle_events`.
The runner writes a literal 0 for it on the `judged_in_container` path, so the
value in `result.json` is not one. The true count is the number of
`class_name == "OracleEvent"` entries in the scenario's own definition, which
is a property of the scenario rather than of the rollout and is therefore the
same for every rollout of it -- so it is read once per scenario file and cached.
Layout handled, as written by `gaia2-runner`:
pass_at > 1: OUTPUT_DIR/run_<N>/<capability>/[<partition>/]<scenario>/result.json
pass_at == 1: OUTPUT_DIR/<capability>/[<partition>/]<scenario>/result.json
`events.jsonl` and `trace.jsonl` sit beside `result.json` in the same leaf.
The capability is the first path segment below the run directory, skipping a
`train|validation|test` partition segment if the dataset root has one. When a run
directory is absent the run number is 1.
Neither the language nor the judge is recorded in the outputs: `gaia2-runner`
persists them only in the TOML that `run_config.json:config_path` points at, and
only for the `run-config` command. Both are read from there when available and
can be overridden on the command line; `--lang` is required when neither the TOML
nor the dataset path reveals it. The same gap is why `--started-at YYYY-MM-DD` is
usually required: `run-dataset` sets no `config_path`, so nothing on that path
dates the run, and dating it from the clock is refused rather than guessed --
see `submission_id`.
`success` is tri-state. A rollout that never produced a judgment (infrastructure
error) stays `null` and is excluded from every denominator rather than counted as
a model failure. `reward` is never read: it is *absent* on the error path, so
defaulting it would silently convert outages into failures.
"""
from __future__ import annotations
import argparse
import functools
import json
import re
import sys
import tomllib
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
# `scripts/` too, so that the re-derivation check is the one in `rederive.py`
# rather than a second spelling of it here. Same reason `check_submission.py`
# imports `judge_matches_published` from this module.
sys.path.insert(0, str(Path(__file__).resolve().parent))
from rederive import check_against
from omnigaia.aggregate import JUDGE_MODEL, JUDGE_PROMPT_VERSION
from omnigaia.archive import (
RESULTS_REPO,
shard_file,
shard_files,
split_stem,
write_shard,
)
from omnigaia.catalog import Harness, Model
from omnigaia.metrics import CAPABILITIES, ENGLISH
#: Dataset partition directories that sit between the capability and the
#: scenario when a local dataset root has them. Mirrors gaia2-runner's own list.
PARTITION_DIRS = frozenset({"train", "validation", "test"})
#: `run_<N>` directories, present only when the run used `pass_at > 1`.
RUN_DIR_RE = re.compile(r"^run_(\d+)$")
#: FLORES-style language codes, as they appear in dataset paths.
LANG_RE = re.compile(r"\b([a-z]{3}_[A-Z][a-z]{3})\b")
#: The runner's own event log, beside `result.json` in each scenario leaf. Small
#: -- tens of KB -- and the only place a turn boundary is recorded.
EVENTS_LOG = "events.jsonl"
#: The LLM call log, also beside `result.json`. ~1.4 MB per leaf, which is why it
#: is streamed and why reading it is optional.
TRACE_LOG = "trace.jsonl"
#: The call that hands a user message to the agent, i.e. the start of a turn.
#: Turns are counted by occurrences of this and by nothing else: the agent's own
#: replies are not turn boundaries, and neither is a tool return.
TURN_TRIGGER = "AgentUserInterface.send_message_to_agent"
#: The `class_name` an oracle event carries in a scenario definition.
ORACLE_CLASS = "OracleEvent"
# --------------------------------------------------------------------------- #
# events.jsonl -- which turn each tool call belongs to
# --------------------------------------------------------------------------- #
def event_name(event: dict) -> str:
"""A logged event's qualified `App.function` name.
Assembled rather than read, because the runner spells it two ways depending
on which writer produced the line: as an `app`/`fn` pair, which is the shape
`result.json:events` also uses, or as one already-qualified `tool_name`. Both
are accepted and neither is guessed at beyond that -- an event with neither
returns "" and simply never matches `TURN_TRIGGER`, which is the right
outcome for a line this does not understand.
"""
app = event.get("app") or event.get("app_name") or ""
fn = event.get("fn") or event.get("function") or event.get("function_name") or ""
if app and fn:
return f"{app}.{fn}"
return str(event.get("tool_name") or event.get("name") or fn or "")
def is_agent_event(event: dict) -> bool:
"""Whether the *agent* made this call, rather than the environment.
The runner's own predicate: a write (`w is True`) with no `event_id`. The
`event_id` is what a scripted environment event carries, so its absence on a
write is what distinguishes something the agent did from something that was
done to it. `is True` rather than truthiness is deliberate and is copied from
the runner: `w` is tri-state there, and a null `w` is "not recorded" rather
than "not a write".
"""
write = event.get("w", event.get("write"))
return write is True and event.get("event_id") is None
def read_event_log(leaf: Path) -> list[dict] | None:
"""Per-event `{turn, is_agent}` for one leaf, in log order. None if absent.
A running counter over `TURN_TRIGGER`, which is the only definition of a turn
the runner records anywhere. The trigger event itself belongs to the turn it
opens, so the counter increments *before* the event is labelled; anything
logged before the first user message is turn 0, because a negative turn would
be a boundary that did not happen rather than a turn that did.
None, not `[]`, when there is no log: "this leaf published no event log" and
"this leaf logged no events" are different facts, and only the second one
means every event in it is attributable to turn 0.
Order is the file's order and is load-bearing: this list is zipped against
`result.json:events` positionally, which is the only correspondence the two
files have.
"""
path = leaf / EVENTS_LOG
if not path.is_file():
return None
attribution: list[dict] = []
turn = -1
try:
with path.open() as fh:
for line in fh:
line = line.strip()
if not line:
continue
event = json.loads(line)
if event_name(event) == TURN_TRIGGER:
turn += 1
attribution.append(
{"turn": max(turn, 0), "is_agent": is_agent_event(event)}
)
except (OSError, json.JSONDecodeError, AttributeError) as exc:
print(f" unreadable {path}: {exc}", file=sys.stderr)
return None
return attribution
def count_turns(attribution: list[dict] | None) -> int | None:
"""How many user turns the log recorded. None when there was no log."""
if attribution is None:
return None
return max((entry["turn"] for entry in attribution), default=-1) + 1
def annotate_events(events: list, attribution: list[dict] | None) -> list:
"""Add `turn` and `is_agent` to each event of a rollout's trace.
In place in the event dicts rather than in a column beside them -- see the
`events` field in `omnigaia/archive.py` for why -- so the existing payload
keys are untouched and the order is exactly the order that came in. A
downstream join depends on both.
The two files are matched positionally, because that is the only
correspondence they have: `result.json:events` and `events.jsonl` are two
renderings of one sequence. A length mismatch therefore means the pairing is
not the one assumed, and the answer is to attribute nothing and say so rather
than to attribute most of it and be wrong about the tail.
"""
if not events or not attribution:
return events
if len(events) != len(attribution):
print(
f" {len(events)} events but {len(attribution)} logged: not "
"attributing turns for this rollout",
file=sys.stderr,
)
return events
return [
{**event, **extra} if isinstance(event, dict) else event
for event, extra in zip(events, attribution, strict=True)
]
# --------------------------------------------------------------------------- #
# trace.jsonl -- chain-of-thought
# --------------------------------------------------------------------------- #
def is_genuine_user_message(message: dict) -> bool:
"""Whether a request message is a real user turn rather than a tool return.
The two serving shapes disagree about where a tool result goes, and counting
them as user messages would inflate every turn number:
* vLLM and the OpenAI-compatible endpoints give a tool return its own
`role == "tool"`, so every `role == "user"` message is genuine;
* Anthropic has no `tool` role and returns results as a `role == "user"`
message whose content is a list of blocks with `type == "tool_result"`.
So the rule is `role == "user"` minus anything carrying a `tool_result`
block. A string content is always genuine -- only the block form can hold a
tool result at all.
"""
if message.get("role") != "user":
return False
content = message.get("content")
if not isinstance(content, list):
return True
return not any(
isinstance(block, dict) and block.get("type") == "tool_result"
for block in content
)
def turn_of_call(request: dict) -> int:
"""Which user turn an LLM call served: genuine user messages so far, minus 1.
The request carries the whole conversation, so its user messages are a count
of the turns opened up to and including this one -- and turns are numbered
from 0, hence the minus. Clamped at 0 for a call with no user message at all,
which is a system-only preflight rather than turn -1.
"""
messages = request.get("messages") or []
turns = sum(
1 for m in messages if isinstance(m, dict) and is_genuine_user_message(m)
)
return max(turns - 1, 0)
def read_reasoning_log(leaf: Path) -> list[dict] | None:
"""Chain-of-thought for one leaf, as `{seq, turn, reasoning}` records.
Streamed a line at a time and never materialised whole: `trace.jsonl` is
~1.4 MB per leaf and ~30 GB per submission, and the records kept out of it
are a small fraction of that. Only calls that actually carried reasoning text
produce a record, so `[]` is the correct and common answer for a provider
that returns none -- three of the seven published submissions emit no
reasoning at all, and one more has no retained traces.
`seq` numbers the reasoning-bearing calls within the rollout, in trace order,
and is not the index of the call in the trace: it is a key for the records
that exist rather than a pointer into a file the archive does not hold.
None when there is no trace to read, which the caller warns about once. A
malformed line is skipped rather than fatal -- a truncated final line is what
a killed run leaves behind, and it should not cost the whole rollout.
"""
path = leaf / TRACE_LOG
if not path.is_file():
return None
records: list[dict] = []
try:
with path.open() as fh:
for line in fh:
line = line.strip()
if not line:
continue
try:
call = json.loads(line)
except json.JSONDecodeError:
continue
if not isinstance(call, dict):
continue
turn = turn_of_call(call.get("request") or {})
choices = (call.get("raw_response") or {}).get("choices") or []
for choice in choices:
if not isinstance(choice, dict):
continue
text = (choice.get("message") or {}).get("reasoning")
if not text:
continue
records.append(
{"seq": len(records), "turn": turn, "reasoning": text}
)
except OSError as exc:
print(f" unreadable {path}: {exc}", file=sys.stderr)
return None
return records
# --------------------------------------------------------------------------- #
# The scenario definition -- num_oracle_events
# --------------------------------------------------------------------------- #
@functools.cache
def count_oracle_events(scenario_file: str) -> int | None:
"""Oracle events in a scenario definition, or None if it cannot be read.
The length of the reference solution, and the value `num_oracle_events` in
`result.json` is supposed to hold: the runner writes a literal 0 there on the
`judged_in_container` path, so it is read from the scenario instead.
A property of the scenario, so every rollout of one scenario has the same
answer and the whole tree resolves to as many reads as there are scenarios --
hence the cache, keyed on the path string because that is what `result.json`
records.
None rather than 0 on any failure. 0 is a real count -- a scenario with no
oracle events -- and the whole reason this function exists is that a 0
standing in for "not computed" is indistinguishable from one.
"""
path = Path(scenario_file)
if not scenario_file or not path.is_file():
return None
try:
scenario = json.loads(path.read_text())
except (OSError, json.JSONDecodeError) as exc:
print(f" unreadable scenario {path}: {exc}", file=sys.stderr)
return None
events = scenario.get("events") or []
return sum(
1
for event in events
if isinstance(event, dict) and event.get("class_name") == ORACLE_CLASS
)
def find_results(output_dir: Path, workers: int) -> list[Path]:
"""Every `result.json` under the output dir, in deterministic order.
IO-bound over a lot of small directories, so the walk is threaded: on a
network filesystem this is where the wall-clock goes.
"""
tops = sorted(p for p in output_dir.iterdir() if p.is_dir())
found: list[list[Path]] = []
if tops:
with ThreadPoolExecutor(max_workers=workers) as pool:
found = list(pool.map(lambda top: sorted(top.rglob("result.json")), tops))
paths = [path for group in found for path in group]
# A pass@1 run can also drop results directly under the output dir.
paths.extend(sorted(output_dir.glob("*/result.json")))
return sorted(set(paths))
def parse_leaf(path: Path, output_dir: Path) -> tuple[int, str] | None:
"""Recover `(run, capability)` from a result.json's path.
Returns None when the path does not look like a scenario leaf, so that
stray files never become rollouts.
"""
parts = path.relative_to(output_dir).parts[:-2] # drop <scenario>/result.json
if not parts:
return None
run = 1
if match := RUN_DIR_RE.match(parts[0]):
run = int(match.group(1))
parts = parts[1:]
parts = tuple(p for p in parts if p not in PARTITION_DIRS)
if not parts:
return None
return run, parts[0]
def read_result(
path: Path, output_dir: Path, lang: str, read_reasoning: bool = True
) -> dict | None:
"""Parse one rollout leaf, or None if it is not one.
Handles all four shapes `gaia2-runner` writes: judged, scenario timeout, and
the two infrastructure-error shapes, the last of which carries only
`scenario_id`, `success: null` and `error`. Every optional key is read with
`.get`, and `success` is taken verbatim so that it stays tri-state.
The whole leaf is read, not just `result.json`: the sibling `events.jsonl`
attributes each tool call to a turn, the sibling `trace.jsonl` yields the
chain-of-thought, and `scenario_file` yields the oracle-event count the
runner failed to write. The module docstring says why each is worth the read.
Every one of the three is optional and absent for some trees, so each returns
None and lands as a null rather than blocking the rollout.
`read_reasoning=False` skips `trace.jsonl` entirely, which is what
`--no-reasoning` is: the one read here that costs ~1.4 MB of IO per leaf, and
the one a contributor may simply not have kept.
Returns the rollout record plus the two scenario-level fields (`scenario_id`,
`capability`) that the bundle carries once rather than per rollout.
"""
parsed = parse_leaf(path, output_dir)
if parsed is None:
return None
run, capability = parsed
leaf = path.parent
try:
result = json.loads(path.read_text())
except (OSError, json.JSONDecodeError) as exc:
print(f" unreadable {path}: {exc}", file=sys.stderr)
return None
reasons = result.get("failure_reasons") or []
if not reasons and result.get("error"):
reasons = [str(result["error"])]
attribution = read_event_log(leaf)
reasoning = read_reasoning_log(leaf) if read_reasoning else None
# The scenario's own count, falling back to what `result.json` recorded only
# when the definition cannot be read. The fallback is usually the literal 0
# the runner writes, which is why it is a last resort rather than a default.
oracle = count_oracle_events(str(result.get("scenario_file") or ""))
if oracle is None:
oracle = result.get("num_oracle_events")
return {
"scenario_id": result.get("scenario_id") or leaf.name,
"capability": capability,
"had_trace_log": (leaf / TRACE_LOG).is_file(),
"rollout": {
"lang": lang,
"run": run,
"success": result.get("success"),
"num_agent_events": result.get("num_agent_events"),
"num_oracle_events": oracle,
"n_turns": count_turns(attribution),
"failure_reason": reasons[0] if reasons else "",
"agent_response": result.get("agent_response") or "",
"judgments": result.get("judgments") or [],
"events": annotate_events(result.get("events") or [], attribution),
"reasoning": reasoning,
},
}
# --------------------------------------------------------------------------- #
# Run configuration
# --------------------------------------------------------------------------- #
def read_run_config(output_dir: Path) -> dict:
"""Merge `run_config.json` with the TOML it points at, if either exists.
`run_config.json` lives at the root of each run directory, so a pass@N tree
has one per run; they agree on everything read here. The TOML is the only
place the judge prompt version and `pass_at` are recorded, and `config_path`
is set by `run-config` but not by `run-dataset`.
"""
candidates = sorted(output_dir.glob("run_*/run_config.json"))
candidates.append(output_dir / "run_config.json")
config: dict = {}
for path in candidates:
if path.exists():
try:
config = json.loads(path.read_text())
except (OSError, json.JSONDecodeError) as exc:
print(f" ignoring {path}: {exc}", file=sys.stderr)
continue
break
if not config:
return {}
toml_path = config.get("config_path")
if toml_path and Path(toml_path).exists():
with open(toml_path, "rb") as fh:
config["_toml"] = tomllib.load(fh)
return config
def infer_lang(config: dict) -> str | None:
"""Find a FLORES-style code in the dataset path, e.g. `.../spa_Latn/data`."""
toml = config.get("_toml") or {}
haystacks = [
str((toml.get("target") or {}).get("dataset_root") or ""),
str((toml.get("target") or {}).get("dataset") or ""),
str(config.get("dataset") or ""),
str(config.get("dataset_cache_dir") or ""),
]
for text in haystacks:
if match := LANG_RE.search(text):
return match.group(1)
return None
def judge_matches_published(judge_model: str, prompt_version: str) -> bool:
"""Whether this run's judge is the one every published cell was graded by.
`JUDGE_MODEL`/`JUDGE_PROMPT_VERSION` are imported rather than restated, so
this cannot drift from the published convention. A `-01`/`-02` suffix is a
serving-endpoint replica of that one judge rather than a different judge --
`omnigaia/aggregate.py` collapses them for exactly that reason -- so the
comparison collapses them too instead of raising a false alarm.
"""
replica = re.fullmatch(rf"{re.escape(JUDGE_MODEL)}(-\d+)?", judge_model)
return bool(replica) and prompt_version == JUDGE_PROMPT_VERSION
def in_catalog(model: str) -> bool:
"""Whether `omnigaia/catalog.py` has display metadata under this name.
`Model` is a closed `StrEnum`, so membership is checked against its values --
`Model(name)` would raise, and a warning is the wrong place to do that.
"""
return model in {str(member) for member in Model}
def describe_run(config: dict) -> dict[str, str]:
"""The provenance worth reporting to whoever is submitting, as strings."""
toml = config.get("_toml") or {}
agent = toml.get("agent") or {}
judge = toml.get("judge") or {}
run = toml.get("run") or {}
return {
"model": str(agent.get("model") or config.get("model") or ""),
"judge_model": str(judge.get("model") or config.get("judge_model") or ""),
"judge_prompt_version": str(judge.get("prompt_version") or ""),
"pass_at": str(run.get("pass_at") or config.get("total_runs") or ""),
"started_at": str(config.get("started_at") or ""),
}
# --------------------------------------------------------------------------- #
# Bundles
# --------------------------------------------------------------------------- #
def submission_id(provider: str, model: str, harness: str, started_at: str) -> str:
"""`<provider>__<model>__<harness>__<YYYYMMDD>`, one submission's directory.
Double underscores separate the parts because every part may contain a single
one, and model slugs routinely do (`qwen-3.6-35b-a3b`).
The date comes from the run, never from the clock: minting it at ingest time
would give the same run a new id on every invocation, so re-ingesting would
*add* a second submission rather than replace the first and double-count the
agent into every denominator. That is also why an undatable run is an error
here rather than a fallback to today -- a date that is merely a timestamp of
when someone ran this script reads as provenance while being fiction. The
seven published at launch have no `started_at` in their bundles and so carry
the three-part form; `--submission` is how to name any other id outright.
"""
day = re.sub(r"[^0-9]", "", started_at)[:8]
if len(day) != 8:
raise ValueError(
f"cannot date this run from started_at={started_at!r}. Pass "
"--started-at YYYY-MM-DD, or --submission to name it outright."
)
return f"{slug(provider)}__{slug(model)}__{slug(harness)}__{day}"
def slug(text: str) -> str:
"""Lowercase, and everything outside `[a-z0-9.-]` collapsed to `-`."""
return re.sub(r"-{2,}", "-", re.sub(r"[^a-z0-9.-]+", "-", text.lower())).strip("-")
def rollout_key(rollout: dict) -> tuple[str, int]:
return rollout["lang"], rollout["run"]
def submission_langs(root: Path, submission: str) -> set[str]:
"""Every language this submission already has a shard for.
Read from the filenames rather than from any file's contents: the shard *is*
the language, so the answer is in the directory listing and no parquet footer
has to be opened.
"""
return {
parts[1]
for path in shard_files(root)
if (parts := split_stem(path.stem)) is not None and parts[0] == submission
}
def archive_rows(
by_scenario: dict[str, dict], provenance: dict, lang: str
) -> list[dict]:
"""This run's rollouts, as archive rows.
The boundary between the ingester's shape and the archive's. Everything the
run recorded is carried across -- the two blobs included -- because the
archive is the source of truth and the run directory it came from is not part
of any repository. What the published tables happen to read is a strictly
smaller set, and not this function's concern.
The scenario-level and submission-level fields are pushed down onto every
row. Columnar encoding stores each once per shard, so the redundancy is free
and the row becomes self-describing.
"""
rows = []
for scenario_id, group in sorted(by_scenario.items()):
for r in sorted(group["rollouts"], key=rollout_key):
events = r.get("events") or None
judgments = r.get("judgments") or None
# `[]` collapses to null here as the other blobs do, but the
# distinction it collapses is real and worth stating: a trace that was
# read and held no reasoning is the ordinary case for a provider that
# emits none, and a trace that was never read is `--no-reasoning` or a
# tree without one. Both land as null, because the column cannot
# carry the difference and the `runs` table is where provenance
# belongs.
reasoning = r.get("reasoning") or None
rows.append(
{
"scenario_id": scenario_id,
"capability": group["capability"],
"lang": lang,
"run": r["run"],
"success": r["success"],
"num_agent_events": r["num_agent_events"],
"failure_reason": r["failure_reason"],
"trace_published": bool(events),
"harness": provenance["harness"] or None,
"judge_model": provenance["judge_model"] or None,
"judge_prompt_version": provenance["judge_prompt_version"] or None,
"num_oracle_events": r.get("num_oracle_events"),
"n_turns": r.get("n_turns"),
# The true length of the stream, written beside the stream
# itself. This ingester caps nothing, so the two agree on
# every row it writes -- which is exactly why the column is
# filled here rather than left for a backfill: the archive's
# existing rows came through a sampler that clipped `events`
# at 150, and a reader has no way to tell a clipped list from
# a short one without this count. Null where nothing was
# published, for the same reason `n_turns` is.
"n_events": len(events) if events else None,
"agent_response": r.get("agent_response") or None,
"model": provenance["model"] or None,
"started_at": provenance["started_at"] or None,
"pass_at": provenance["pass_at"] or None,
# Null, not "[]": "none was published" is not "one was
# published and was empty".
"judgments": json.dumps(judgments, ensure_ascii=False)
if judgments
else None,
"events": json.dumps(events, ensure_ascii=False)
if events
else None,
"reasoning": json.dumps(reasoning, ensure_ascii=False)
if reasoning
else None,
}
)
return rows
def report_extraction(records: list[dict], no_reasoning: bool) -> None:
"""Say what the three extra reads found, and warn where they found nothing.
A warning rather than a refusal in every case. None of the three artifacts is
guaranteed to be in a tree -- an older runner wrote no `events.jsonl`, and a
contributor may have deleted 30 GB of `trace.jsonl` long before deciding to
submit -- and refusing an otherwise complete run over an optional column
would cost the leaderboard the rollouts to gain a field nothing publishes.
What it must not do is be silent. A null `reasoning` column is
indistinguishable after the write from a provider that emits no reasoning, so
the moment to say which one happened is here, while the tree is still in
front of whoever ran this.
"""
total = len(records)
turns = sum(1 for r in records if r["rollout"]["n_turns"] is not None)
oracle = sum(1 for r in records if r["rollout"]["num_oracle_events"] is not None)
print(
f" turns: {turns}/{total} rollouts attributed, "
f"oracle events: {oracle}/{total} counted from the scenario definition"
)
if not turns:
print(
f" no {EVENTS_LOG} in any leaf, so no tool call is attributed to a "
"turn and `n_turns` is null on every row. The trace is still "
"published; only the per-turn attribution is missing.",
file=sys.stderr,
)
if no_reasoning:
print(f" not reading {TRACE_LOG} (--no-reasoning); `reasoning` is null")
return
traced = sum(1 for r in records if r["had_trace_log"])
with_reasoning = sum(1 for r in records if r["rollout"]["reasoning"])
print(f" reasoning: {with_reasoning}/{total} rollouts, {traced} traces read")
if not traced:
print(
f" no {TRACE_LOG} in any leaf, so `reasoning` is null on every row. "
"That is expected for a tree whose traces were not retained; pass "
"--no-reasoning to skip the lookup and this warning.",
file=sys.stderr,
)
elif not with_reasoning:
print(
f" {traced} traces read and no reasoning text in any of them. That "
"is the expected result for a provider that returns none -- see the "
"`reasoning` field in omnigaia/archive.py -- and a surprise for one "
"that does.",
file=sys.stderr,
)
def rederive_check(out: Path, skip: bool) -> int:
"""Refuse to ingest onto an archive the committed tables have drifted from.
Returns an exit code to return, or 0 to carry on. The check is the *first*
thing `main` does -- before the walk, before a shard is written -- because a
refusal after the write would be advice about a file that already exists, and
because there is no post-write check to be had: writing the shard is itself
what makes the tables stale, so a re-derivation check run afterwards would
fail every time and mean nothing.
What it can protect is the state before that. If the five tables in
`data/tables/` already fail to follow from the archive at `out`, then someone
committed a shard without re-deriving, or edited a CSV by hand, and the
re-derivation this run is about to make necessary would sweep their change up
with this one. Re-deriving first separates the two into two commits that each
say what they are.
Run on `--dry-run` too, and deliberately: a dry run exists to say what the
real one would do, and "it would be refused" is part of that.
Note that the tables it reads are this repository's own, wherever this script
was copied to -- `omnigaia.derive.DATA_DIR` -- while the archive is the one
named by `-o`. The pairing is the invariant, and it is the pair that has to
be consistent, not either one alone.
"""
if skip:
print(f"not checking data/tables/ against {out} (--no-rederive-check)")
return 0
stale = check_against(out)
if not stale:
print(f"data/tables/ still derives from the archive at {out}")
return 0
print(
f"the tables committed under data/tables/ do not follow from the "
f"archive at {out}:",
file=sys.stderr,
)
for line in stale:
print(line, file=sys.stderr)
print(
f"\nSo this run is refused rather than written on top of that. If {out} "
f"is a checkout of {RESULTS_REPO}, re-derive and commit the tables "
f"first, as their own change:\n"
f" python scripts/rederive.py --archive {out}\n"
"If it is not -- a scratch archive, or a first shard written with no "
"published archive to hand -- then there is nothing for the tables to "
"follow from and --no-rederive-check is the right answer.",
file=sys.stderr,
)
return 2
def main() -> int:
ap = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
ap.add_argument("output_dir", type=Path, help="a gaia2-runner output directory")
ap.add_argument(
"--model",
help="agent id for the rollouts; defaults to the one in the run config",
)
ap.add_argument(
"--provider",
help="who publishes the weights, the `<org>` half of the Hub repo id "
"(e.g. moonshotai); part of the submission id",
)
ap.add_argument(
"--lang",
help="FLORES-style code, e.g. spa_Latn; inferred from the dataset path "
"when possible",
)
ap.add_argument(
"--harness",
type=Harness,
choices=list(Harness),
default=Harness.OPENCLAW,
help=f"what drove the agent; recorded in every bundle and part of the "
f"submission id (default: {Harness.OPENCLAW}). A scaffold not listed here "
f"is a pull request adding it to `Harness` in omnigaia/catalog.py",
)
ap.add_argument(
"--started-at",
help="the run's date, when the run config does not record one",
)
ap.add_argument(
"--submission",
help="name the submission outright instead of deriving "
"<provider>__<model>__<harness>__<YYYYMMDD>",
)
ap.add_argument(
"-o",
"--out",
type=Path,
required=True,
help="a checkout of the archive dataset "
"(facebook/omnilingual-gaia2-results) to write the shard into",
)
ap.add_argument(
"--dry-run",
action="store_true",
help="report what would be written without touching any shard",
)
ap.add_argument(
"--no-rederive-check",
action="store_true",
help="write even if the tables committed under data/tables/ do not "
"already derive from the archive at -o. What a scratch archive, or a "
"first shard written with no published archive to hand, needs",
)
ap.add_argument(
"--no-reasoning",
action="store_true",
help=f"do not read {TRACE_LOG}, leaving the `reasoning` column null. It "
"is ~1.4 MB per scenario leaf and ~30 GB per submission, and a "
"contributor who did not retain traces has nothing to read; the rest of "
"the ingest is unaffected",
)
ap.add_argument(
"--keep-all-capabilities",
action="store_true",
help="keep splits outside the four leaderboard capabilities (e.g. time)",
)
ap.add_argument("--workers", type=int, default=32, help="threads for the walk")
args = ap.parse_args()
if not args.output_dir.is_dir():
print(f"not a directory: {args.output_dir}", file=sys.stderr)
return 2
if code := rederive_check(args.out, args.no_rederive_check):
return code
config = read_run_config(args.output_dir)
provenance = describe_run(config)
model = args.model or provenance["model"]
if not model:
print(
"no --model given and none in the run config; pass --model explicitly",
file=sys.stderr,
)
return 2
lang = args.lang or infer_lang(config)
if not lang:
print(
"could not determine the language: pass --lang (e.g. --lang spa_Latn)",
file=sys.stderr,
)
return 2
started_at = args.started_at or provenance["started_at"]
if not args.submission and not args.provider:
print(
"no --provider given; it is part of the submission id. Pass the "
"`<org>` half of the Hub repo id (e.g. --provider moonshotai), or "
"--submission to name the submission outright.",
file=sys.stderr,
)
return 2
try:
submission = args.submission or submission_id(
args.provider, model, args.harness, started_at
)
except ValueError as exc:
print(exc, file=sys.stderr)
return 2
paths = find_results(args.output_dir, args.workers)
print(f"found {len(paths)} result.json files under {args.output_dir}")
with ThreadPoolExecutor(max_workers=args.workers) as pool:
parsed = pool.map(
lambda path: read_result(
path, args.output_dir, lang, read_reasoning=not args.no_reasoning
),
paths,
)
records = [row for row in parsed if row is not None]
if not args.keep_all_capabilities:
dropped = {
r["capability"] for r in records if r["capability"] not in CAPABILITIES
}
if dropped:
print(f" dropping splits outside the leaderboard: {sorted(dropped)}")
records = [r for r in records if r["capability"] in CAPABILITIES]
if not records:
print(
"no rollouts parsed; is this a gaia2-runner output tree?", file=sys.stderr
)
return 1
by_scenario: dict[str, dict] = defaultdict(
lambda: {"capability": "", "rollouts": []}
)
for record in records:
group = by_scenario[record["scenario_id"]]
group["capability"] = record["capability"]
group["rollouts"].append(record["rollout"])
graded = sum(1 for r in records if r["rollout"]["success"] is not None)
print(
f"{submission} / {lang}: {len(records)} rollouts over "
f"{len(by_scenario)} scenarios, {graded} graded, {len(records) - graded} "
"ungraded (excluded from every denominator)"
)
report_extraction(records, no_reasoning=args.no_reasoning)
judge, prompts = provenance["judge_model"], provenance["judge_prompt_version"]
if judge:
print(f" judge: {judge} prompt_version={prompts or 'unset'}")
if not judge_matches_published(judge, prompts):
print(
f" that is not the published judge: this run was graded by "
f"{judge} with the {prompts or 'unset'} prompts, not by "
f"{JUDGE_MODEL} with the {JUDGE_PROMPT_VERSION} prompts, so its "
"cells are not comparable to the published ones. Nothing is "
"blocked -- the bundles record the judge and the leaderboard "
"shows it, so the entry says so. Re-grade with the published "
"judge if you want it read as comparable.",
file=sys.stderr,
)
else:
print(
" judge unknown: run_config.json has no config_path (run-dataset does "
f"not set one). Say in the PR which judge graded this run -- only runs "
f"graded by {JUDGE_MODEL} with the {JUDGE_PROMPT_VERSION} prompts are "
"comparable to the published cells."
)
provenance_meta = {
"submission": submission,
"model": provenance["model"] or model,
"harness": str(args.harness),
"started_at": started_at,
"judge_model": provenance["judge_model"],
"judge_prompt_version": provenance["judge_prompt_version"],
"pass_at": provenance["pass_at"],
}
rows = archive_rows(by_scenario, provenance_meta, lang)
shard = shard_file(args.out, submission, lang)
existed = shard.exists()
verb = "would write" if args.dry_run else "wrote"
if not args.dry_run:
# One shard, replaced wholesale. Nothing else in the archive is touched:
# the shard is exactly (this submission, this language), so re-ingesting
# is idempotent and no other language is rewritten.
write_shard(args.out, submission, lang, rows)
print(
f"{verb} {len(rows):,} rollouts over {len(by_scenario)} scenarios to "
f"{shard} ({'replaced' if existed else 'new'})"
)
if not args.dry_run:
if ENGLISH not in submission_langs(args.out, submission):
print(
f" {submission} has no {ENGLISH} rollouts yet, so it will not be "
"listed: every cell is scored against the submission's own English "
"cell. Ingest English and it appears.",
file=sys.stderr,
)
if not in_catalog(submission):
print(
f" {submission} is not in the catalog, so it will be listed under "
"that raw slug with an empty `Type` column, and the data contract "
"will fail on the missing metadata. Add a `Model` member and an "
"`AGENTS` row for it in omnigaia/catalog.py -- the "
"`<org>/<model>` id and whether the weights are open -- in the "
"same pull request, or ask for one there.",
file=sys.stderr,
)
print(
f"Check it: python scripts/check_submission.py {submission} "
f"--archive {args.out}"
)
print(f"Re-derive: python scripts/rederive.py --archive {args.out}")
print(
"The tables under data/tables/ derived from the archive as it was a "
"moment ago, so they are now stale by exactly this shard -- that is "
"what the re-derive above fixes, and what the same check will refuse "
"the next ingest over if it is skipped. The two are committed "
"together: the shard to the dataset, the re-derived tables to the "
"Space. See the README."
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
#!/usr/bin/env python3
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
"""Turn a `gaia2-runner` output directory into a shard of the rollout archive.
This is the whole submission path. Run it once per (agent, language) and it
writes one `data/<submission>__<lang>.parquet` -- the archive's write unit, and
the only format this leaderboard publishes rollouts in -- into a checkout of the
archive dataset, ready to commit and open a pull request against it:
python scripts/ingest_run.py OUTPUT_DIR --provider my-org --model my-agent \\
--lang spa_Latn --started-at 2026-03-17 -o ../omnilingual-gaia2-results
`-o` is required and is the *other* repository: the archive is published as its
own dataset, and nothing here writes into this one.
A *submission* is one agent as run by one harness on one date, and it is what the
leaderboard treats as an entry: two harnesses driving the same underlying model
are two rows, because a Gaia2 score is as much a property of the scaffold as of
the model. Its id is `<provider>__<model>__<harness>__<YYYYMMDD>` unless
`--submission` names it outright, and the date comes from the run rather than from
the clock so that re-ingesting replaces a submission instead of minting a second
one. `--harness` is a closed vocabulary -- `Harness` in `omnigaia/catalog.py` --
so that the column it lands in stays groupable instead of accumulating spellings
of one scaffold.
The id is half the shard's filename and its `submission` column both, so a
submission is one string end to end. It is not what the leaderboard shows: that
is the `<provider>/<model>` repo id from `AGENTS`, so the id can carry the
provenance without the table having to display it.
One shard is written and nothing else in the archive is touched, so a submission
adds files and rewrites none. Re-running a language replaces that language's
shard rather than adding a second one beside it, and ingesting the same run twice
is a no-op.
The five tables committed under `data/tables/` in this repository are a build
product of that archive, so writing a shard makes them stale by exactly one
shard. Two things follow, and the ingester does both. *Before* it writes, it
checks that the committed tables still derive byte for byte from the archive at
`-o` -- `check_against` in `scripts/rederive.py`, the same comparison
`rederive.py --check` makes -- and refuses the run if they do not, because
ingesting onto an already-drifted archive folds this submission's change and
someone else's into one re-derivation and the pull request can no longer say
which is which. *After* it writes, it prints the re-derive command; it does not
run it, because the tables belong to a different repository and to a different
commit than the shard. `--no-rederive-check` skips the check, which is what an
archive that is not a checkout of the published dataset -- a scratch one, or a
first shard written offline -- needs.
Four artifacts are read out of each scenario leaf, and `result.json` is only the
first of them. It used to be the only one, on the argument that nothing the
leaderboard reports needs the rest -- which is true and is the wrong test: the
archive is the source of truth and the run directory it came from is in nobody's
repository, so a field not extracted here is a measurement that ceases to exist
the moment the tree is deleted. What each of the other three is for:
* `events.jsonl` -- turn attribution. The tool calls in `result.json:events`
carry no notion of which user turn they belong to, and the conversation
boundary is not recoverable from them: it is a running count of
`AgentUserInterface.send_message_to_agent` in the runner's own event log. Read
from there, each event in the archive's `events` blob gains a `turn` and an
`is_agent`, and the rollout gains an `n_turns` scalar so that the common
question is answerable without parsing the blob.
* `trace.jsonl` -- chain-of-thought, into the `reasoning` column. This is the
expensive one: ~1.4 MB per leaf and ~30 GB per submission, so it is streamed a
line at a time and never held whole. `--no-reasoning` skips it, which is what a
contributor who did not retain traces needs; an absent `trace.jsonl` is a
warning and not a failure, for the same reason.
* the scenario definition at `result.json:scenario_file` -- `num_oracle_events`.
The runner writes a literal 0 for it on the `judged_in_container` path, so the
value in `result.json` is not one. The true count is the number of
`class_name == "OracleEvent"` entries in the scenario's own definition, which
is a property of the scenario rather than of the rollout and is therefore the
same for every rollout of it -- so it is read once per scenario file and cached.
Layout handled, as written by `gaia2-runner`:
pass_at > 1: OUTPUT_DIR/run_<N>/<capability>/[<partition>/]<scenario>/result.json
pass_at == 1: OUTPUT_DIR/<capability>/[<partition>/]<scenario>/result.json
`events.jsonl` and `trace.jsonl` sit beside `result.json` in the same leaf.
The capability is the first path segment below the run directory, skipping a
`train|validation|test` partition segment if the dataset root has one. When a run
directory is absent the run number is 1.
Neither the language nor the judge is recorded in the outputs: `gaia2-runner`
persists them only in the TOML that `run_config.json:config_path` points at, and
only for the `run-config` command. Both are read from there when available and
can be overridden on the command line; `--lang` is required when neither the TOML
nor the dataset path reveals it. The same gap is why `--started-at YYYY-MM-DD` is
usually required: `run-dataset` sets no `config_path`, so nothing on that path
dates the run, and dating it from the clock is refused rather than guessed --