SETUP.md

TriageSieve-OpenEnv: Complete Setup & Usage Guide

This guide covers everything needed to run, test, build, and deploy the TriageSieve-OpenEnv environment on any system.

---

Prerequisites

Tool	Version	Check command
Python	>= 3.10	python --version
uv (recommended)	latest	uv --version
Docker	latest	docker --version
Git	latest	git --version
openenv-core	>= 0.2.2	openenv --version

Install uv (if not installed)

# Linux / macOS
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

---

1. Initial Setup

Clone and install dependencies

git clone <your-repo-url> triagesieve_env
cd triagesieve_env

# Option A: using uv (recommended - fast, reproducible)
uv sync

# Option B: using pip
python -m venv .venv
# Linux/macOS: source .venv/bin/activate
# Windows: .venv\Scripts\activate
pip install -e ".[dev]"

Verify installation

# Should print package version info
python -c "from triagesieve_env.models import ActionType; print('OK:', list(ActionType))"

---

2. Running Tests

Full test suite (~729 tests, ~4 minutes)

python -m pytest tests/ -q

Single test file

python -m pytest tests/test_rewards.py -q
python -m pytest tests/test_transitions.py -q
python -m pytest tests/test_determinism.py -q
python -m pytest tests/test_environment_part1.py -q
python -m pytest tests/test_environment_part2.py -q
python -m pytest tests/test_scripted_expert.py -q
python -m pytest tests/test_episode_engine.py -q
python -m pytest tests/test_openenv_integration.py -q

Single test by name

python -m pytest tests/test_transitions.py -k "test_route_before_classify" -q

Stop on first failure with traceback

python -m pytest tests/ -x --tb=short

Run with coverage

python -m pytest tests/ --cov=server --cov=baseline --cov-report=term-missing

Notes

• tests/test_llm_baseline.py skips most tests unless a real LLM API key is set (this is expected).
• All tests are deterministic and seed-based — no flaky tests.

---

3. Running the Scripted Expert Baseline

The scripted expert is an oracle agent that reads hidden ground truth. It proves the environment is solvable.

Smoke playthrough (all difficulties)

python scripts/smoke_playthrough.py

Output:

easy:   1.0000  PASS (threshold=0.90)
medium: 1.0000  PASS (threshold=0.85)
hard:   0.3833  PASS (threshold=0.35)

Single difficulty

python scripts/smoke_playthrough.py --difficulty easy
python scripts/smoke_playthrough.py --difficulty medium
python scripts/smoke_playthrough.py --difficulty hard

Custom seed

python scripts/smoke_playthrough.py --seed 0
python scripts/smoke_playthrough.py --seed 100 --difficulty easy

Quiet mode (suppress step-by-step trace)

python scripts/smoke_playthrough.py --quiet

---

4. Running the Server Locally

Start the FastAPI server

# Option A: via project entry point (recommended)
uv run server

# Option B: uvicorn with fully-qualified module path
python -m uvicorn triagesieve_env.server.app:app --reload --host 0.0.0.0 --port 8000

Verify server is running

curl http://localhost:8000/health
# Should return: {"status": "ok"}

curl http://localhost:8000/docs
# Opens Swagger UI in browser

Test with the Python client

import asyncio
from triagesieve_env import TriageSieveEnv
from triagesieve_env.models import TriageSieveAction, ActionType

async def main():
    env = TriageSieveEnv(base_url="http://localhost:8000")
    result = await env.reset(seed=42, difficulty="easy", mode="eval_strict")
    obs = result.observation
    print(f"Tickets: {len(obs.inbox_summaries)}, Budget: {obs.action_budget_remaining}")

    # Open first ticket
    tid = obs.inbox_summaries[0].ticket_id
    result = await env.step(TriageSieveAction(
        action_type=ActionType.OPEN_TICKET,
        ticket_id=tid,
    ))
    print(f"Result: {result.observation.last_action_result}")
    print(f"Focused ticket subject: {result.observation.focused_ticket.subject}")

asyncio.run(main())

---

5. Docker Build & Run

Build the Docker image

# From project root (where Dockerfile is)
docker build -t triagesieve_env .

This uses the OpenEnv multi-stage Dockerfile:

• Stage 1 (builder): Installs dependencies via uv sync from uv.lock
• Stage 2 (runtime): Copies only .venv + code, runs uvicorn on port 8000

Run the container

docker run -p 8000:8000 triagesieve_env

Verify container health

curl http://localhost:8000/health

Test with Docker client

import asyncio
from triagesieve_env import TriageSieveEnv

async def main():
    # Connects to container on port 8000
    env = await TriageSieveEnv.from_docker_image("triagesieve_env")
    result = await env.reset(seed=42, difficulty="easy")
    print(f"Tickets: {len(result.observation.inbox_summaries)}")
    await env.close()

asyncio.run(main())

---

6. Running the Inference Script

The inference.py is the hackathon-mandated evaluation script. It runs an LLM agent against the Dockerized environment and reports scores.

Required environment variables

# REQUIRED
export HF_TOKEN="your-huggingface-token"
export LOCAL_IMAGE_NAME="triagesieve_env"

# OPTIONAL (have defaults)
export API_BASE_URL="/static-proxy?url=https%3A%2F%2Frouter.huggingface.co%2Fv1"   # default
export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"            # default

Windows (PowerShell)

$env:HF_TOKEN = "your-huggingface-token"
$env:LOCAL_IMAGE_NAME = "triagesieve_env"

Windows (CMD)

set HF_TOKEN=your-huggingface-token
set LOCAL_IMAGE_NAME=triagesieve_env

Steps to run

# 1. Build the Docker image (if not already built)
docker build -t triagesieve_env .

# 2. Set env vars (see above)

# 3. Run inference
python inference.py

What it does

1. For each difficulty (easy, medium, hard):
   • Spins up the environment via Docker
   • Calls env.reset(seed, difficulty) to get the inbox
   • Loops: serializes observation → sends to LLM → parses action → env.step(action)
   • Prints [START], [STEP], [END] to stdout (mandatory format for automated judging)
2. Prints a summary with scores per task

Expected stdout format

[START] task=easy env=triagesieve_env model=Qwen/Qwen2.5-72B-Instruct
[STEP] step=1 action=open_ticket:T001 reward=0.01 done=false error=null
[STEP] step=2 action=classify_ticket:T001:billing reward=0.03 done=false error=null
...
[END] success=true steps=5 score=0.850 rewards=0.01,0.03,0.01,0.01,0.85

[START] task=medium env=triagesieve_env model=Qwen/Qwen2.5-72B-Instruct
...

Using a different model

export MODEL_NAME="meta-llama/Llama-3.3-70B-Instruct"
python inference.py

Using a different API endpoint

export API_BASE_URL="http://localhost:11434/v1"  # e.g., local Ollama
export MODEL_NAME="llama3.3"
python inference.py

---

7. Episode Generation & Validation

Generate the seeded episode bank

python scripts/generate_episodes.py --seed 0 --count 100 --difficulty all --output data/seeded_episodes.jsonl

Options:

• --seed <int>: Master seed (episodes get seeds seed+0, seed+1, ...)
• --count <int>: Number of episodes to generate
• --difficulty <easy|medium|hard|all>: Filter by difficulty (all = round-robin)
• --output <path>: Output JSONL file

Validate the episode bank

python scripts/validate_episode_bank.py --input data/seeded_episodes.jsonl

Checks:

1. Parse pass: Valid JSON, required keys present, non-zero tickets
2. Determinism pass: Re-renders episodes from (seed, difficulty) and verifies they match
3. Solvability pass: Runs scripted expert and checks score >= threshold

---

8. Deploying to Hugging Face Spaces

Validate before deploying

openenv validate
# Expected: [OK] triagesieve: Ready for multi-mode deployment

Push to HF Spaces

# Login to Hugging Face (if not already)
huggingface-cli login

# Push
openenv push your-username/triagesieve_env

Verify deployment

curl -X POST https://your-username-triagesieve-env.hf.space/reset \
  -H "Content-Type: application/json" \
  -d '{}'
# Should return 200 with initial observation JSON

---

9. Project Structure Reference

triagesieve_env/
├── inference.py              # Hackathon inference script (LLM → environment)
├── Dockerfile                # Docker image definition (multi-stage, OpenEnv base)
├── .dockerignore             # Excludes .venv, .git, __pycache__ from Docker build
├── openenv.yaml              # OpenEnv manifest (name, runtime, port)
├── pyproject.toml            # Package metadata + dependencies
├── uv.lock                   # Locked dependency versions
├── README.md                 # Judge-facing documentation
├── SETUP.md                  # This file
├── CLAUDE.md                 # AI assistant instructions
├── __init__.py               # Package exports
├── models.py                 # All Pydantic models + enums (Action, Observation, State)
├── client.py                 # TriageSieveEnv(EnvClient) async client
├── server/
│   ├── app.py                # FastAPI entrypoint via create_app()
│   ├── triagesieve_env_environment.py  # Environment(step/reset/state)
│   ├── episode_engine.py     # Deterministic episode generation from archetypes
│   ├── policy_graph.py       # SOP DAG definitions + UJCS computation
│   ├── scorer.py             # Terminal scoring, penalties, final score formula
│   ├── hint_engine.py        # Guided-mode hints (train_guided only)
│   └── Dockerfile            # Copy of root Dockerfile
├── baseline/
│   ├── scripted_expert.py    # Oracle policy (reads hidden truth) — proves solvability
│   └── llm_baseline.py       # LiteLLM-based agent (no hidden truth access)
├── data/
│   ├── archetypes.json       # 18 scenario archetypes with SOP graphs
│   ├── templates.json        # Reply/closure templates
│   ├── routing_rules.json    # Queue prerequisites + issue family mapping
│   ├── sla_rules.json        # Customer tier → SLA deadline mapping
│   └── seeded_episodes.jsonl # Pre-rendered episode cache (100 episodes)
├── scripts/
│   ├── generate_episodes.py  # CLI episode generator
│   ├── validate_episode_bank.py  # Validates bank parse + determinism + solvability
│   └── smoke_playthrough.py  # Runs scripted expert, asserts thresholds
├── tests/                    # 729 tests (pytest)
│   ├── test_transitions.py   # State machine edge cases
│   ├── test_rewards.py       # Scoring regression tests
│   ├── test_determinism.py   # Seed replay tests
│   ├── test_openenv_integration.py  # reset/step/state contract
│   ├── test_environment_part1.py    # Format gate, action dispatch
│   ├── test_environment_part2.py    # Workflows, merge, close
│   ├── test_episode_engine.py       # Episode generation
│   ├── test_scripted_expert.py      # Expert correctness
│   ├── test_llm_baseline.py         # LLM baseline (skips without API key)
│   └── test_validate_episode_bank.py # Bank validation
└── outputs/
    ├── logs/                 # Runtime logs (gitignored)
    └── evals/                # Structured traces (gitignored)

---

10. Scoring Quick Reference

FinalScore = TerminalBusinessScore (max 0.85)
           + 0.15 x UJCS_OpenEnv
           - EpisodePenalties
           clamped to [0, 1]

Terminal business score components (per ticket, weighted by priority)

Component	Weight
Classification correctness	0.15
Impact/urgency correctness	0.15
Queue correctness	0.20
Missing-info handling	0.10
Escalation correctness	0.10
Duplicate/non-actionable handling	0.05
Template choice correctness	0.05
Terminal status correctness	0.05

Priority weights

Priority	Weight
critical	2.0
high	1.5
medium	1.0
low	0.5

Penalties

Penalty	Value
Invalid action	-0.03
Avoidable reassignment	-0.05
Unnecessary escalation	-0.05
Urgent-ticket SLA mishandling	-0.05 to -0.10

Task budgets

Difficulty	Tickets	Budget
easy	1	6 steps
medium	2-3	12 steps
hard	3-4	14 steps

---

11. Troubleshooting

"Module not found" errors

Make sure you're running from the project root and your venv is active:

cd triagesieve_env
source .venv/bin/activate   # Linux/macOS
.venv\Scripts\activate      # Windows

Docker build fails

• Ensure Docker Desktop is running
• Check .dockerignore excludes .venv/ (otherwise the build context is huge)
• Try: docker build --no-cache -t triagesieve_env .

Tests fail with import errors

# Ensure dev dependencies are installed
uv sync
# or: pip install -e ".[dev]"

inference.py can't connect to environment

# Make sure Docker image is built
docker build -t triagesieve_env .

# Make sure LOCAL_IMAGE_NAME matches
export LOCAL_IMAGE_NAME="triagesieve_env"

# Check Docker is running
docker ps

openenv validate fails

# Install openenv-core
pip install "openenv-core[core]>=0.2.2"

# Run from project root (where openenv.yaml is)
cd triagesieve_env
openenv validate