Benchmarks and Recipes

How QitOS separates framework capabilities, benchmark adapters, and reproducible recipe baselines.

Benchmarks and Recipes

QitOS keeps three different layers separate by design.

1. Framework layer

This is the reusable kernel (the core AgentModule + Engine execution loop):

• AgentModule + Engine
• DesktopEnv
• ActionSpace
• EnvironmentAdapter
• family presets
• qita replay and visual inspection

Framework code should stay benchmark-agnostic.

2. Benchmark layer

Dataset-specific integration belongs here:

• qitos.benchmark.desktop for the starter benchmark family
• qitos.benchmark.osworld for the real OSWorld adapter path
• benchmark-specific runtimes
• benchmark-specific evaluators/scorers
• benchmark-native task metadata and artifact (a persistent output file or data record from a run) handling

If something involves test_all.json, evaluator bridges, setup/postconfig, qcow2 boot inputs, or benchmark-native scoring, it belongs here.

The benchmark layer is the legacy interface for dataset adapters. New code should use recipes directly. The benchmark runners are being simplified into thin delegation layers that call recipes.

3. Recipe layer

Recipes are the canonical location for reproducible baseline methods:

• canonical single-agent baselines
• benchmark baseline methods
• multimodal starter methods

Available benchmark recipes:

Recipe	Module	Description
GAIA	qitos.recipes.benchmarks.gaia	Multi-step reasoning with web and file tools
TauBench	qitos.recipes.benchmarks.tau_bench	Airline/retail customer service simulation
CyBench	qitos.recipes.benchmarks.cybench	Cybersecurity CTF challenges
CyberGym	qitos.recipes.benchmarks.cybergym	Interactive cybersecurity gym environments
Desktop	qitos.recipes.desktop.osworld_starter	Desktop UI automation

Each recipe may be accompanied by an eval_config.yaml in qitos/recipes/benchmarks/eval_configs/ that specifies recommended defaults (model, max steps, max tokens, scorer).

Using eval configs

import yaml
from pathlib import Path
from qitos.harness import resolve_builtin_preset

config_path = Path("qitos/recipes/benchmarks/eval_configs/gaia.yaml")
with open(config_path) as f:
    config = yaml.safe_load(f)

# Apply defaults from config
model = config["defaults"]["model"]
max_steps = config["defaults"]["max_steps"]

# Combine with a family preset for advisory defaults
preset = resolve_builtin_preset(model)

Why this split matters

This split solves three real problems:

• benchmark runners no longer depend on example files
• one baseline can be reused by examples, docs, and benchmark runners
• future qitos-recipes extraction becomes a packaging move instead of a redesign

This separation keeps QitOS viable as a research-first framework.

W&B integration

To stream benchmark metrics to Weights & Biases alongside recipe execution:

from qitos.tracing import add_trace_processor
from qitos.tracing.wandb_processor import WandbTraceProcessor

processor = WandbTraceProcessor(
    project="qitos-benchmarks",
    tags=["gaia", "eval"],
)
add_trace_processor(processor)

See W&B Integration for full documentation.

If you are adding a new benchmark family, continue with Third-party benchmark integration.