> ## Documentation Index
> Fetch the complete documentation index at: https://qitor.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarks and Recipes

> How QitOS separates framework capabilities, benchmark adapters, and reproducible recipe baselines.

# Benchmarks and Recipes

QitOS keeps three different layers separate by design.

## 1. Framework layer

This is the reusable kernel (the core AgentModule + Engine execution loop):

* `AgentModule + Engine`
* `DesktopEnv`
* `ActionSpace`
* `EnvironmentAdapter`
* family presets
* qita replay and visual inspection

Framework code should stay benchmark-agnostic.

## 2. Benchmark layer

Dataset-specific integration belongs here:

* `qitos.benchmark.desktop` for the starter benchmark family
* `qitos.benchmark.osworld` for the real OSWorld adapter path
* benchmark-specific runtimes
* benchmark-specific evaluators/scorers
* benchmark-native task metadata and artifact (a persistent output file or data record from a run) handling

If something involves `test_all.json`, evaluator bridges, setup/postconfig, qcow2 boot inputs, or benchmark-native scoring, it belongs here.

<Note>
  The benchmark layer is the **legacy interface** for dataset adapters. New code should
  use recipes directly. The benchmark runners are being simplified into thin delegation
  layers that call recipes.
</Note>

## 3. Recipe layer

Recipes are the **canonical location** for reproducible baseline methods:

* canonical single-agent baselines
* benchmark baseline methods
* multimodal starter methods

Available benchmark recipes:

| Recipe   | Module                                  | Description                                  |
| -------- | --------------------------------------- | -------------------------------------------- |
| GAIA     | `qitos.recipes.benchmarks.gaia`         | Multi-step reasoning with web and file tools |
| TauBench | `qitos.recipes.benchmarks.tau_bench`    | Airline/retail customer service simulation   |
| CyBench  | `qitos.recipes.benchmarks.cybench`      | Cybersecurity CTF challenges                 |
| CyberGym | `qitos.recipes.benchmarks.cybergym`     | Interactive cybersecurity gym environments   |
| Desktop  | `qitos.recipes.desktop.osworld_starter` | Desktop UI automation                        |

Each recipe may be accompanied by an `eval_config.yaml` in `qitos/recipes/benchmarks/eval_configs/` that specifies recommended defaults (model, max steps, max tokens, scorer).

### Using eval configs

```python theme={null}
import yaml
from pathlib import Path
from qitos.harness import resolve_builtin_preset

config_path = Path("qitos/recipes/benchmarks/eval_configs/gaia.yaml")
with open(config_path) as f:
    config = yaml.safe_load(f)

# Apply defaults from config
model = config["defaults"]["model"]
max_steps = config["defaults"]["max_steps"]

# Combine with a family preset for advisory defaults
preset = resolve_builtin_preset(model)
```

## Why this split matters

This split solves three real problems:

* benchmark runners no longer depend on example files
* one baseline can be reused by examples, docs, and benchmark runners
* future `qitos-recipes` extraction becomes a packaging move instead of a redesign

This separation keeps QitOS viable as a research-first framework.

### W\&B integration

To stream benchmark metrics to Weights & Biases alongside recipe execution:

```python theme={null}
from qitos.tracing import add_trace_processor
from qitos.tracing.wandb_processor import WandbTraceProcessor

processor = WandbTraceProcessor(
    project="qitos-benchmarks",
    tags=["gaia", "eval"],
)
add_trace_processor(processor)
```

See [W\&B Integration](/guides/wandb-integration) for full documentation.

If you are adding a new benchmark family, continue with [Third-party benchmark integration](/guides/third-party-benchmark-integration).