> ## Documentation Index
> Fetch the complete documentation index at: https://qitor.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Desktop Starter Benchmark

> The official v0.5 desktop benchmark path for OSWorld-compatible starter tasks.

# Desktop Starter Benchmark

`desktop-starter` is the first official multimodal starter benchmark (a standardized evaluation suite for measuring agent performance) family in QitOS.

It is scoped as an **OSWorld-compatible starter**:

* desktop / computer-use task structure
* screenshot + a11y + OCR + UI candidates
* provider-neutral GUI actions
* unified `BenchmarkRunResult` rows
* qita replay / export / visual inspection

It does **not** claim full official OSWorld parity yet.

## Run the starter benchmark

```bash theme={null}
qit bench run \
  --benchmark desktop-starter \
  --split starter \
  --strategy desktop_baseline \
  --model-name qwen-plus \
  --model-family qwen \
  --base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
  --output ./artifacts/desktop-starter.jsonl
```

For a deterministic local smoke run:

```bash theme={null}
qit bench run \
  --benchmark desktop-starter \
  --split starter \
  --strategy desktop_smoke \
  --output ./artifacts/desktop-starter-smoke.jsonl
```

## What the starter benchmark measures

Each result row includes the standard benchmark fields plus desktop-specific metadata:

* success / stop reason
* step count
* action count
* critic (a module that evaluates each step and can trigger retries or stops) count
* token usage
* latency
* failure tags

Current failure taxonomy:

* `perception_failure`
* `grounding_failure`
* `planning_failure`
* `action_selection_failure`
* `execution_environment_failure`
* `stop_completion_failure`

## What makes this an official v0.5 path

The desktop starter benchmark is the first multimodal path where all of these now line up:

* benchmark tasks
* baseline agent
* unified runner output
* trace artifacts (persistent output files from a run)
* qita visual inspection
* docs/tutorial story

That is the release bar for v0.5.

The real benchmark adapter now lives separately under `osworld`.