> ## Documentation Index
> Fetch the complete documentation index at: https://qitor.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Design Lab: evidence and limitations

> Separate framework correctness, design mechanisms and real-model task outcomes.

The six courses are executable teaching projects, not reproductions of product
internals or paper benchmark scores. A successful tool call is not a successful
research conclusion. The [lab overview](/tutorials/agent-design-lab) explains the
framework/application boundary; the repository execution ledger preserves dated
implementation and validation evidence.

## What the installed consumers prove

| Consumer                   | Observed mechanism                                                                                                   | Not implied                                                            |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| ReAct / PlanAct            | Shared canonical composition, feedback, multiple plan revisions and a static control                                 | Planner training or paper scores                                       |
| Pi-like / Claude-Code-like | Real Docker edits/checks, independent child Session review, join and parent continuation                             | Product parity, malicious-code-proof grading or file rollback          |
| Hermes-like                | Reopened facts/history, revisioned procedures, catalog without bodies, complete selected instructions                | Correct memory selection on every real task                            |
| Voyager-inspired           | Three checked programs, persisted mastery ordering, dependency loading and fresh-process composition/reuse in Docker | Open-ended learned curriculum, Minecraft or a signed skill marketplace |
| Cross-course               | Installed PlanAct + persistent notebook + separately installed Pi tool, with real request compaction                 | A second scheduler or another execution kernel                         |

Python 3.10 and 3.12 outside-checkout consumers passed. The horizontal consumer
made 12 requests, 10 plan revisions and 9 distinct compaction receipts; notebook
records remained after cleanup. Scripted models prove mechanisms, not intelligence.
The final live matrix is tracked separately below, including non-passing attempts.

## Framework and course defects found by the projects

* Custom `agent_factory` now enters the resource-owning composition root.
* Persistent skill revisions replace duplicated course stores; selected skill
  instructions are not silently truncated. Memdir deletion is explicit.
* Artifact references are evidence, not privileged developer instructions.
* Child workspace restoration cannot overwrite the authorized conversation.
* Same-owner pause continuation traverses restoring and persists its final head.
* Encoded JSON scalar inspection distinguishes escaped program text from real
  private paths. Session contract failures do not silently advance the loop.
* Native preflight blocks close their tool batches, including blocked/executed
  siblings, without duplicate compatibility results.
* Course fixes cover packaged history identifiers, report/search instructions and
  explicit child budget allocation. Those remain application policy, not defaults
  imposed on every agent.

## Real Qwen experiment contract

Three tasks per project, three repetitions, plus static-plan/no-memory/no-skills
controls. Hermes and Voyager perform learning before a new-process recall run.
Controls start from the same post-learning store bytes, before default recall
writes another episode. A failed seed is not evidence of a causal benefit.

The bounded qualification configuration uses 24 steps/requests and 600 seconds
per task, 10,240 output tokens, and at most four concurrent project/repetition
groups. Public project defaults remain configurable at 80/80/3600. There is no
total experiment request or cost quota. Provider/model configuration is explicit;
TLS verification is not disabled and another model is not silently substituted.

The matrix completed **99 invocations**: 54 default task trials, 18 preceding
learning runs and 27 controls. No whole-lab live qualification claim is made.

| Project          | Default task passed / attempted | Learning       | Control                   |
| ---------------- | ------------------------------- | -------------- | ------------------------- |
| ReAct            | 9 / 9                           | Not applicable | Shared tasks with PlanAct |
| PlanAct          | 7 / 9                           | Not applicable | Static: 9 / 9             |
| Pi-like          | 2 / 9                           | Not applicable | Not run                   |
| Claude-Code-like | 2 / 9                           | Not applicable | Not run                   |
| Hermes-like      | 0 / 9                           | 8 / 9          | No memory: 0 / 9          |
| Voyager-inspired | 3 / 9                           | 6 / 9          | No skills: 1 / 9          |

These are small, bounded teaching experiments, not rankings or evidence that
dynamic planning, memory or skills generally improve task success. One Voyager
learning attempt rejected snapshot content and produced no final report; it is
counted as non-passing. Controller timeouts and manual interventions in this final
matrix were zero; declared Session budget stops still count as failures.

All 18 groups preserve installed-file digests. Relative to the final source,
the live candidate differs only in `_action_runtime.py` (the subsequent mixed
blocked/executed compatibility-result deduplication) and the Hermes course agent
(subsequent literal-search/report-shape clarification). Those changes passed
permanent deterministic and installed checks; the 99-run matrix was **not** rerun
on them. Therefore it does not qualify the final source's complete live behavior.

The repository ledger includes a counts-only aggregate and installed identity
document digests. Earlier source groups and every failure remain separately in
private storage; none is replaced by a best-of retry. No provider payload or
sensitive path is published.

## How to inspect and learn from a failure

Each run writes `report.json`, a Session identifier and a canonical trajectory in
the chosen private run directory. Use the project's `inspect` command, then follow
model request, tool terminal, verification, snapshot and final events. Compare
the independent checker with the last completed operation, not the final prose.

Distinguish these observed outcomes:

1. **Framework defect:** an orphan batch or wrong child input required a permanent
   counterexample and framework fix; prompting around it was not accepted.
2. **Transport failure:** `provider_connection_failed` can occur after code checks
   pass. Without a final outcome, the end-to-end attempt still fails.
3. **Task failure:** final prose with wrong numbers, missing citations, unsuccessful
   memory recall or unverified code is not success.
4. **Budget stop:** a loop reaching its declared guard remains non-passing; no
   silent budget extension or hidden successful continuation is credited.

Raw trajectories may contain model output, source code and private information.
Keep them outside Git. `scripts/report_agent_design_lab.py` emits allowlisted
counts only; it does not authorize publication of the underlying data.

Remaining boundaries include bounded synthetic tasks, cooperative cancellation,
local Docker rather than VM isolation, no external-effect exactly-once guarantee,
non-transactional multi-file Memdir forgetting and no automatic signed dependency
resolution for skills. These projects are research starting points, not a claim
that every possible agent application is qualified.