Skip to content

Learnable, observable agents

Status: proposed. This document changes no code. It puts a training and observability core underneath O4VE and builds the product vertically on top of it. It describes the layers, the interfaces and the first experiment.

Thesis

Agent applications are crowded and quickly absorbed by labs and platforms. What stays defensible is the loop that produces capability and the ability to see inside it:

  1. Environments that are verifiable: deterministic worlds with a grader that is a pure function of what happened.
  2. A minimal harness: the model acts through a few primitives, with no prompt scaffolding doing the reasoning for it.
  3. Training on those environments with reinforcement learning from verifiable rewards (GRPO).
  4. Observability: what changed inside the model when it learned, read from its internals, not inferred from its outputs.

The goal is to understand how models create and reason through environments, not only to make them score higher. A result we cannot explain is not done.

Building vertically

O4VE is one stack, owned top to bottom. Each layer is useful on its own and feeds the one below it:

Layer What it is Status
Application (upstream vertical) The simulation platform already in this repository: control plane (apps/api), console (apps/web), resource model, environment replicas, experiments, attribution Kept. Its docs, pitch and plans remain the product direction
Environments and harness Verifiable worlds, graders, the three-primitive harness, trajectory records New, this RFC
Training GRPO on verifiable rewards, manifests, held-out evaluation New, this RFC
Observability Lens readouts on what the model learned, built on vitrum's tooling New, this RFC

The application layer becomes a customer of the core: its environment replicas are environments under the protocol below, its experiments are training and evaluation runs, and its attribution claims rest on graded, replayable trajectories. Work on the core does not delete or freeze the application layer; it gives it a moat it did not have.

Boundary

Public (urmzd/o4ve, o4ve.com) Private (this repository)
The site and the observatory, which renders published run artifacts Environments, graders, harness, training code, lens code
The artifact schema (below), so published runs can be read Raw trajectories, checkpoints, datasets, experiment logs
Later: a client that speaks the environment protocol The architecture and these RFCs

A run is published by exporting a summary and a sample of its trajectories as artifacts. Nothing else leaves this repository.

Interfaces

Environment protocol

Language-agnostic: JSON messages over a process boundary or HTTP, with a Python binding for training. gymnasia's Gymnasium API is the reference for naming.

Call Returns Rule
spec() name, version, tools (JSON Schema per tool), task ids Stable per version; a change bumps the version
reset(task, seed) first observation, state_digest Same task and seed give the same digest
step(action) observation, done, state_digest Invalid actions return an error observation and change nothing
grade(trajectory) score in [0, 1], criteria [{id, pass, evidence}] Pure function of start state, final state, actions and final answer. No model judges

Environments are worlds we write. The take-home that inspired this direction is not reused: none of its code, contracts, seeds or tasks.

Harness primitives

The model sees an observation and emits text. The harness parses exactly three outcomes and nothing else:

Primitive Meaning
reason Free text before an action. The environment never sees it; the trajectory records it
act(tool, args) One tool call, validated against spec()
answer(text) Ends the episode

No planner, no retries, no hidden system prompt beyond the tool list and the task. A budget of turns and tokens ends a stuck episode with a recorded reason. Keeping the harness this thin is what makes a learned behavior attributable to the model rather than to the scaffolding.

Trajectory record

Content-addressed so a run can be replayed and verified:

{
  "run": "r_...", "env": "helpdesk@1", "task": "t_04", "seed": 7,
  "model": {"base": "...", "checkpoint": "sha256:..."},
  "steps": [
    {"t": 0, "observation": "...", "reason": "...",
     "action": {"tool": "search_tickets", "args": {}},
     "result": "...", "state_digest": "sha256:...", "tokens": 212}
  ],
  "answer": "...",
  "grade": {"score": 1.0, "criteria": [{"id": "c1", "pass": true, "evidence": "..."}]}
}

Lens readouts

Attached per step when a run is traced:

  • Layer-wise vocabulary projections of the residual stream at the reasoning and action tokens (logit lens, and a Jacobian lens where cheaper and better).
  • Attribution of the chosen action to earlier reasoning and observation tokens.
  • Before and after training: which readouts moved at the same step of the same task and seed.

vitrum's tooling (tracing, causal probing, faithfulness coverage, budget-matched manifests) is reused. Its readable-by-construction architectures are not the first bet: they score at chance on in-context recall, so they cannot yet reason through an environment. Capable open models, read post hoc, come first.

Training

GRPO on verifiable rewards:

  • For each task and seed, sample a group of G trajectories, grade each, and use the group-relative advantage (r - mean) / std with a KL penalty to the reference model.
  • Small open models (0.5B to 3B) first, so a run fits a single GPU.
  • Every run writes a manifest: base model, data, environment versions, seed, git SHA, hardware, hyperparameters. A result without a manifest is not reported.
  • Held-out tasks and seeds measure generalization; a gain only on training tasks is reported as such.

The observatory

o4ve.com renders published artifacts so a run can be inspected by anyone:

  • Run summary: reward over training, held-out score, manifest.
  • Episode viewer: each step's observation, reasoning, action and result, with the grade's criteria and evidence.
  • Lens view: readouts at each step, and the before/after difference.

The site reads artifacts in the schema above from a public location. Until the first real run is published, any example it shows is labeled as an example.

First experiment

  1. Two environments we write, each with 10 to 20 tasks and deterministic graders.
  2. A ~1.5B instruction model through the three-primitive harness: baseline success on train and held-out tasks.
  3. GRPO for a fixed budget; report the same scores, with manifests.
  4. Trace matched episodes before and after; publish one run to the observatory with its lens view.

Success is a measured held-out gain plus at least one learned behavior we can point to in the lens readouts.

Alternatives

  • Stay only an application-layer simulation platform. Rejected as the whole strategy: on its own it duplicates agent platforms and is easy to copy. It stays as the upstream vertical on top of the core.
  • Train readable architectures from scratch first. Deferred: vitrum's current variants are bigram-class.
  • Model judges for reward. Rejected for the first experiment: a judge adds a second model to explain. Deterministic graders keep the reward inspectable.

Changes

  1. Environment protocol and two worlds, with graders and tests.
  2. Harness and trajectory records, with replay verification.
  3. GRPO training loop with manifests.
  4. Lens readouts on top of vitrum's tooling.
  5. Artifact export, and the observatory on o4ve.com.
  6. Later: a public client for the environment protocol.

Open questions

  • Which base model family for the first runs, given licensing for published readouts.
  • Where training runs: GCP credits, or a GPU provider for longer runs.
  • How much of each trajectory to publish without exposing environment internals that would let a model overfit the public examples.