Run a comparison and read the results

What it does

A comparison runs your validated tasks with the current setup and the new setup, several times each. Every attempt runs in its own short-lived sandbox. After the agent finishes, Fullbeam runs the pull request's hidden tests against the result. Then it shows how often each setup passed, which tasks got worse, and what it cost.

Fullbeam has two comparison flows. Pick one using Which comparison should I use? in the guide.

Before you start

• You have validated tasks. See Build tasks from merged pull requests.
• You have described both setups. See Set up agents, models and provider keys.
• The workspace's primary owner prepares and starts the run.
• Runs make real agent and model calls and may incur charges.

Harness evaluations (pull request based, Codex)

Steps

1. Open Qualification and select Open harness evaluations.
2. Select New evaluation.
3. Choose the Repository and enter the Pull request number of your harness-only pull request. Select Resolve.
4. Check the changed harness files and the change label, such as HARNESS_ONLY or MODEL_ONLY. The base commit is the current side and the head commit is the new side. There is no baseline to pick.
5. Set the Planning budget per attempt (USD). The default is 20. It is used for planning only and is not a spending cap.
6. Decide whether to keep attempt files. If you tick the option, Fullbeam keeps encrypted attempt inputs and changes for seven days so you can inspect them. If you don't, the changes are discarded and can't be inspected or published later.
7. Select Prepare comparison. Preparing freezes the comparison for review. It does not start any work.
8. Under Review this exact comparison, check:
   • the number of attempts (tasks × 2 configurations × 2 repetitions);
   • the list of tasks;
   • the Time limit per attempt;
   • the Planning budget per attempt and Total planning budget.
9. Tick the acknowledgement and select Start comparison.

To stop a run, select Cancel comparison. Select Edit comparison settings to change something before you start.

What you'll see

The Harness evaluations area has three views:

• Overview: cards for Evaluation runs, Verified pass rate, Model cost estimate and Total tokens, a pass-rate chart, and a Candidate models table.
• Run history: every run, with search and a status filter (In progress, Completed, Needs attention).
• Compare runs: pick two runs to compare candidate pass rate, cost and tokens per candidate attempt, and Task-by-task results.

While a run is going, its page shows an In progress view with attempts so far. When it finishes, it shows five tabs:

• Outcomes: passes for current and new, and one row per attempt.
• Tasks: each frozen task, the prompt the agent got (Show the frozen prompt), and the two check lists: Preserve existing behaviour and Fix the reported behaviour.
• Changes: what each attempt changed, if you kept attempt files. From here you can select Create draft review PR to open the attempt's changes as a draft pull request.
• Usage: tokens and an estimated cost breakdown.
• Configuration: the two setups side by side, with the model, reasoning effort and frozen commit.

Each attempt gets one of these outcomes: Pass, Functional fail, Regression, Policy violation, Agent timeout, Config error, Infrastructure error or Cancelled.

A workspace with no runs yet sees example rows under a Fixture data banner. They are not your results.

Limits

• Codex only. Model calls go through Fullbeam's hosted runner account, not your own key.
• It always runs every validated task in the workspace, with 2 repetitions per setup.
• The planning budget is not a spending cap. Charges may exceed it.
• Provider runtime versions, network isolation and reasoning effort are not verified.
• Costs are estimates from observed token counts and a fixed rate table. They are not an invoice. Reasoning tokens are always Unknown.
• Pass rate is passing attempts divided by all attempts in the run. Attempts that hit an infrastructure error still count in that total, so check the Outcomes tab before comparing numbers.

Compare your current and proposed setup (Codex or Claude Code)

Steps

1. Open Qualification and select Start with Codex or Start with Claude Code.
2. Fill in the form:
   • Comparison name and Decision question;
   • Current setup and Proposed setup;
   • Regression suite;
   • Decision rule. If none exists, create and freeze one first in the Release Policy builder under Advanced release and connection controls;
   • your provider credential;
   • Repetitions (default 3).
3. Review the count, budget and allowance summary.
4. Tick the acknowledgement and select Save comparison for review.
5. Fullbeam runs readiness checks. When they pass, select Freeze and launch.

Saved runs appear under Saved comparisons. Use Compare next change to start another comparison from one you already ran.

What you'll see

Results appear under Agent Stack Cost Qualification:

• Comparable tasks and any Observed regression;
• cost per accepted task for the current setup (shown as baseline) and the new one (shown as candidate), and how the two compare;
• Evidence coverage;
• a recommendation for each kind of work: Move to canary, Restrict to selected workloads, Keep baseline or Continue evaluation.

"Cost per accepted task" includes the cost of every attempt, including failed ones, divided by the number of tasks that passed under the decision rule. When a value is missing, Fullbeam shows Unavailable. The Annual projection is always Unavailable today.

If a run fails, is cancelled, or has attempts with an infrastructure error, select Review and run again. This creates a new run. The old evidence is kept.

Optional: reports, overrides and a rollout plan

Under Advanced release and connection controls an owner can:

• download the decision report as JSON, Markdown or PDF;
• record an override with a reason and an expiry;
• prepare a rollout plan from Canary planning and seal it.

A sealed plan is a record for your own deployment tools. Fullbeam does not send it anywhere or deploy anything.

Limits

• Both setups must use the same agent, provider, model and network policy. This flow compares instructions and skills, not models.
• The results are marked as preview, shadow-only evidence. They inform your decision. They do not authorize a release.
• The preview's Technical limitations note says that exact reasoning enforcement and the independent hidden-input verification boundary are not yet available in this flow.
• Setup commands in a fullbeam.yaml can reach declared public package registries. Only use repositories you trust.

How to read any result

• Unknown or Unavailable means Fullbeam did not measure it. It never means zero. A number with a trailing * is a subtotal over partial data.
• Look at regressions first. A cheaper setup that fails more tasks is not an improvement.
• Results describe your tasks and your sample size. They don't prove that a change caused an outcome elsewhere.
• To test a new model when it ships, start a new comparison. With Harness evaluations, open a pull request that changes the model in .codex/config.toml. There is no one-click rerun of a finished benchmark.