Fullbeam guide

Fullbeam builds a private benchmark from pull requests your team already merged. You use it to answer one question: does a new coding-agent setup do better on your code than the one you use today?

Each merged pull request becomes a task. The task starts from the commit before the change. The pull request's own tests are held back and used to grade the result, so the agent never sees them.

Private preview. Fullbeam is invitation-only today, and Fullbeam runs the benchmarks for you. Results are advisory. They cannot authorize or deploy a release. Contact us for access.

How it works

1. Pick merged pull requests. Fullbeam lists merged pull requests from the repositories you connect.
2. Get a spec that doesn't leak the answer. Fullbeam's AI writes a short spec for each pull request. Fullbeam checks that the spec doesn't give away solution code, new file names, the hidden tests or the pull request itself. You approve the clean specs.
3. Check the task. Fullbeam confirms that the tests fail before the change and pass after it, and that a deliberately broken change fails.
4. Run both setups. Your current setup and the new one each attempt every task several times. Each attempt runs in its own short-lived sandbox.
5. Grade with your tests. A separate step runs the pull request's tests against each attempt. That step never gets model keys.
6. Read the results. You see pass rates, cost, and any task that got worse. Anything Fullbeam could not measure is shown as Unknown or Unavailable, never as zero.

A "setup" is everything that shapes how the agent works: the agent (for example Codex or Claude Code), the model, and the instructions, skills and config files around it. In the app, a saved, versioned setup is called a Stack Release.

What Fullbeam does not do

• Fullbeam does not deploy a setup, change your agents' settings, or approve pull requests. Your own tools (GitOps, MDM, your agent vendor or your internal platform) roll out changes. GitHub stays responsible for review, branch protection and merge.
• Fullbeam does not claim that a setup caused an outcome. It reports what happened on your tasks, with the sample size.
• There is no button to rerun a finished benchmark with a new model. To test a new model on the same tasks, start a new comparison.

Where things are in the app

Open your team workspace. The sidebar has these items:

• Overview: workspace status and the GitHub connection prompt.
• Stack Registry: saved setups (Stack Releases).
• Qualification: start comparisons and read results. The PR-based flow opens from here with Open harness evaluations.
• Canary planning: jumps to the optional rollout plan inside Qualification.
• Regression suite: groups of tasks (suites) used by the setup comparison.
• Cases: your tasks. This is where you import merged pull requests.
• Connections: GitHub, Linear and Jira connections.
• Audit trail: data boundaries, retention, roles and security events.
• Settings, Members and, when enabled, Billing.

Some page titles differ from the sidebar. For example, the Cases page is titled "Regression suite". The steps in these docs use the labels you click.

Who can do what

Workspaces have two roles: Owner and Member. One owner is the Primary Owner, shown on the Members page.

• Members can read results and ask an owner to install the GitHub App.
• Owners see the setup controls. They can install the GitHub App and import and approve specs.
• Many actions need the primary owner. These include adding provider keys, authorizing task validation, creating and freezing suites, and preparing and starting comparisons. Other owners can see these controls, but Fullbeam refuses the action.

To add a teammate, open Members, select Invite Members, enter their email and pick the smallest role they need. Repository access is checked separately through GitHub, so a workspace role alone does not show someone private repository data.

End-to-end walkthrough

1. Connect GitHub

On Overview, select Install the GitHub App and choose the repositories Fullbeam may use. Fullbeam then imports recent pull request history in the background. See Connect GitHub.

2. Build tasks from merged pull requests

Open Cases and use Import merged pull requests. Pick a repository, tick the pull requests you want, and select Infer specs for N pull requests. Review the specs and select Approve all clean specs. Then prepare each task with a verification command and authorize the checks. A task that passes every check is frozen and ready to use. See Build tasks from merged pull requests.

3. Describe the setups you want to compare

How you describe the two setups depends on which comparison you use:

• Harness evaluations: open a pull request in your repository that changes only harness files, such as AGENTS.md, skills or .codex/config.toml. The pull request's base commit is the current setup and its head commit is the new setup.
• Compare your current and proposed setup: capture each setup from Git and save it as a Stack Release. Add your own provider key.

See Set up agents, models and provider keys.

4. Run the comparison

Open Qualification and start one of the two flows. Review the task count, number of attempts and the cost warning, then start the run. See Run a comparison and read the results.

5. Read the results and decide

Look at the pass rate for each setup, the tasks that got worse, and the cost. Check what is marked Unknown or Unavailable before you draw a conclusion. If you decide to switch, roll the change out with your own tools.

Which comparison should I use?

Fullbeam has two comparison flows. They support different agents today.

Harness evaluations (the newer flow):

• Codex only.
• Compares a pull request's base commit (current) with its head commit (new). The pull request may change the model, instructions, skills or Codex config.
• Runs every validated task in the workspace, twice per setup.
• Model calls go through Fullbeam's hosted runner account, not your own key. Fullbeam enables this flow for your workspace during the preview.

Compare your current and proposed setup:

• Codex or Claude Code.
• Both setups must use the same agent, provider, model and network policy. Only instructions and skills can differ. It does not compare models.
• Runs a suite you choose, with a repetition count you choose (default 3).
• Uses your own OpenAI or Anthropic key.

OpenCode is not supported for comparisons in either flow today.

Limits in the private preview

• Access is invitation-only and runs are managed by Fullbeam.
• Results are advisory. The preview does not grant release authority.
• Both flows show the warning that runs make real model calls and may incur charges. The planning budget shown in Harness evaluations is not a spending cap.
• Sandboxes are configured for Daytona's EU target. You cannot choose a region.
• The GitHub check and comment publishing are built but are still being verified against production GitHub.

For problems, see Troubleshooting and FAQ. For data handling, see Security and access.