Pin everything that can change
Record the model, agent version, instruction files, skills, tool schemas, permissions, workflow, checks and runtime image. When any one of them changes, that is a new version to test.
Product / Versioned setups
What is a versioned coding agent setup?
Two teams can use the same model and get very different results, because the agent, instructions, tools, permissions and checks all shape the work. Fullbeam records the whole setup under one version before you benchmark it. We call a versioned setup a Stack Release.
In practice
What this looks like on your code
Every result links back to the setup, the tasks and the commit it came from. If something wasn't measured, you see that too, instead of a number that looks complete.
Leave unknowns as Unknown
Hosted model providers don't publish every version or internal change. Fullbeam records the provider, the time and whatever identity is available, and marks the rest Unknown instead of guessing.
Keep the plan apart from the run
The version describes what your team approved. Where Fullbeam can observe the run, a separate record shows the models, tools, plugins, permissions and subagents the agent actually used, including fallbacks that changed who did the work.
A trail behind every default
When a platform owner promotes, restricts, pauses or replaces a setup, your team can trace that call back to the code it was tested on, the benchmark rules, what was excluded and what happened.
Got a coding agent change coming up?
Whatever setup becomes your default, you can trace it back to the version, the tasks, the benchmark rules and the results behind the call.
Your current setupThe change you want to tryPRs you already mergedAn answer for each kind of work