An agent runs an experiment

A language model designs and runs an experiment on the projectile simulation through a short, bounded list of tools, and CI checks the recorded run by recomputing every tool result without a model or a key.

Course

Classical mechanics, lesson 14 of 28

Slot

AI0, number 1 of the 9 slots of the path Physics reasoning and scientific agents

Methods

Simulation, LLM agents

Difficulty

Intermediate, level 2 of 3

What you’ll learn

• Give a language model a short, bounded list of tools to run a simulation
• Read a recorded agent run as an experiment
• Recompute every tool result of a run without a model or a key
• Say what a replay does and does not show about an agent’s conclusion

Prerequisites

Projectile motion with quadratic drag.

Before you begin

• Python: functions, dictionaries, and environment variables
• Large language models: a program that continues a text

Related

none

Previous

Induction: magnetic flux and the electromotive force

Next

A counterexample hunter

With scripts, this page keeps a record in your browser of whether you completed the lesson. Nothing is stored without them.

With scripts, this place suggests what to do next, with the reason, from the progress saved in your browser. Without them, follow the learning path.

At what angle should a tennis ball be thrown to land as far away as possible? Without air the answer is forty-five degrees, as the projectile lesson shows. With drag the ball loses speed along its whole path, the best angle moves lower, and no formula gives it. The projectile lesson answered by scanning the angle and running the simulation at each one. This lesson hands the choices of that scan to a language model. The model reads the question, asks for simulations, reads their results, and decides what to ask next, until it believes it can answer. The program around the model is small, and this lesson builds all of it: a loop, a list of the functions the model may call, and a check on every call.

What makes this safe to teach is what the model is not given. It cannot write or run code, open a file, start a program, or reach the network. It can only name one of two functions of this site’s simulation library and give numbers within stated bounds. Everything else it says is text.

What makes it checkable is a recording. A live model is not repeatable, needs a key, and costs money, so the site never calls one. The lesson shows a recorded run: the model’s messages are replayed from a file in the repository, but every simulation the model asked for is run again by the build, and the page shows the new results. If the simulation changes, the build fails instead of showing numbers that no code produced. What that check proves about the model’s conclusion, and what it does not, is the subject of the second half of the lesson.

Assumptions

1. The question has a checkable answer. The agent looks for the launch angle of the longest throw. The simulation of the projectile lesson can answer that, and so can you. For a question without a way to check the answer, an agent’s confident conclusion would be worth nothing.
2. The model is untrusted. Its output is data, whatever it says and however it is worded. If the harness treated the output as something to run, a wrong or manipulated answer would act on your machine. Here it can only select a function from a list.
3. The allowlist bounds are the safety boundary. A bound that is too wide lets one call run for an hour or return nonsense: with a time step of zero the simulation would never end. The bounds on the time step below are there for that reason.
4. The step limit ends the run. A model can keep asking forever. The loop stops after a fixed number of messages whether or not the model is done.
5. A recording is a record, not a proof. It shows what one model once did with this question. It does not show what the model will do again, and it does not show that the model’s reasoning was sound. The numbers in it are checked; the words are not. The page says which model made the recording and how, and a reader who doubts a recording can read the committed file.
6. The key stays on your machine. The live run uses your own provider key, read from an environment variable. Anything you commit or paste can be copied, and a key that leaks can be used by others at your cost.

How the agent is built

An agent here is three parts and a loop.

1. A model client is the only part that knows a provider. Given the conversation so far and a description of the tools, it returns the model’s next message: some text and zero or more tool calls.
2. An allowlist is the list of tools. Each tool wraps one function of the simulation library with named, numeric arguments and a lower and upper bound for each.
3. The loop asks the client for a message, checks each tool call against the allowlist, runs the calls that pass, sends the results back, and repeats. It stops when the model asks for no tool or when it has reached the step limit.

The parts are easier to hold in mind as a flow of messages, with the line drawn between what is trusted and what is not:

def draw_tool_flow() -> Figure:
    """The model, the harness, the allowlist, and the simulation, as a
    sketch."""
    fig, ax = diagrams.canvas(6.4, 3.0, (-0.2, 12.6), (-1.6, 3.4))
    names = (("model", 0), ("harness", 1), ("allowlist", 2), ("simulation", 3))
    centres = {name: (1.1 + 3.4 * i, 2.0) for name, i in names}
    for name, _ in names:
        diagrams.state_box(
            ax,
            centres[name],
            name,
            width=2.2,
            height=0.9,
            colour=0 if name == "model" else 1,
        )
    for left, right, label in (
        ("model", "harness", "tool call"),
        ("harness", "allowlist", "name, bounds"),
        ("allowlist", "simulation", "only if it passes"),
    ):
        (x0, y), (x1, _) = centres[left], centres[right]
        diagrams.arrow(ax, (x0 + 1.15, y), (x1 - 1.15, y), colour=1)
        diagrams.text(
            ax, ((x0 + x1) / 2 + (0.0 if left != "model" else 0.9), y + 0.65), label
        )
    diagrams.line(ax, [(2.8, 3.1), (2.8, -0.9)], colour="#57606a", linestyle=":")
    diagrams.text(ax, (2.6, 3.2), "untrusted", ha="right", colour="#57606a")
    diagrams.text(ax, (3.0, 3.2), "trusted", ha="left", colour="#57606a")
    back = -0.4
    diagrams.line(
        ax,
        [
            (centres["simulation"][0], 1.5),
            (centres["simulation"][0], back),
            (centres["model"][0] + 0.6, back),
        ],
        colour=0,
        linestyle="--",
    )
    diagrams.arrow(
        ax,
        (centres["model"][0] + 0.6, back),
        (centres["model"][0] + 0.6, 1.5),
        colour=0,
        linestyle="--",
    )
    diagrams.text(ax, (6.3, back - 0.4), "result, as text", colour=0)
    diagrams.arrow(
        ax,
        (centres["allowlist"][0] - 0.4, 1.5),
        (centres["harness"][0] + 0.4, 1.5),
        colour=0,
        linestyle="--",
    )
    diagrams.text(
        ax, (centres["harness"][0] + 1.7, 0.95), "rejected, with a reason", colour=0
    )
    fig.tight_layout()
    return fig

src/pbc/mechanics/experiments/agent_experiment.py, lines 52 to 110, read from the repository when this page was built.

Figure 1: conceptual One turn of the loop. The model only produces text and tool calls; the harness checks each call against the allowlist, and only a call that passes reaches the simulation. The result goes back to the model as text. The model is untrusted, and the allowlist is the boundary.

The diagram is a sketch, not a measurement. It answers the question the paragraph above raises: if the model can only ask, what stops a bad request? Every request passes the allowlist on the way in, and everything that comes back, a result or a refusal, is text the model reads and cannot execute.

What the agent may and may not do

This is the complete list for this lesson.

• range_with_drag: Simulate the throw of the tennis ball with quadratic air drag and return the horizontal distance in metres at which it lands. The result depends on dt.
  • speed: launch speed in m/s; allowed values 1 to 50
  • angle: launch angle in degrees above the horizontal; allowed values 1 to 89
  • dt: simulation time step in s (smaller is slower); allowed values 0.001 to 0.05
• range_without_drag: The exact range in metres of the same throw in vacuum, v^2 sin(2 angle) / g.
  • speed: launch speed in m/s; allowed values 1 to 50
  • angle: launch angle in degrees above the horizontal; allowed values 1 to 89

The agent may:

• call range_with_drag and range_without_drag, with the arguments above;
• make at most 12 model messages, and at most four tool calls in one message;
• write plain text as its answer.

The agent may not, and the harness has no code that could let it:

• run code the model wrote, or evaluate any text the model sent;
• start a program or a shell command;
• read or write a file;
• open a network connection. The one network request of a live run is the client’s own call to the provider;
• change the ball: its mass is 58 g and its drag constant 0.00106 kg/m, fixed in the lesson code;
• call a function that is not in the list, or call one with an argument outside its bounds. Such a call is rejected without being executed, and the model is told why.

Each bound has a reason. The launch speed and the angle are bounded to the range in which the simulation of the projectile lesson means something: the angle excludes the throw along the ground and the throw straight up, whose ranges are zero or degenerate. The time step has two bounds for two different reasons: the lower bound limits the cost of one call, because the number of steps grows as the step shrinks, and the upper bound limits the error, because explicit Euler is first order and a coarse step is visibly wrong (the first exercise measures both ends). A bound is a decision about what the model may ask for, not a property of the physics.

A boundary like this is only as good as its tests. The unit tests of the harness try tool names such as os.system and eval, numbers that are not finite, strings where a number belongs, extra arguments, and code in the place of a value. A test also reads the source of the harness and fails if it imports a module that starts programs or opens connections, or calls eval, exec, or open, outside the one module that reads and writes the replay fixture.

Keys

A live run needs a key from the provider. The rules are the same for any provider:

• Keep the key in an environment variable of your shell, here OPENAI_API_KEY. Never put it in code, in a notebook, or in a file you commit. The file .env.example in the repository lists the name and nothing else.
• The client reads the variable once, into an object that prints as [redacted]. The loop removes the key from every message the model returns, text and tool calls alike, before the message reaches a transcript, the screen, or the next request, even if the model’s own text repeats it. The client removes it from error messages too. A test sets a key and checks the returned transcript, the printed output, the log, the exception text, and the saved recording.
• Nothing on this site, and no workflow of the repository, holds a key. The build never contacts a provider.

What changes in a live run

The recorded run below is one run of one model. Your live run will differ:

• The path differs. The model may try other angles in another order, ask for other time steps, or take more or fewer steps. Language models do not repeat themselves.
• The numbers do not differ for the same question. If your run asks for the range at 40 degrees with a time step of 0.002 s, it gets exactly the number on this page, because the simulation is deterministic.
• The conclusion should agree to the precision the question asks for, the best angle to within a degree. If it does not, one of the runs is wrong, and the numbers in the transcript let you find out which.

Go deeper: Function calling (OpenAI, Handling function calls, and Strict mode; API documentation.)

How a provider’s interface hands a tool call to your program: a name and JSON-encoded arguments that you execute and answer. It documents the round trip of the loop below for the provider the lesson’s client uses, and it describes a strict mode for schemas that the lesson does not rely on, because its allowlist checks every call itself. The lesson does not need the page.

What an agent’s claim is, and how it is tested

The model’s last message holds a claim: a sentence such as “the best angle is between two values”. It was produced by a program that continues a text, so it has the authority of the evidence behind it and no more. Three things can be tested, and they are different tests.

1. Did the numbers come from the code? The replay recomputes every tool result and compares it with the recorded one. A figure the model cites that the simulation does not give is found, because the tool results are what the page shows. It does not reach a number the model states in its own words without a tool call.
2. Does the conclusion follow? The model reasons in text, and the build does not check that reasoning. What it can do is test the conclusion against a computation the model did not control: a scan of the same simulation, made by this page, that finds the best angle independently.
3. Is the model right about the world? Nothing here measures a ball.

These three kinds of evidence have different names on this site. What a simulation recomputed by the build says about the simulated ball is numerical evidence, and a claim that rests on it is labelled numerically-verified. A statement from a measurement of a real ball would be experimental, which this lesson has none of. A statement proved for every case by a proof assistant would be formal, which the Lean lessons give. The agent’s conclusion, where the build recomputes the numbers it rests on, is numerical, and the lesson labels only that part.

Code

The model client interface is everything the harness knows about a provider.

class ModelClient(Protocol):
    """The only provider-facing surface of the harness.

    A client names its model, holds its credentials, and returns the next
    assistant message for a conversation. The harness removes every secret in
    ``secrets`` from each message before anything else sees it. Switching
    provider means writing one class with this shape (``pbc.agents.clients``).
    """

    model: str
    secrets: tuple[Secret, ...]

    def complete(
        self, system: str, conversation: Sequence[Turn], tools: Sequence[ToolSpec]
    ) -> ModelMessage: ...

src/pbc/agents/messages.py, lines 70 to 84, read from the repository when this page was built.

A tool validates a call before anything runs. Arguments must be exactly the declared ones, each a finite number within its bounds.

@dataclass(frozen=True)
class Tool:
    """An allowlisted function of ``pbc``.

    ``function`` takes the validated parameters as keyword arguments and
    returns a mapping from result names to numbers.
    """

    name: str
    description: str
    parameters: tuple[Parameter | Choice, ...]
    function: Callable[..., Mapping[str, float]]

    def validate(self, arguments: Any) -> dict[str, float | int | str]:
        """The arguments if they are exactly the declared ones, each within
        its bounds; otherwise raise ``ToolRejected``."""
        if not isinstance(arguments, dict):
            raise ToolRejected(
                f"the arguments must be an object, got {type(arguments).__name__}"
            )
        declared = {parameter.name: parameter for parameter in self.parameters}
        unknown = sorted(str(key) for key in arguments if key not in declared)
        if unknown:
            raise ToolRejected(f"unknown arguments: {', '.join(unknown)}")
        missing = sorted(name for name in declared if name not in arguments)
        if missing:
            raise ToolRejected(f"missing arguments: {', '.join(missing)}")
        return {
            name: parameter.validate(arguments[name])
            for name, parameter in declared.items()
        }

    def spec(self) -> ToolSpec:
        return ToolSpec(
            name=self.name,
            description=self.description,
            parameters={
                "type": "object",
                "properties": {p.name: p.schema() for p in self.parameters},
                "required": [p.name for p in self.parameters],
                "additionalProperties": False,
            },
        )

src/pbc/agents/tools.py, lines 87 to 129, read from the repository when this page was built.

The allowlist is a dictionary: the name the model sends is looked up in it and nowhere else. A name that is not a key is rejected.

class Allowlist:
    """The tools of one lesson, by name."""

    def __init__(self, tools: tuple[Tool, ...], budget: int | None = None):
        if budget is not None and budget < 1:
            raise ValueError(f"the budget must be at least 1, got {budget}")
        self.budget = budget
        self._used = 0
        names = [tool.name for tool in tools]
        if len(set(names)) != len(names):
            raise ValueError(f"duplicate tool names in {names}")
        self._tools = {tool.name: tool for tool in tools}

    @property
    def tools(self) -> tuple[Tool, ...]:
        return tuple(self._tools.values())

    def tool(self, name: str) -> Tool:
        """The tool called ``name``; ``KeyError`` if there is none."""
        return self._tools[name]

    def specs(self) -> tuple[ToolSpec, ...]:
        return tuple(tool.spec() for tool in self.tools)

    def dispatch(self, call: ToolCall) -> ToolResult:
        """Validate ``call`` and run it, or return the reason it was refused.

        The name is only a key into the allowlist: a name that is not in it
        is rejected without looking anything else up. When the allowlist has a
        budget, the calls that pass validation are counted, and one beyond the
        budget is refused unexecuted; a rejected call costs nothing. A function that
        refuses its input (``ValueError``, ``RuntimeError``) gives an error
        result as well; any other exception is a bug and propagates.
        """
        tool = self._tools.get(call.name) if isinstance(call.name, str) else None
        if tool is None:
            return self._refused(call, f"{call.name!r} is not an allowed tool")
        try:
            arguments = tool.validate(call.arguments)
        except ToolRejected as reason:
            return self._refused(call, str(reason))
        if self.budget is not None and self._used >= self.budget:
            return self._refused(
                call, f"the budget of {self.budget} experiments is spent"
            )
        self._used += 1
        try:
            content = dict(tool.function(**arguments))
        except (ValueError, RuntimeError) as reason:
            return self._refused(call, str(reason))
        return ToolResult(call_id=call.id, name=call.name, ok=True, content=content)

    @staticmethod
    def _refused(call: ToolCall, reason: str) -> ToolResult:
        return ToolResult(
            call_id=str(call.id),
            name=str(call.name),
            ok=False,
            content={"error": reason},
        )

src/pbc/agents/tools.py, lines 132 to 191, read from the repository when this page was built.

The loop:

def run(
    client: ModelClient,
    allowlist: Allowlist,
    system: str,
    task: str,
    max_steps: int,
    on_step: Callable[[Step], None] | None = None,
) -> Transcript:
    """Run the agent for at most ``max_steps`` model messages.

    Each message is first cleared of the client's secrets, so that a key the
    model repeats reaches no transcript, callback, or later request. Each tool
    call of each message is then validated against ``allowlist`` and run if
    it passes; the results go back to the model in the next request.
    """
    if max_steps < 1:
        raise ValueError(f"max_steps must be at least 1, got {max_steps}")
    conversation: list[Turn] = [UserTurn(task)]
    steps: list[Step] = []
    finished = False
    for _ in range(max_steps):
        message = sanitize(
            client.complete(system, conversation, allowlist.specs()), client.secrets
        )
        results = []
        for number, call in enumerate(message.tool_calls):
            if number >= MAX_CALLS_PER_MESSAGE:
                results.append(
                    ToolResult(
                        call_id=str(call.id),
                        name=str(call.name),
                        ok=False,
                        content={
                            "error": f"at most {MAX_CALLS_PER_MESSAGE} tool calls"
                            " per message"
                        },
                    )
                )
            else:
                results.append(allowlist.dispatch(call))
        step = Step(message, tuple(results))
        steps.append(step)
        if on_step is not None:
            on_step(step)
        conversation.append(message)
        conversation.extend(results)
        if not message.tool_calls:
            finished = True
            break
    return Transcript(
        model=client.model,
        system=system,
        task=task,
        steps=tuple(steps),
        finished=finished,
    )

src/pbc/agents/loop.py, lines 61 to 116, read from the repository when this page was built.

Switching provider

The harness knows a provider by one value, PROVIDER, in src/pbc/agents/providers.py. To use another provider, write one module src/pbc/agents/clients/<name>.py with a function make_client() that returns an object of the shape above, and set PROVIDER to "<name>". The OpenAI client in src/pbc/agents/clients/openai.py is the example to copy: it reads its own key variable, builds a request from the conversation, reads the response, and imports its SDK nowhere else. Nothing outside the client names its key variable, so the other provider’s key is the only one you need. The replay client, the allowlist, and the tests do not change.

def load_client(provider: str = PROVIDER) -> ModelClient:
    """The live client of ``provider``, built from the key in the environment."""
    if not _NAME.fullmatch(provider):
        raise ValueError(f"{provider!r} is not a provider name")
    module = importlib.import_module(f"pbc.agents.clients.{provider}")
    return module.make_client()

src/pbc/agents/providers.py, lines 29 to 34, read from the repository when this page was built.

Replay

The replay client returns the recorded messages. The function replay runs the loop with it, so every tool call goes through the allowlist and the simulation again, and then compares each recomputed result with the recorded one.

def replay(
    recording: Recording,
    allowlist: Allowlist,
    *,
    system: str,
    task: str,
    tolerance: float,
    max_steps: int,
    rerecord: str,
) -> Transcript:
    """Run the recording again and check the recomputed tool results.

    Returns the transcript of the replay, whose tool results are the
    recomputed ones. Raises ``ReplayMismatch`` when the lesson's task or
    system prompt changed, or a recomputed result differs from the recorded
    one by more than ``tolerance`` (relative and absolute), and says how to
    re-record with ``rerecord``, the command to run.
    """
    how = f" If the change is intended, record the run again with: {rerecord}"
    if recording.task != task or recording.system != system:
        raise ReplayMismatch(
            "the task or system prompt of the lesson differs from the recording." + how
        )
    transcript = run(ReplayClient(recording), allowlist, system, task, max_steps)
    if len(transcript.steps) != len(recording.steps):
        raise ReplayMismatch(
            f"the replay has {len(transcript.steps)} steps, the recording"
            f" {len(recording.steps)}." + how
        )
    for number, (was, now) in enumerate(
        zip(recording.steps, transcript.steps, strict=True), start=1
    ):
        for old, new in zip(was.results, now.results, strict=True):
            difference = _differences(old, new, tolerance)
            if difference is not None:
                raise ReplayMismatch(
                    f"step {number}, tool {new.name}: {difference}"
                    f" (tolerance {tolerance:g})." + how
                )
    return transcript

src/pbc/agents/replay.py, lines 175 to 214, read from the repository when this page was built.

Worked examples

The allowlist refuses what it should

These calls are what a misbehaving or manipulated model might send. None of them reaches a simulation.

def misbehaving_calls(experiment: Experiment) -> None:
    """Six calls that a misbehaving model might send, through the allowlist;
    record the table."""
    allowlist = tools()
    attempts = [
        ToolCall("1", "range_with_drag", {"speed": 25, "angle": 40, "dt": 0.01}),
        ToolCall("2", "os.system", {"command": "echo hello"}),
        ToolCall("3", "range_with_drag", {"speed": 25, "angle": 40, "dt": 0.0}),
        ToolCall("4", "range_with_drag", {"speed": 25, "angle": 40}),
        ToolCall("5", "range_with_drag", {"speed": "25", "angle": 40, "dt": 0.01}),
        ToolCall("6", "range_without_drag", {"speed": 25, "angle": 40, "code": "1+1"}),
    ]
    rows = []
    for attempt in attempts:
        result = allowlist.dispatch(attempt)
        shown = ", ".join(
            f"{key}={value!r}" for key, value in attempt.arguments.items()
        )
        rows.append(
            (
                attempt.name,
                shown,
                result.content["range_m"] if result.ok else None,
                "ran" if result.ok else result.content["error"],
            )
        )
    experiment.table(
        "misbehaving-calls",
        rows,
        [
            ("tool the model named", "", ""),
            ("arguments", "", ""),
            ("range", "m", ".3f"),
            ("outcome", "", ""),
        ],
        "Six calls that a misbehaving model might send, and what the allowlist did "
        "with each; the range is empty where nothing ran.",
    )

src/pbc/mechanics/experiments/agent_experiment.py, lines 113 to 150, read from the repository when this page was built.

Six calls that a misbehaving model might send, and what the allowlist did with each; the range is empty where nothing ran.

tool the model named	arguments	range (m)	outcome
range_with_drag	speed=25, angle=40, dt=0.01	35.789	ran
os.system	command='echo hello'		'os.system' is not an allowed tool
range_with_drag	speed=25, angle=40, dt=0.0		dt must be between 0.001 and 0.05, got 0
range_with_drag	speed=25, angle=40		missing arguments: dt
range_with_drag	speed='25', angle=40, dt=0.01		speed must be a number, got str
range_without_drag	speed=25, angle=40, code='1+1'		unknown arguments: code

Only the first call ran. The others were turned away with a reason, which is also what the model reads in its next request. The boundary does not depend on the model meaning well: it treats every call the same way.

A recorded run, replayed

The cell below replays the committed recording. The model’s messages are the recorded ones. Each tool result is computed by this build, now.

    recording, transcript = replay_fixture(LESSON)
    experiment.display("run", RunDisplay(recording, transcript))

src/pbc/mechanics/experiments/agent_experiment.py, lines 370 to 371, read from the repository when this page was built.

Recorded run. The model messages below were recorded from claude-sonnet-5-5 (run by Claude Code in a session, not through a provider API) on 2026-10-08. This build did not call a model. It ran every tool call again; the results shown are the ones it computed.

Step 1. Model message (recorded):

I will start with the vacuum range at 45 degrees as a reference, and scan the angle with drag at a coarse step dt = 0.01 s: 35, 40 and 45 degrees.

• range_without_drag(speed=25, angle=45): recomputed by this build: range_m = 63.7323
• range_with_drag(speed=25, angle=35, dt=0.01): recomputed by this build: range_m = 35.2776
• range_with_drag(speed=25, angle=40, dt=0.01): recomputed by this build: range_m = 35.7893
• range_with_drag(speed=25, angle=45, dt=0.01): recomputed by this build: range_m = 35.4501

Step 2. Model message (recorded):

The range peaks near 40 degrees; a parabola through 35, 40 and 45 degrees puts the maximum near 40.5. But these values depend on dt, so I repeat 40, 41 and 45 degrees at dt = 0.002 s before I trust them.

• range_with_drag(speed=25, angle=40, dt=0.002): recomputed by this build: range_m = 35.7379
• range_with_drag(speed=25, angle=41, dt=0.002): recomputed by this build: range_m = 35.739
• range_with_drag(speed=25, angle=45, dt=0.002): recomputed by this build: range_m = 35.4084

Step 3. Model message (recorded):

Going from dt = 0.01 to 0.002 s moved the range at 40 degrees by about 0.05 m, so the coarse step was not converged. The ranges at 40 and 41 degrees now agree to 0.001 m, which puts the maximum between them. I check convergence once more at the smallest allowed step, dt = 0.001 s, for 40, 41 and 45 degrees.

• range_with_drag(speed=25, angle=40, dt=0.001): recomputed by this build: range_m = 35.7315
• range_with_drag(speed=25, angle=41, dt=0.001): recomputed by this build: range_m = 35.7328
• range_with_drag(speed=25, angle=45, dt=0.001): recomputed by this build: range_m = 35.4032

Step 4. Model message (recorded):

Conclusion: the best launch angle is between 40 and 41 degrees, about 40.5, with a range of about 35.73 m. Evidence: at dt = 0.001 s the ranges are 35.7315 m at 40 degrees and 35.7328 m at 41 degrees, and they differ by under 0.002 m, so the maximum lies between them; a parabola through 35, 40 and 45 degrees at dt = 0.01 s gave 40.5 as well. The time step has nearly converged: from dt = 0.002 s to 0.001 s the range changed by 0.006 m at 40 degrees and 0.005 m at 45 degrees, against 0.05 m from 0.01 s to 0.002 s. At 45 degrees the range is 35.403 m, so the best angle lands the ball about 0.33 m (0.9 per cent) further. Without drag the best angle would be 45 degrees and the range 63.7 m, so drag lowers the best angle and nearly halves the range.

The model ended the run with a message that requests no tool.

The messages come from claude-sonnet-5-5 (run by Claude Code in a session, not through a provider API), recorded on 2026-10-08. A different model or a later date would write different messages.

The check that makes the page honest

Two things enter the replay and one comes out of it:

def draw_replay_check() -> Figure:
    """What the replay compares, as a sketch."""
    fig, ax = diagrams.canvas(6.4, 3.4, (-0.2, 12.6), (-1.0, 4.4))
    diagrams.state_box(
        ax, (1.4, 3.4), "recorded messages", width=2.8, height=0.8, colour=0
    )
    diagrams.state_box(
        ax, (1.4, 0.6), "recorded results", width=2.8, height=0.8, colour=0
    )
    diagrams.state_box(
        ax, (5.4, 3.4), "allowlist + simulation", width=3.2, height=0.8, colour=1
    )
    diagrams.state_box(ax, (7.4, 0.6), "compare", width=2.0, height=0.8, colour=2)
    diagrams.state_box(ax, (11.2, 2.2), "page", width=1.8, height=0.8, colour=1)
    diagrams.state_box(ax, (11.2, 0.0), "build fails", width=1.8, height=0.8, colour=3)
    diagrams.arrow(ax, (2.8, 3.4), (3.8, 3.4), colour=0)
    diagrams.text(ax, (3.3, 3.9), "same calls", colour=0)
    diagrams.arrow(ax, (5.4, 3.0), (7.0, 1.0), colour=1)
    diagrams.text(ax, (6.9, 2.3), "new results", ha="left", colour=1)
    diagrams.arrow(ax, (2.8, 0.6), (6.4, 0.6), colour=0)
    diagrams.arrow(ax, (8.4, 0.9), (10.3, 2.0), colour=1)
    diagrams.text(ax, (8.9, 1.9), "agree", ha="left", colour=1)
    diagrams.arrow(ax, (8.4, 0.4), (10.3, 0.1), colour=3)
    diagrams.text(ax, (9.3, -0.35), "differ", colour=3)
    diagrams.text(
        ax, (1.4, 2.0), "the words are shown,\nnever compared", colour="#57606a"
    )
    fig.tight_layout()
    return fig

src/pbc/mechanics/experiments/agent_experiment.py, lines 153 to 181, read from the repository when this page was built.

Figure 2: conceptual What the replay keeps and what it recomputes. The recorded messages are reused as they are; every recorded tool result is thrown away and computed again by the simulation. The two results are compared, and a difference beyond the tolerance fails the build. The model’s words are never compared.

The diagram is a sketch and computes nothing. It answers the question the section raises: what exactly is recomputed? The tool results, and only those. That is why the check can catch a changed simulation and cannot tell whether the model reasoned well.

The replay compares every recomputed result with the recorded one and allows a relative difference of 1e-06. To see it work, change the simulation a little: here the drag constant is one per cent larger, as if the ball had been given a rougher surface, and the recording is replayed against that.

def rough_ball(recording) -> None:
    """The replay against a ball with one per cent more drag, which the
    comparison of the results must catch."""
    original = tools()
    rough = Allowlist(
        (
            Tool(
                "range_with_drag",
                "rough ball",
                (SPEED, ANGLE, TIME_STEP),
                lambda speed, angle, dt: {
                    "range_m": projectile_range(speed, angle, MASS, 1.01 * DRAG, dt)
                },
            ),
            original.tools[1],
        )
    )

    try:
        replay(
            recording,
            rough,
            system=SYSTEM,
            task=TASK,
            tolerance=TOLERANCE,
            max_steps=MAX_STEPS,
            rerecord="uv run --extra openai python -m pbc.agents.projectile_lab"
            " --record",
        )
    except ReplayMismatch as mismatch:
        print(mismatch)

src/pbc/mechanics/experiments/agent_experiment.py, lines 184 to 214, read from the repository when this page was built.

step 1, tool range_with_drag: range_m was 35.27758864537781 when recorded and is 35.14527765986668 now (tolerance 1e-06). If the change is intended, record the run again with: uv run --extra openai python -m pbc.agents.projectile_lab --record

The build fails with that message. The maintainer then runs the agent live, looks at the new run, and commits the new recording. Nothing on the page can silently differ from what the code computes.

The agent’s evidence

The figure plots what the agent saw: the ranges the replay recomputed at the smallest time step the agent used, against the angle, and the curve a scan of the same simulation gives.

def draw_agent_evidence(transcript) -> Figure:
    """The ranges the agent asked for at a fine step, on an independent
    scan."""
    fine = 0.002
    angles = np.arange(30.0, 50.5, 1.0)
    scan = [projectile_range(25.0, a, MASS, DRAG, fine) for a in angles]

    asked = {}
    for step in transcript.steps:
        for call, result in zip(step.message.tool_calls, step.results, strict=True):
            if (
                result.ok
                and call.name == "range_with_drag"
                and call.arguments["dt"] <= fine
            ):
                asked[call.arguments["angle"]] = result.content["range_m"]

    fig, ax = plt.subplots(figsize=(6.4, 3.6))
    ax.plot(angles, scan, color="#00429d", label="scan, dt = 0.002 s")
    ax.plot(
        list(asked),
        list(asked.values()),
        "o",
        color="#93003a",
        label="asked by the agent",
    )
    ax.set_xlabel("launch angle (degrees)")
    ax.set_ylabel("range (m)")
    ax.legend()
    return fig

src/pbc/mechanics/experiments/agent_experiment.py, lines 217 to 246, read from the repository when this page was built.

Figure 3: simulated The ranges the agent asked for lie on the curve that an independent scan of the simulation gives: the best angle is a little below forty-five degrees, and the curve is flat near its maximum, so nearby angles give almost the same range.

The data are simulated ranges of a tennis ball with quadratic drag. The curve is a scan made by this page, and the points are results the replay recomputed; the two come from the same simulation, so agreement shows that the agent’s numbers are the simulation’s, not that the simulation is right about a real ball.

The agent’s last message names an interval for the best angle and gives its evidence in words. The build does not read those words. What it can do is repeat the question with a computation of its own: the two angles the agent tried at its smallest step that gave the longest ranges bracket an interval, and a scan in steps of a quarter of a degree, at the smallest step the allowlist allows, finds the best angle independently.

def agent_conclusion(experiment: Experiment, transcript):
    """The ranges the agent asked for at its smallest step, and an
    independent scan around its best two angles; record the table of the
    scan and return the bracket and its best angle, in degrees."""
    smallest = min(
        call.arguments["dt"]
        for step in transcript.steps
        for call in step.message.tool_calls
        if call.name == "range_with_drag"
    )
    at_smallest = {}
    for step in transcript.steps:
        for call, result in zip(step.message.tool_calls, step.results, strict=True):
            if (
                result.ok
                and call.name == "range_with_drag"
                and call.arguments["dt"] == smallest
            ):
                at_smallest[call.arguments["angle"]] = result.content["range_m"]

    best_two = sorted(sorted(at_smallest, key=at_smallest.get)[-2:])
    bracket_low, bracket_high = best_two
    grid = np.arange(bracket_low - 2.0, bracket_high + 2.0 + 1e-9, 0.25)
    grid_ranges = [projectile_range(25.0, a, MASS, DRAG, smallest) for a in grid]
    best_angle = grid[int(np.argmax(grid_ranges))]
    best_range = max(grid_ranges)
    range_at_45 = projectile_range(25.0, 45.0, MASS, DRAG, smallest)

    experiment.table(
        "agent-conclusion",
        [
            ("best angle of the scan", best_angle, "degrees"),
            ("range at the best angle", best_range, "m"),
            ("range at 45 degrees", range_at_45, "m"),
            ("gain over 45 degrees", best_range - range_at_45, "m"),
            (
                "gain over 45 degrees",
                100 * (best_range - range_at_45) / range_at_45,
                "per cent",
            ),
        ],
        [("quantity", "", ""), ("value", "", ".4g"), ("unit", "", "")],
        "An independent scan of the same simulation at the agent's smallest step, "
        "in steps of a quarter of a degree.",
    )
    return bracket_low, bracket_high, best_angle

src/pbc/mechanics/experiments/agent_experiment.py, lines 249 to 294, read from the repository when this page was built.

An independent scan of the same simulation at the agent's smallest step, in steps of a quarter of a degree.

quantity	value	unit
best angle of the scan	40.5	degrees
range at the best angle	35.74	m
range at 45 degrees	35.4	m
gain over 45 degrees	0.3332	m
gain over 45 degrees	0.9411	per cent

Numerically verified (the simulated tennis ball with quadratic drag at 25 m/s, explicit Euler at the smallest allowed step, no measurement): At the smallest time step the agent used, the two angles it tried with the longest ranges bracket the best angle of an independent scan of the simulation, and the best angle lands the ball further than forty-five degrees by the amount in the table. The bracket is 40 to 41 degrees, and the scan’s best angle is 40.50 degrees, inside it. This is the part of the agent’s conclusion that the build recomputes. That the agent reasoned well to reach it, and that its remarks about convergence are right, is not something the build checks; the reader can compare them with the time-step table of the first exercise.

Run the agent yourself

This needs your own key and costs a few cents. The build does not run it, so the commands are marked.

Not verified. The build of this site did not run or check the material in this box.

# once: install the optional provider package
uv sync --extra openai

# in this shell only: the key stays out of code and files
export OPENAI_API_KEY="..."

uv run --extra openai python -m pbc.agents.projectile_lab

It prints each message and tool call as they happen. Never paste the key into a file, a notebook, or a command you will keep in a history that is shared. If a key ever appears in one, revoke it at the provider and make a new one.

The maintainer records the replay fixture of this page from a live run with the same command and --record, reads the file, and commits it.

Limits

• One recorded run. The page shows what one model once did. A live run differs in its path, and nothing here says how often a model finds the right answer.
• A replay checks numbers, not reasoning. It shows that every result the agent was shown is what this code computes. It does not show that the agent read them correctly, and the build never reads its words. The conclusion is compared with a scan only in the part the page computes.
• Numerical evidence about a model. The ranges are those of explicit Euler with quadratic drag for one ball. No ball was thrown and measured, so nothing on this page is experimentally supported, and nothing is proved.
• The bounds are decisions, tested but not proved. The unit tests try the attacks listed above and read the source for imports; they do not prove that no other route exists.
• The live run is not verified. The commands to run the agent yourself need a key, cost money, and are not run by the build, so they are marked.

Self-check. The replay recomputed every tool result of the run and found no difference from the recording. Does that show that the agent’s conclusion is correct? What would have to be checked in addition?

With scripts, you can record in this browser that you answered this question.

Solution to the self-check

No. It shows that the numbers the agent was shown, and the numbers the page displays, are the ones the code computes. The conclusion is a sentence the model wrote after reading them, and a model can read correct numbers and conclude something they do not support. The page tests the conclusion in a second, separate way: a scan of the simulation that the model did not control gives the best angle, and it falls inside the interval the agent’s best two runs bracket. Even both together are evidence about the simulated ball only; whether a real ball behaves so is settled by measuring one.

Exercises

Exercise 1. The time step is bounded below by 0.001 s. Call the simulation through the allowlist with the smallest allowed step and with the largest, at 40 degrees, and say how much the range changes. Why does the allowlist have a lower bound at all?

Solution to exercise 1

def step_bounds(experiment: Experiment) -> None:
    """Exercise 1: the range at forty degrees with the smallest and the
    largest step the allowlist accepts; record the table."""
    allowlist = tools()
    fine = allowlist.dispatch(
        ToolCall("a", "range_with_drag", {"speed": 25, "angle": 40, "dt": 0.001})
    )
    coarse = allowlist.dispatch(
        ToolCall("b", "range_with_drag", {"speed": 25, "angle": 40, "dt": 0.05})
    )
    experiment.table(
        "step-bounds",
        [
            ("smallest allowed step", 0.001, fine.content["range_m"]),
            ("largest allowed step", 0.05, coarse.content["range_m"]),
            ("difference", None, coarse.content["range_m"] - fine.content["range_m"]),
        ],
        [("step", "", ""), ("time step", "s", ".3f"), ("range", "m", ".3f")],
        "The range of the throw at forty degrees with the smallest and the largest "
        "time step the allowlist accepts.",
    )

src/pbc/mechanics/experiments/agent_experiment.py, lines 297 to 317, read from the repository when this page was built.

The range of the throw at forty degrees with the smallest and the largest time step the allowlist accepts.

step	time step (s)	range (m)
smallest allowed step	0.001	35.732
largest allowed step	0.050	36.042
difference		0.311

Explicit Euler has an error proportional to the step, so the largest step is visibly off, which is why the system prompt tells the model to check convergence. The lower bound is a cost limit: the number of steps grows as the step shrinks, and a step of zero would never finish. A bound on the cost of one call is part of the boundary, not only a bound on what it may touch.

Exercise 2. Add a third tool to the allowlist that returns the flight time of a throw without drag, for a speed and an angle within the same bounds as range_without_drag. Show that a call with an angle of 90 degrees is rejected and one with 60 degrees is not. The flight time is $2 v \sin\theta / g$.

Solution to exercise 2

def new_tool(experiment: Experiment) -> None:
    """Exercise 2: a new tool for the flight time, called inside and outside
    its bounds; record the table."""
    flight_time = Tool(
        "flight_time_without_drag",
        "Flight time in seconds of the same throw in vacuum.",
        (SPEED, ANGLE),
        lambda speed, angle: {
            "time_s": float(2 * speed * np.sin(np.radians(angle)) / g)
        },
    )
    extended = Allowlist((*tools().tools, flight_time))
    rows = []
    for angle in (90, 60):
        result = extended.dispatch(
            ToolCall("c", "flight_time_without_drag", {"speed": 25, "angle": angle})
        )
        rows.append(
            (
                angle,
                "ran" if result.ok else "rejected",
                result.content["time_s"] if result.ok else None,
                "" if result.ok else result.content["error"],
            )
        )
    experiment.table(
        "new-tool",
        rows,
        [
            ("angle", "degrees", "d"),
            ("outcome", "", ""),
            ("flight time", "s", ".3f"),
            ("reason", "", ""),
        ],
        "The new tool called with an angle outside its bounds and with one inside "
        "them.",
    )

src/pbc/mechanics/experiments/agent_experiment.py, lines 320 to 356, read from the repository when this page was built.

The new tool called with an angle outside its bounds and with one inside them.

angle (degrees)	outcome	flight time (s)	reason
90	rejected		angle must be between 1 and 89, got 90
60	ran	4.416

The new tool inherits the bounds of the parameters it reuses, so the call at ninety degrees is refused before the function runs. Nothing else about the loop or the replay changes; a lesson that uses the tool would record a new run.

Replay this experiment

This page was built from commit 6417e2ff90258bd2d2ba21d217eb7666fd24f680 of the repository.

In a clone at that commit, with the project’s Python packages installed, this command runs the experiment of this page:

uv run --locked --no-dev python -m pbc.mechanics.experiments.agent_experiment results

Files, tests, and every command

The files behind it:

• site/lessons/agents-llm/01-an-agent-runs-an-experiment/index.qmd: this page, with every code cell.
• src/pbc/agents/tools.py: code the experiment runs.
• src/pbc/agents/loop.py: code the experiment runs.
• src/pbc/agents/replay.py: code the experiment runs.
• src/pbc/agents/projectile_lab.py: code the experiment runs.
• src/pbc/mechanics/experiments/agent_experiment.py: code the experiment runs.
• src/pbc/experiment.py: code the experiment runs.
• tests/unit/test_agents_tools.py: tests of that code.
• tests/unit/test_agents_replay.py: tests of that code.
• tests/unit/test_agent_experiment_experiment.py: tests of that code.

With Git and uv installed (see the setup page), run:

git clone https://github.com/taspinar/physics-by-construction.git
cd physics-by-construction
git checkout 6417e2ff90258bd2d2ba21d217eb7666fd24f680
uv sync --locked --no-dev
uv run --locked --no-dev python -m pbc.mechanics.experiments.agent_experiment results

The experiment writes every figure of this page into results/, as <label>.png, and every number and table of the page, as the page shows it, into results/results.json. Its numbers match this page to the digits shown, and its figure files are the ones the page shows; digits beyond those can differ between machines, because floating-point libraries differ. Change the code of the experiment and run it again to see what changes.

The experiment replays the recorded run of the agent: it runs every tool call of the recording again and needs no model and no key.

Not verified. The build of this site did not run or check the material in this box.

Optional: the live run. It asks a language model to run the experiment again, with your own account. It needs an OpenAI account and its API key in the variable OPENAI_API_KEY, and the optional dependency openai; a run costs a few cents. The key stays in your shell, never in a file.

# needs: an OpenAI account, its API key, and the extra openai
uv sync --locked --no-dev --extra openai
export OPENAI_API_KEY="..."   # this shell only
uv run --locked --no-dev --extra openai python -m pbc.agents.projectile_lab