Full, partial, and no prior knowledge under one protocol

The same language model probes the same hidden spring three times, told the form of the law, only the names of its parameters, or nothing; the build scores the three claims on the same held-out settings and shows what the agent’s own readings would have supported.

Course

Classical mechanics, lesson 19 of 28

Slot

D2, number 2 of the 13 slots of the path Scientific law discovery

Methods

Simulation, LLM agents

Difficulty

Advanced, level 3 of 3

What you’ll learn

Set up a comparison of conditions in which only the prior knowledge differs, and check that it does
Infer a spring’s force law from the motion it causes, and say why late times constrain it most
Separate what a run’s readings support from what the agent concluded from them
Explain why one run per condition cannot rank the conditions

Prerequisites

Discovering a force law with controlled probes, The harmonic oscillator.

Before you begin

Trigonometry: the cosine and its period
Statistics: least squares and the chi-square of residuals

Related

none

Previous

Discovering a force law with controlled probes

Next

Unlearning a familiar law: a modified force

With scripts, this page keeps a record in your browser of whether you completed the lesson. Nothing is stored without them.

With scripts, this place suggests what to do next, with the reason, from the progress saved in your browser. Without them, follow the learning path.

Does it help an agent to be told what kind of law it is looking for? The question sounds easy and is easy to answer badly. Give one run the form of the law and another nothing, and let the first also have more probes, a different model, or easier settings to be judged on, and the comparison measures all of those at once. A fair comparison changes one thing.

This lesson asks the question of a spring. A mass on a spring of hidden stiffness is released at rest, and the agent can only read where the mass is at a time of its choosing. The force law, $F = -kx$ , is never seen directly; it shows in the motion it causes. The same model probes the same spring three times under the Discovery evaluation protocol: told the form of the motion (full), told only the name of the parameter to report (partial), or told nothing beyond the controls and the reading (none). Everything else is held equal, and the build checks that it is.

The build then scores the three claims on the same held-out settings, sets them beside baselines that use no model, and asks a second question the scores alone cannot answer: did the readings each run collected contain the answer, whatever the agent concluded from them?

Assumptions

The spring is simulated and linear. The mass of one kilogram moves under $F = -kx$ with no friction (pbc.mechanics.oscillator), so a release at rest from amplitude $A$ gives $x(t) = A\cos\omega t$ with $\omega = \sqrt{k/m}$ . A claim is an angular frequency $\omega$ ; the stiffness follows as $k = m\omega^2$ . A spring with friction or a stiffening force would be outside the family, and no claim could be right.
The readings are noisy, the noise is known and repeatable. Each reading has Gaussian noise; the same setting gives the same reading, so the replay is exact.
Only the task text differs between conditions. The model, the system prompt, the tools and their bounds, the budget of probes, the step limit, the seed, the tolerance, and the held-out settings are those of one protocol object. The owner of the project set this as a condition of the lesson, and a cell below checks it.
The tools are part of every condition. The tool that takes the claim names its parameter “omega, angular frequency” in all three conditions, because the claims must be comparable. Even under none the agent learns that an angular frequency is wanted, which is a hint at an oscillation.
One run per condition. Each condition was run once, on one seed. That is enough to show the protocol and to read three runs carefully; it is not enough to say which condition is better.

A force law seen through its motion

For a mass $m$ on a spring of stiffness $k$ , Newton’s second law and Hooke’s law give

$m\ddot x = -kx, \qquad x(0) = A,\ \dot x(0) = 0 \quad\Longrightarrow\quad x(t) = A\cos(\omega t),\quad \omega = \sqrt{k/m}.$

The motion does not depend on the amplitude except through the factor $A$ : the period is the same for a small swing and a large one. That is the signature of a linear force, and an agent can test it by releasing from two amplitudes and comparing $x/A$ at the same time.

Where to probe. A reading at time $t$ constrains the phase $\omega t$ . An error $\delta\omega$ in the frequency shifts the phase by $t\,\delta\omega$ , which grows with time. So a reading at a late time constrains $\omega$ more than an early one with the same noise, as long as the time is known: from $x/A = \cos(\omega t)$ one gets $\omega t$ only up to the branches of the arccosine, and the early readings tell which branch a late one is on. A large amplitude makes the relative noise small. The probes allowed run up to a few seconds; the claim is scored at later times, where a small error in $\omega$ has grown into a large error in the position.

The score. As in the force-law lesson, the held-out error is the root-mean-square difference between the claim’s prediction and the noise-free motion at the held-out settings, here relative to the amplitude (the displacement passes through zero). The law is recovered when the error is at most 0.05 and the run ended on its own.

Three conditions, one protocol

def tasks(experiment: Experiment) -> None:
    """The task text of each condition; record the table."""
    experiment.table(
        "d2-tasks",
        [(c, task_of(RUNS[NAMES[c]].protocol, c)[1]) for c in CONDITIONS],
        [("condition", "", ""), ("the task the model was given", "", "")],
        "The task text of the three conditions; it is the only input that differs.",
    )

src/pbc/mechanics/experiments/prior_knowledge_in_discovery.py, lines 57 to 64, read from the repository when this page was built.

The task text of the three conditions; it is the only input that differs.
condition	the task the model was given
full	Set amplitude (m) from 0.1 to 2; time (s) from 0 to 3, and read displacement (m), which has noise. The law has the form: displacement = amplitude cos(omega time). Find omega. Then submit your claim. It is scored on settings you cannot probe.
partial	Set amplitude (m) from 0.1 to 2; time (s) from 0 to 3, and read displacement (m), which has noise. Find the law, and report omega of it. Then submit your claim. It is scored on settings you cannot probe.
none	Set amplitude (m) from 0.1 to 2; time (s) from 0 to 3, and read displacement (m), which has noise. Find the law that gives it, and report the parameters you choose. Then submit your claim. It is scored on settings you cannot probe.

def fairness(experiment: Experiment) -> None:
    """What the three runs share, from the protocol and the record of each
    run; record the table."""
    entries = [RUNS[NAMES[c]] for c in CONDITIONS]
    protocols = [PROTOCOLS[e.protocol] for e in entries]
    same = {
        "protocol object": len({id(p) for p in protocols}) == 1,
        "model": len({e.info.model for e in entries}) == 1,
        "system prompt": len({task_of(e.protocol, e.condition)[0] for e in entries})
        == 1,
        "tools and bounds": len(
            {
                str(environment_allowlist(environment(e.protocol)).specs())
                for e in entries
            }
        )
        == 1,
        "budget and step limit": len({(p.budget, p.max_steps) for p in protocols}) == 1,
        "seed": len({e.info.seed for e in entries}) == 1,
        "tolerance": len({p.tolerance for p in protocols}) == 1,
        "held-out settings": len(
            {str(environment(e.protocol)._held_out) for e in entries}
        )
        == 1,
        "task text": len({task_of(e.protocol, e.condition)[1] for e in entries}) == 1,
    }
    experiment.table(
        "d2-fairness",
        [(what, "same" if equal else "differs") for what, equal in same.items()],
        [("what the runs share", "", ""), ("across the three conditions", "", "")],
        "What the three runs share, checked by the build from the protocol and"
        " the record of each run.",
    )

src/pbc/mechanics/experiments/prior_knowledge_in_discovery.py, lines 67 to 99, read from the repository when this page was built.

What the three runs share, checked by the build from the protocol and the record of each run.
what the runs share	across the three conditions
protocol object	same
model	same
system prompt	same
tools and bounds	same
budget and step limit	same
seed	same
tolerance	same
held-out settings	same
task text	differs

Numerically verified (the protocol and the run records of this lesson): Across the three conditions the model, the system prompt, the tools, the budget, the step limit, the seed, the tolerance, and the held-out settings are the same, and only the task text differs. The held-out settings are compared by the build and not printed: they are the judge’s, and this page does not show them.

measure: Set the controls and read the displacement (m), with noise. Each call uses one of 20 probes.
- amplitude: starting displacement, released at rest (m); allowed values 0.1 to 2
- time: time since the release (s); allowed values 0 to 3
submit: State your claim as the parameters of the law and end the run. It uses no probe.
- omega: angular frequency (rad/s); allowed values 0.1 to 20

In every condition the agent may call measure at most 20 times and submit once to state its claim, in at most 12 messages of at most 4 tool calls each. Calls beyond that number in one message are refused unrun and cost no probe. It may not run code, read a file, reach the network, or call the judge.

Code

The fit of a run’s own readings uses the fitting helper of the inverse-physics lesson on the readings the replay recomputed. It answers what the probes support, separately from what the agent said.

def probes(protocol: str, transcript: Transcript) -> Dataset:
    """The probes a run made: the accepted ``measure`` calls of a replayed
    transcript, with the readings the replay recomputed."""
    world = environment(protocol)
    settings, values = [], []
    for step in transcript.steps:
        for call, result in zip(step.message.tool_calls, step.results, strict=True):
            if call.name == MEASURE and result.ok:
                settings.append({k: float(v) for k, v in call.arguments.items()})
                values.append(result.content[world.observation.name])
    return Dataset(tuple(settings), np.array(values), world.observation.sigma)

src/pbc/discovery/force_lab.py, lines 228 to 238, read from the repository when this page was built.

def fit_probes(protocol: str, data: Dataset) -> FitResult | None:
    """The public family of ``protocol`` fitted to ``data`` from each of its
    starts, the best fit kept, with its standard errors; ``None`` when no fit
    can be made."""
    world = environment(protocol)
    names = tuple(c.name for c in world.claim)
    bounds = ([c.low for c in world.claim], [c.high for c in world.claim])

    def curve(index: np.ndarray, *params: float) -> np.ndarray:
        claim = dict(zip(names, params, strict=True))
        return np.array([world.predict(claim, data.settings[int(i)]) for i in index])

    best = None
    for start in world.starts:
        try:
            fit = fit_curve(
                curve,
                np.arange(len(data.values), dtype=float),
                data.values,
                start,
                sigma=data.sigma,
                names=names,
                bounds=bounds,
            )
        except ValueError:
            continue
        if best is None or fit.chi2 < best.chi2:
            best = fit
    return best

src/pbc/discovery/force_lab.py, lines 241 to 269, read from the repository when this page was built.

def score_transcript(protocol: Protocol, transcript: Transcript, seed: int) -> Score:
    """Score ``transcript`` by running its tool calls again on a fresh
    environment of ``seed``. The same calls give the same score."""
    environment = protocol.make(seed)
    allowlist = environment_allowlist(environment)
    probes = refused = 0
    claim: dict[str, float] | None = None
    for step in transcript.steps:
        for number, call in enumerate(step.message.tool_calls):
            # The loop refuses the calls of a message beyond its limit unrun.
            if number >= MAX_CALLS_PER_MESSAGE:
                refused += 1
                continue
            result = allowlist.dispatch(call)
            if not result.ok:
                refused += 1
            elif call.name == MEASURE:
                probes += 1
            elif call.name == SUBMIT:
                claim = {k: float(v) for k, v in call.arguments.items()}
    error = math.inf if claim is None else environment.held_out_error(claim)
    recovered = transcript.finished and error <= protocol.tolerance
    return Score(seed, probes, refused, claim, error, recovered)

src/pbc/discovery/protocol.py, lines 161 to 183, read from the repository when this page was built.

Worked examples

The three recorded runs

All three were recorded on the same day by the same Claude model working in a Claude Code session, each as a fresh sub-agent that was given the system prompt, its task, and the tool descriptions, and nothing else. It chose every call after reading the readings the harness computed. None of them knew of the other two.

    runs = {c: replay_run(NAMES[c], LESSON) for c in CONDITIONS}

src/pbc/mechanics/experiments/prior_knowledge_in_discovery.py, line 273, read from the repository when this page was built.

Recorded run. The model messages below were recorded from claude-sonnet-5-5 in a Claude Code session on 2026-10-16. This build did not call a model. It ran every tool call again; the results shown are the ones it computed.

Step 1. Model message (recorded):

Starting measurements. First probes: amplitude 1.0 at several times to get a rough sense of omega, plus amplitude 2.0 at time 0.5 to check amplitude independence.

measure(amplitude=1, time=0.5): recomputed by this build: displacement = 0.62429
measure(amplitude=1, time=1): recomputed by this build: displacement = -0.152509
measure(amplitude=1, time=1.5): recomputed by this build: displacement = -0.876257
measure(amplitude=1, time=2): recomputed by this build: displacement = -0.955727
measure(amplitude=1, time=0.2): rejected before it ran: at most 4 tool calls per message
measure(amplitude=2, time=0.5): rejected before it ran: at most 4 tool calls per message

Step 2. Model message (recorded):

Results so far suggest omega is about 1.75 rad/s (the four readings give 1.72 to 1.80). I'll refine it with later times, which are more sensitive to omega. I'll also check that the law doesn't depend on amplitude. I'm sending 4 calls this time because of the limit.

measure(amplitude=1, time=3): recomputed by this build: displacement = 0.456928
measure(amplitude=1, time=2.5): recomputed by this build: displacement = -0.402325
measure(amplitude=2, time=3): recomputed by this build: displacement = 0.934914
measure(amplitude=2, time=0.5): recomputed by this build: displacement = 1.27655

Step 3. Model message (recorded):

The late-time readings give omega of about 1.73 rad/s. At t=3, amplitude 1 gives 1.729 and amplitude 2 gives 1.734. At t=2.5, amplitude 1 gives 1.719. The amplitude-2 reading at t=0.5 gives 1.76, but early times are less precise. The phase is the same at both amplitudes, so the law is amplitude-independent. I'm submitting omega = 1.73.

submit(omega=1.73): recomputed by this build: submitted = 1

Step 4. Model message (recorded):

Submitted omega = 1.73 rad/s. The run is complete. The submission was accepted (result: submitted 1.0).

The model ended the run with a message that requests no tool.

Step 1. Model message (recorded):

Starting measurements. First, probe amplitude 0.5 at several times to see the oscillation period.

measure(amplitude=0.5, time=0.5): recomputed by this build: displacement = 0.294874
measure(amplitude=0.5, time=1): recomputed by this build: displacement = -0.0919432
measure(amplitude=0.5, time=1.5): recomputed by this build: displacement = -0.404658
measure(amplitude=0.5, time=2): recomputed by this build: displacement = -0.459587
measure(amplitude=0.5, time=2.5): rejected before it ran: at most 4 tool calls per message
measure(amplitude=0.5, time=3): rejected before it ran: at most 4 tool calls per message

Step 2. Model message (recorded):

Four probes used so far (16 left). With x = A cos(w t) and A = 0.5, the readings give w of about 1.89 at t=0.5, 1.75 at t=1.0, 1.67 at t=1.5 and 1.37 at t=2.0. The apparent frequency drifts, so the law may depend on amplitude, or the noise may be large. Next I will check amplitude dependence with a large and a small amplitude at the same time, then continue at A=0.5 with at most 4 calls per message.

measure(amplitude=2, time=0.5): recomputed by this build: displacement = 1.27655
measure(amplitude=0.1, time=0.5): recomputed by this build: displacement = 0.0895339
measure(amplitude=0.5, time=2.5): recomputed by this build: displacement = -0.198576
measure(amplitude=0.5, time=3): recomputed by this build: displacement = 0.222181

Step 3. Model message (recorded):

I fit x = A cos(w t) to all ten readings, taking the branch of the arccosine that fits each one. The estimates of w are 1.89, 1.75, 1.67, 1.77, 1.72 and 1.72 at A=0.5, and 1.76 at A=2.0. They agree to within noise, so the oscillation is close to harmonic with no clear amplitude dependence. The A=0.1 reading was too small to be informative. The mean is about 1.75 rad/s. Submitting that now.

submit(omega=1.75): recomputed by this build: submitted = 1

Step 4. Model message (recorded):

Submitted omega = 1.75 rad/s and the run is over. I used 10 of the 20 probes. The data fit x = A cos(w t) with w of about 1.75 rad/s at both A=0.5 and A=2.0, with no clear amplitude dependence. The submission was accepted and returned a score of 1.0.

The model ended the run with a message that requests no tool.

Step 1. Model message (recorded):

Starting measurements. First probe: amplitude 1.0 m at several times to see the oscillation, then I will vary time and amplitude to get the period and check whether it depends on amplitude.

measure(amplitude=1, time=0.5): recomputed by this build: displacement = 0.62429
measure(amplitude=1, time=1): recomputed by this build: displacement = -0.152509
measure(amplitude=1, time=1.5): recomputed by this build: displacement = -0.876257
measure(amplitude=1, time=2): recomputed by this build: displacement = -0.955727
measure(amplitude=1, time=2.5): rejected before it ran: at most 4 tool calls per message
measure(amplitude=1, time=3): rejected before it ran: at most 4 tool calls per message

Step 2. Model message (recorded):

Fitting x = A cos(w t) to the four readings at A=1 gives w of about 1.72 to 1.78 rad/s. Next I will check whether w depends on amplitude. I will use small and large amplitudes at two times, with at most 4 calls per message.

measure(amplitude=0.2, time=1): recomputed by this build: displacement = -0.00790354
measure(amplitude=2, time=1): recomputed by this build: displacement = -0.335028
measure(amplitude=0.2, time=2.5): recomputed by this build: displacement = -0.0738905
measure(amplitude=2, time=2.5): recomputed by this build: displacement = -0.733642

Step 3. Model message (recorded):

The results are consistent with linear, amplitude-independent motion x = A cos(w t). At A=2.0, t=1.0 gives x/A = -0.168, so w = 1.74. At A=2.0, t=2.5 gives x/A = -0.367, so w = 1.74. The A=1.0 data give 1.72 to 1.78. The A=0.2 data are noisier but agree (about 1.6 to 1.73). The best estimate is w = 1.74 rad/s, so I am submitting it.

submit(omega=1.74): recomputed by this build: submitted = 1

Step 4. Model message (recorded):

The claim omega = 1.74 rad/s was submitted and accepted (result: submitted 1.0). I used 10 of the 20 probes. The data fit x = A cos(omega t) with omega independent of amplitude, over A from 0.2 to 2.0 m. No further action is needed.

The model ended the run with a message that requests no tool.

Each run sent more calls in its first message than the harness runs, and the extra calls were refused without a reading; the agents adapted.

The scores on the same held-out settings

def score_table(experiment: Experiment, scores) -> None:
    """The three claims scored on the same held-out settings; record the
    table."""
    experiment.table(
        "d2-scores",
        [
            (c, s.probes, s.refused, s.claim["omega"], s.held_out_error, s.outcome)
            for c, s in scores.items()
        ],
        [
            ("condition", "", ""),
            ("probes", "", "d"),
            ("refused", "", "d"),
            ("claimed omega", "rad/s", ".3g"),
            ("held-out error", "relative", ".3g"),
            ("outcome", "", ""),
        ],
        "The three claims, scored again by the build on the same held-out settings.",
    )

src/pbc/mechanics/experiments/prior_knowledge_in_discovery.py, lines 102 to 120, read from the repository when this page was built.

The three claims, scored again by the build on the same held-out settings.
condition	probes	refused	claimed omega (rad/s)	held-out error (relative)	outcome
full	8	2	1.73	0.0167	recovered
partial	8	2	1.75	0.141	not recovered
none	8	2	1.74	0.0637	not recovered

Numerically verified (one recorded run per condition, one seed, the simulated spring of this protocol): Scored on the same held-out settings, the claim of the full condition recovers the law and the claims of partial and none do not. The three claims differ in the second decimal of the frequency. That is enough: the held-out times are late, and a small error in $\omega$ has grown into a phase error there.

What the readings supported

Each run spent 8 probes. Fit the motion to the readings of each run, exactly those its agent saw:

def probe_fits(experiment: Experiment, runs, scores):
    """The motion fitted to each run's own readings; record the table and
    return the fits and their held-out errors."""
    fits = {c: fit_probes(PROTOCOL, probes(PROTOCOL, runs[c][1])) for c in CONDITIONS}
    own = {
        c: environment(PROTOCOL).held_out_error({"omega": float(f.params[0])})
        for c, f in fits.items()
    }
    experiment.table(
        "d2-probe-fits",
        [
            (
                c,
                float(f.params[0]),
                float(f.stderr[0]),
                f.chi2,
                f.dof,
                own[c],
                (scores[c].claim["omega"] - float(f.params[0])) / float(f.stderr[0]),
            )
            for c, f in fits.items()
        ],
        [
            ("condition", "", ""),
            ("fitted omega", "rad/s", ".4g"),
            ("standard error", "rad/s", ".2g"),
            ("chi-square", "", ".3g"),
            ("degrees of freedom", "", "d"),
            ("held-out error of the fit", "relative", ".3g"),
            ("claim minus fit", "standard errors", ".2g"),
        ],
        "The motion fitted to each run's own readings, and how far the agent's"
        " claim lies from that fit.",
    )
    return fits, own

src/pbc/mechanics/experiments/prior_knowledge_in_discovery.py, lines 123 to 157, read from the repository when this page was built.

The motion fitted to each run's own readings, and how far the agent's claim lies from that fit.
condition	fitted omega (rad/s)	standard error (rad/s)	chi-square	degrees of freedom	held-out error of the fit (relative)	claim minus fit (standard errors)
full	1.731	0.003	5.93	7	0.00708	-0.39
partial	1.73	0.0097	7.39	7	0.0141	2
none	1.735	0.0038	4.28	7	0.0268	1.2

Numerically verified (the readings of the three recorded runs): The readings of every one of the three runs determine the frequency well enough to recover the law: a least-squares fit to each run’s own readings has a held-out error within the tolerance. The difference between the conditions is therefore not in the information the agents collected. It is in what each agent concluded from it. The full agent read the frequency off its late readings and said why. The partial agent averaged estimates from early and late times with equal weight, and the early ones, whose phase is least sensitive to the frequency, pulled the mean away. The last column says how far each claim is from what its data support, in units of the fit’s standard error.

Beside the baselines

def beside_the_baselines(experiment: Experiment, scores) -> None:
    """The three runs beside the baselines without a model; record the
    table."""
    rows = comparison(
        tuple(scores[c] for c in CONDITIONS), baselines(PROTOCOL), TOLERANCE
    )
    for row, condition in zip(rows, CONDITIONS, strict=False):
        row["who"] = f"agent, {condition}"
    experiment.table(
        "d2-baselines",
        [
            (
                r["who"],
                r["probes"],
                r["held_out_error"],
                "yes" if r["recovered"] else "no",
                r["note"],
            )
            for r in rows
        ],
        [
            ("method", "", ""),
            ("probes", "", "d"),
            ("held-out error", "", ".3g"),
            ("recovered", "", ""),
            ("note", "", ""),
        ],
        "The three runs and the baselines without a model, on the same spring"
        " with the same budget.",
    )

src/pbc/mechanics/experiments/prior_knowledge_in_discovery.py, lines 160 to 189, read from the repository when this page was built.

The three runs and the baselines without a model, on the same spring with the same budget.
method	probes	held-out error	recovered	note
agent, full	8	0.0167	yes	recovered
agent, partial	8	0.141	no	not recovered
agent, none	8	0.0637	no	not recovered
grid + least squares	20	0.00754	yes	fit consistent with the noise
random + least squares	20	0.00743	yes	fit consistent with the noise
grid + sparse identification	20	inf	no	the environment has no library of terms

Least squares recovers the law from the full budget on a grid and at random. Sparse identification has no library of terms for an oscillation and reports that it cannot run, a failure that is shown rather than dropped.

The comparison in one picture

def draw_condition_errors(scores, own) -> Figure:
    """The held-out error of each condition's claim beside that of the fit
    to the same run's readings, with the tolerance."""
    x = np.arange(len(CONDITIONS))
    fig, ax = plt.subplots(figsize=(6.4, 3.6))
    ax.bar(
        x - 0.18,
        [scores[c].held_out_error for c in CONDITIONS],
        0.36,
        color="#00429d",
        label="agent's claim",
    )
    ax.bar(
        x + 0.18,
        [own[c] for c in CONDITIONS],
        0.36,
        color="#93c4d2",
        label="fit to the run's readings",
    )
    ax.axhline(TOLERANCE, color="#93003a", linestyle="--", label="tolerance")
    ax.set_xticks(x, CONDITIONS)
    ax.set_xlabel("prior knowledge")
    ax.set_ylabel("held-out error (relative)")
    ax.legend(fontsize=8)
    return fig

src/pbc/mechanics/experiments/prior_knowledge_in_discovery.py, lines 192 to 216, read from the repository when this page was built.

A grouped bar chart with three groups, full, partial and none. In each group a dark bar shows the held-out error of the agent's claim and a light bar the error of the fit to the run's own readings. A dashed horizontal line marks the tolerance. All light bars are below it. The dark bar of full is below it; the dark bars of partial and none are above it, partial the highest. — Figure 1: simulated Held-out error for each condition: the agent’s claim against the least-squares fit to the same run’s readings, with the tolerance as a dashed line. Every fit is below the tolerance; only the claim of the full condition is.

The figure shows errors of simulated runs that this build scored again. The gap between a dark bar and the light bar beside it is the cost of the agent’s own reasoning, measured against what its readings supported. Its size is not a property of the condition: with one run each, it could be the luck of one estimate.

Run the agents yourself

This needs your own key and costs a few cents per run. The build does not run it, so the commands are marked.

Not verified. The build of this site did not run or check the material in this box.

uv sync --locked --no-dev --extra openai
export OPENAI_API_KEY="..."
cd site/lessons/agents-llm/06-prior-knowledge-in-discovery
uv run --locked --no-dev --extra openai python -m pbc.discovery.force_lab prior-full prior-partial prior-none

With --record the command writes the fixture of each condition in the current directory. A comparison worth stating needs many runs per condition, on several seeds; such a benchmark runs outside the build and would be marked “not verified” here.

Limits

One run per condition cannot rank the conditions. The full condition recovered and the others did not, but a second run of each could reverse that. The page shows a protocol and three runs, not an effect of prior knowledge.
The conditions leak through the tools. The claim’s parameter is named and described in every condition. Under none the agent still learns that an angular frequency is wanted.
The family is fixed. A claim is one frequency of an undamped linear spring. An agent that suspected friction or a non-linear force could not say so in its claim, only in its words, which are not scored.
The agents computed without code. They estimated the frequency in their heads from the readings; the harness gives no calculator. A fit is what a calculator would have given, and the gap is partly arithmetic.
Simulated, not measured. Nothing here is a reading of a real spring.
The recordings are not API runs. They are Claude Code sessions, told by the session to answer in the harness’s format and to use no tool of their own. A run through the provider API may differ, and the sessions reported no token count or price.

Self-check. The full condition recovered the law and the other two did not, on the same held-out settings. What does this comparison support?

That telling an agent the form of the law makes it recover the law.

There is one run per condition. A single success and two failures are consistent with no effect at all, and the fits show that every run had the information it needed.
That on these three runs the agents’ conclusions, not the readings they collected, decided the outcome, and that more runs are needed to say anything about the conditions.

Right. The fits to each run’s own readings all recover the law; the claims differ because of how each agent estimated the frequency.
That the none condition is better than the partial one, because its error is smaller.

Both failed, and with one run each the order of two failures says nothing.

Solution to the self-check

It supports a statement about three runs: under one protocol, with everything but the task equal, the full run’s claim predicts the held-out settings within the tolerance and the other two do not, while the readings of all three runs would have supported a recovery. The difference lies in each agent’s estimate, not in its data. It does not support a statement about the conditions: that needs many runs per condition and seed, and a test of whether the difference in recovery rates is larger than chance.

Exercises

Exercise 1. The agents spent 8 probes each; the baselines spent 20. Give the grid and random baselines the same 8 probes on the same spring, and say whether they still recover the law.

Solution to exercise 1

def the_agents_budget(experiment: Experiment, probes_used: int) -> None:
    """Exercise 1: the baselines on the same spring with the agents' number
    of probes; record the table."""
    small = replace(PROTOCOLS[PROTOCOL], budget=probes_used)
    results = run_baselines(small.make(SEED), SEED)
    experiment.table(
        "small-budget",
        [
            (
                r.name,
                r.probes,
                r.held_out_error,
                "yes" if r.success and r.held_out_error <= TOLERANCE else "no",
                r.reason or "fit consistent with the noise",
            )
            for r in results
        ],
        [
            ("method", "", ""),
            ("probes", "", "d"),
            ("held-out error", "", ".3g"),
            ("recovered", "", ""),
            ("note", "", ""),
        ],
        "The baselines on the same spring with the agents' number of probes.",
    )

src/pbc/mechanics/experiments/prior_knowledge_in_discovery.py, lines 219 to 244, read from the repository when this page was built.

The baselines on the same spring with the agents' number of probes.
method	probes	held-out error	recovered	note
grid + least squares	8	0.00419	yes	fit consistent with the noise
random + least squares	8	0.023	yes	fit consistent with the noise
grid + sparse identification	8	inf	no	the environment has no library of terms

With the agents’ number of probes the grid fails and the random settings recover the law here. The grid’s fit is consistent with the noise and still wrong: with so few distinct times, another frequency, an alias whose cosine passes through the same few readings, fits them as well as the right one. The chi-square cannot tell the two apart; the held-out settings can. A few well-chosen probes, as the agents’ fits show, do better than a few regular ones placed without regard to the phase.

Exercise 2. An error $\delta\omega$ in the frequency shifts the phase at time $t$ by $t\,\delta\omega$ . For the claim of the partial condition, take the difference from the fit of its own readings as $\delta\omega$ , and compute the phase shift at 3 s, the latest time the tools accept, and at 9 s. Why does a claim that looks close at the probed times fail later?

Solution to exercise 2

def phase_shift(experiment: Experiment, scores, fits) -> None:
    """Exercise 2: the phase shift the partial claim's frequency error causes
    at the latest probed time and at three times it; record the table."""
    delta = scores["partial"].claim["omega"] - float(fits["partial"].params[0])
    experiment.table(
        "phase-shift",
        [(t, t * delta) for t in (3.0, 9.0)],
        [("time", "s", ".0f"), ("phase shift", "rad", ".2g")],
        "The phase shift that the partial claim's frequency error causes, by the time.",
    )

src/pbc/mechanics/experiments/prior_knowledge_in_discovery.py, lines 247 to 256, read from the repository when this page was built.

The phase shift that the partial claim's frequency error causes, by the time.
time (s)	phase shift (rad)
3	0.059
9	0.18

The phase shift grows in proportion to the time. At the latest probed time it is small, so the claim’s prediction differs from the readings by about the noise, and the agent could not see its error there. At three times that time the same error is three times larger in phase, and the predicted position, which is the amplitude times the cosine of the phase, is off by a fraction of the amplitude that exceeds the tolerance. Scoring at settings beyond the probed range is what exposes it.

Replay this experiment

This page was built from commit 6417e2ff90258bd2d2ba21d217eb7666fd24f680 of the repository.

In a clone at that commit, with the project’s Python packages installed, this command runs the experiment of this page:

uv run --locked --no-dev python -m pbc.mechanics.experiments.prior_knowledge_in_discovery results

Files, tests, and every command

The files behind it:

site/lessons/agents-llm/06-prior-knowledge-in-discovery/index.qmd: this page, with every code cell.
src/pbc/discovery/force_lab.py: code the experiment runs.
src/pbc/discovery/environment.py: code the experiment runs.
src/pbc/discovery/protocol.py: code the experiment runs.
src/pbc/discovery/baselines.py: code the experiment runs.
src/pbc/data/fit.py: code the experiment runs.
src/pbc/mechanics/experiments/prior_knowledge_in_discovery.py: code the experiment runs.
src/pbc/experiment.py: code the experiment runs.
tests/unit/test_force_lab.py: tests of that code.
tests/unit/test_discovery.py: tests of that code.
tests/unit/test_prior_knowledge_in_discovery_experiment.py: tests of that code.

With Git and uv installed (see the setup page), run:

git clone https://github.com/taspinar/physics-by-construction.git
cd physics-by-construction
git checkout 6417e2ff90258bd2d2ba21d217eb7666fd24f680
uv sync --locked --no-dev
uv run --locked --no-dev python -m pbc.mechanics.experiments.prior_knowledge_in_discovery results

The experiment writes every figure of this page into results/, as <label>.png, and every number and table of the page, as the page shows it, into results/results.json. Its numbers match this page to the digits shown, and its figure files are the ones the page shows; digits beyond those can differ between machines, because floating-point libraries differ. Change the code of the experiment and run it again to see what changes.

The experiment replays the recorded run of the agent: it runs every tool call of the recording again and needs no model and no key.

Not verified. The build of this site did not run or check the material in this box.

Optional: the live run. It asks a language model to run the experiment again, with your own account. It needs an OpenAI account and its API key in the variable OPENAI_API_KEY, and the optional dependency openai; a run costs a few cents. The key stays in your shell, never in a file.

# needs: an OpenAI account, its API key, and the extra openai
uv sync --locked --no-dev --extra openai
export OPENAI_API_KEY="..."   # this shell only
uv run --locked --no-dev --extra openai python -m pbc.discovery.force_lab prior-full prior-partial prior-none