Unlearning a familiar law: a modified force

A language model probes a central force whose law is not the inverse square it knows; the build shows that the familiar law fails the readings, that a baseline without a model identifies the modified one, and reports the agent’s claim exactly as it scored.

Course

Classical mechanics, lesson 20 of 28

Slot

D3, number 3 of the 13 slots of the path Scientific law discovery

Methods

Simulation, LLM agents

Difficulty

Advanced, level 3 of 3

What you’ll learn

Test a familiar law against readings and reject it with a stated criterion
Identify a modified exponent with a baseline that uses no model, and score it beyond the probed range
Tell a fit that is consistent where it was probed from one that is right where it was not
Report an agent’s result as it scored, whether or not the baseline did better

Prerequisites

Discovering a force law with controlled probes, The Kepler orbit.

Before you begin

Statistics: least squares and the chi-square of residuals
Logarithms: the logarithm of a product and of a power

Related

none

Previous

Full, partial, and no prior knowledge under one protocol

Next

Particles in a box

With scripts, this page keeps a record in your browser of whether you completed the lesson. Nothing is stored without them.

With scripts, this place suggests what to do next, with the reason, from the progress saved in your browser. Without them, follow the learning path.

A body is pulled toward a centre, and the pull weakens with distance. Anyone who has learned mechanics knows what comes next: the inverse square, the law of gravity and of the electric force between charges. It is so familiar that it is easy to read into measurements that do not quite say it. But the inverse square is a result about our world, not a necessity: in a world with another number of dimensions, or around a source of another shape, the exponent differs. A discovery method that only ever finds the inverse square has not discovered it.

This lesson puts a language model in front of a central force whose law is deliberately modified: it still falls as a power of the distance, but not the second power. The box, the tools, the budget, and the scoring are those of the force-law lesson; only the hidden world differs. The model is not warned.

The lesson has a rule, set by the owner of the project before any run: the point is not to make the agent find the modified law at any cost. The build first shows that the familiar law fails the readings and that a method without a model identifies the modified one. Then it reports what the agent claimed and how that claim scored, as it is. Had the agent fallen back on the inverse square, the page would report a failure, not explain it away.

Assumptions

The world is simulated, and its law is in the family. The force is $F = k\,m\,r^{-p}$ , strength $k$ times mass $m$ times distance $r$ to the power $-p$ , with a hidden strength and a hidden exponent that is not two (CentralForceEnvironment of pbc.discovery). The page does not print the hidden values; it prints claims and their scores.
The familiar law is a hypothesis to test, not a starting point. The inverse-square fit holds the exponent at two and fits only the strength. It is judged by the same rule as every baseline: its chi-square must be what the noise alone gives.
The same protocol as the force-law lesson. Budget, step limit, seed, tolerance, tools, and the condition of prior knowledge (partial: the names of the parameters, not the form) are equal; the task text is the same word for word. Only the world differs.
The rules came before the run. The world, the tolerance, and the rule for reporting the agent were fixed in pbc.discovery.force_lab before the run was recorded.
The agent is untrusted and its words are data. The score is a function of its tool calls.

Testing a familiar law

A law is rejected by its residuals, not by its plausibility. Fit the familiar law $F = k\,m\,r^{-2}$ to the readings, with only $k$ free. With $N$ readings of noise $\sigma$ , the chi-square of the residuals,

$\chi^2 = \sum_{i=1}^{N}\left(\frac{F_i - k\,m_i\,r_i^{-2}}{\sigma}\right)^2,$

has an expected value of $N-1$ , the degrees of freedom, and a standard deviation of $\sqrt{2(N-1)}$ when the law is right. The protocol accepts a fit when the chi-square is within three standard deviations of its expectation. A chi-square far above that says the law does not describe the readings, whatever the strength.

A modified exponent shows in a simple quantity: $F r^2/m$ . Under the inverse square it is the same at every distance. Under $r^{-p}$ it changes as $r^{\,2-p}$ , falling with distance if $p > 2$ and rising if $p < 2$ . Its trend across the probed distances is the signature of the modification, and the free fit of the force-law lesson, with both $k$ and $p$ free, measures it.

In-sample and out-of-sample. A fit that passes the chi-square has described the readings where they were taken. It need not predict anywhere else. A sum of several powers can follow a modified law closely over the probed distances and part from it beyond them, and the held-out settings of the protocol, beyond the largest distance the tools accept, are where that shows.

What the agent may and may not do

measure: Set the controls and read the force (N), with noise. Each call uses one of 20 probes.
- distance: distance from the centre (m); allowed values 1 to 4
- mass: mass of the test body (kg); allowed values 0.5 to 5
submit: State your claim as the parameters of the law and end the run. It uses no probe.
- strength: coefficient of the law (N m^p/kg); allowed values 0.1 to 100
- exponent: power p of the distance (1); allowed values 0 to 6

The agent may call measure at most 20 times and submit to state its claim, in at most 12 messages of at most 4 tool calls each, and write text. It may not run code, read a file, reach the network, or call the judge.

def same_task(experiment: Experiment) -> None:
    """Whether the agent was given what the agent of the force-law lesson
    was given; record the table."""
    same = task_of(RUNS[RUN].protocol, RUNS[RUN].condition) == task_of(
        RUNS["black-box"].protocol, RUNS["black-box"].condition
    )
    experiment.table(
        "d3-same-task",
        [
            (
                "the system prompt and the task are those of the force-law lesson",
                "yes" if same else "no",
            )
        ],
        [("check", "", ""), ("result", "", "")],
        "The agent was given word for word what the agent of the force-law"
        " lesson was given.",
    )

src/pbc/mechanics/experiments/modified_force_law.py, lines 55 to 72, read from the repository when this page was built.

The agent was given word for word what the agent of the force-law lesson was given.
check	result
the system prompt and the task are those of the force-law lesson	yes

Code

The familiar-law baseline: the probes fitted with the exponent held at the inverse square. It is linear in the strength, so the fit is one sum.

def fit_familiar(
    world: Environment, data: Dataset | None = None, name: str = "inverse-square law"
) -> BaselineResult:
    """The probes of ``world``, its grid unless ``data`` is given, fitted with
    the exponent held at the inverse square and only the strength free
    (weighted least squares, which is linear in the strength).

    It succeeds only when that fit is consistent with the noise, as every
    baseline must; on a modified law it is not, and the result says so."""
    if not isinstance(world, CentralForceEnvironment):
        raise ValueError("the familiar law is a law of the central force")
    if data is None:
        data = grid_probe(world)
    x = np.array(
        [s["mass"] * s["distance"] ** -FAMILIAR_EXPONENT for s in data.settings]
    )
    strength = float(np.sum(x * data.values) / np.sum(x * x))
    chi2 = float(np.sum(((data.values - strength * x) / data.sigma) ** 2))
    dof = len(data.values) - 1
    claim = {"strength": strength, "exponent": FAMILIAR_EXPONENT}
    error = world.held_out_error(claim)
    reason = ""
    if not consistent(chi2, dof):
        reason = (
            f"chi-square {chi2:.3g} for {dof} degrees of freedom: the"
            " inverse-square law does not describe the measurements"
        )
    return BaselineResult(name, len(data.settings), not reason, claim, error, reason)

src/pbc/discovery/force_lab.py, lines 286 to 313, read from the repository when this page was built.

The sparse baseline looks for the force as a short sum of the terms $m\,r^{-n}$ for whole numbers n; a law with a fractional exponent is not such a sum, but a sum can resemble it over a range.

def sparse_identification(
    environment: Environment,
    data: Dataset,
    library: Mapping[str, Callable[[Mapping[str, float]], float]],
    threshold: float = 0.2,
    name: str = "sparse-identification",
) -> BaselineResult:
    """Sequentially thresholded least squares over a library of terms: fit the
    coefficients, drop those smaller than ``threshold`` times the largest, and
    fit the rest again.

    It finds a law only if the law is a sum of the library's terms; for one
    that is not, the best sparse sum does not fit and the result says so."""
    if not library:
        return _failed(name, data, "the environment has no library of terms")
    terms = list(library)
    design = (
        np.array([[library[t](s) for t in terms] for s in data.settings]) / data.sigma
    )
    target = data.values / data.sigma
    active = np.ones(len(terms), dtype=bool)
    coefficients = np.zeros(len(terms))
    for _ in range(10):
        if not active.any():
            break
        coefficients[:] = 0.0
        coefficients[active], *_ = np.linalg.lstsq(
            design[:, active], target, rcond=None
        )
        small = active & (np.abs(coefficients) < threshold * np.abs(coefficients).max())
        if not small.any():
            break
        active &= ~small
    coefficients[~active] = 0.0
    chi2 = float(np.sum((target - design @ coefficients) ** 2))
    dof = len(target) - int(active.sum())
    claim = {t: float(c) for t, c in zip(terms, coefficients, strict=True) if c != 0.0}

    def predict(settings: Mapping[str, float]) -> float:
        return sum(c * library[t](settings) for t, c in claim.items())

    error = environment.score_predictor(predict)
    reason = ""
    if not claim:
        reason = "no term of the library survives the threshold"
    elif not consistent(chi2, dof):
        reason = (
            f"chi-square {chi2:.3g} for {dof} degrees of freedom: no sparse sum"
            " of the library's terms describes the measurements"
        )
    return BaselineResult(
        name, len(data.settings), not reason, claim or None, error, reason
    )

src/pbc/discovery/baselines.py, lines 158 to 210, read from the repository when this page was built.

Worked examples

The familiar law against the readings

def the_familiar_law(experiment: Experiment):
    """The inverse-square law fitted to the grid's readings of the modified
    world; record the table and return the fit."""
    familiar = familiar_law(RUN)
    experiment.table(
        "d3-familiar",
        [
            ("probes", f"{familiar.probes}"),
            (
                "fitted strength, exponent held at two",
                f"{familiar.claim['strength']:.4g}",
            ),
            ("held-out error", f"{familiar.held_out_error:.3g}"),
            ("verdict", "rejected" if not familiar.success else "accepted"),
            ("reason", familiar.reason),
        ],
        [("quantity", "", ""), ("value", "", "")],
        "The inverse-square law fitted to the grid baseline's readings of the"
        " modified world.",
    )
    return familiar

src/pbc/mechanics/experiments/modified_force_law.py, lines 75 to 95, read from the repository when this page was built.

The inverse-square law fitted to the grid baseline's readings of the modified world.
quantity	value
probes	20
fitted strength, exponent held at two	11.58
held-out error	1.91
verdict	rejected
reason	chi-square 1.68e+05 for 19 degrees of freedom: the inverse-square law does not describe the measurements

Numerically verified (the simulated modified central force, the grid of the protocol's budget): The inverse-square law does not describe the readings of this world: its chi-square is far beyond what the noise gives, and its prediction at the held-out distances is off by more than the tolerance.

A baseline identifies the modified law

def the_baselines(experiment: Experiment, familiar):
    """The baselines without a model, with the familiar law beside them;
    record the table and return the baselines."""
    results = baselines(RUN)
    experiment.table(
        "d3-baselines",
        [
            (
                b.name,
                b.claim.get("exponent") if b.claim else None,
                b.held_out_error,
                "yes" if b.success else "no",
                "yes" if recovered(b) else "no",
                b.reason or "fit consistent with the noise",
            )
            for b in (*results, familiar)
        ],
        [
            ("method", "", ""),
            ("claimed exponent", "", ".4g"),
            ("held-out error", "relative", ".3g"),
            ("fit consistent", "", ""),
            ("recovered", "", ""),
            ("note", "", ""),
        ],
        "The baselines without a model on the modified world, with the familiar"
        " law beside them.",
    )
    return results

src/pbc/mechanics/experiments/modified_force_law.py, lines 98 to 126, read from the repository when this page was built.

The baselines without a model on the modified world, with the familiar law beside them.
method	claimed exponent	held-out error (relative)	fit consistent	recovered	note
grid + least squares	2.499	0.00295	yes	yes	fit consistent with the noise
random + least squares	2.5	0.00127	yes	yes	fit consistent with the noise
grid + sparse identification		0.158	yes	no	fit consistent with the noise
grid + inverse-square law	2	1.91	no	no	chi-square 1.68e+05 for 19 degrees of freedom: the inverse-square law does not describe the measurements

Numerically verified (the simulated modified central force, one seed, the protocol's budget): Least squares of the free family, on the grid and on random settings, recovers the modified law: its claim predicts the held-out distances within the tolerance. Its exponent is the claim of a fit, not the hidden value, and this page prints no hidden value. Sparse identification shows the other lesson of this world: its sum of whole powers is consistent with the readings, so the chi-square accepts it, and it still fails beyond the probed range. Its exponent column is empty because its claim is a sum of terms, not one power.

The agent’s run

    recording, transcript, score = replay_run(RUN, LESSON)

src/pbc/mechanics/experiments/modified_force_law.py, line 331, read from the repository when this page was built.

Recorded run. The model messages below were recorded from claude-sonnet-5-5 in a Claude Code session on 2026-10-16. This build did not call a model. It ran every tool call again; the results shown are the ones it computed.

Step 1. Model message (recorded):

Starting measurements. First probes at mass 1 and 4 kg, at distances 1, 2 and 4 m, to see the scale of the force and how it depends on each control.

measure(distance=1, mass=1): recomputed by this build: force = 12.0041
measure(distance=2, mass=1): recomputed by this build: force = 2.12805
measure(distance=4, mass=1): recomputed by this build: force = 0.3823
measure(distance=1, mass=4): recomputed by this build: force = 47.9577
measure(distance=2, mass=4): rejected before it ran: at most 4 tool calls per message

Step 2. Model message (recorded):

The measurements fit F = k * m / d^p. Mass dependence is linear: F/m is 12.00 at m=1 and 11.99 at m=4 for d=1. Over d = 1, 2, 4 m the force drops by factors of 5.64 and 5.57, which gives p of about 2.5. The strength k is F*d^2.5/m, which is 12.0, 12.04 and 12.2 across the three distances, so about 12.05. My fifth probe (d=2, m=4) was rejected by the 4-calls-per-message limit and I did not repeat it, because the other four already pin down both parameters. Submitting strength 12.05 and exponent 2.5.

submit(strength=12.05, exponent=2.5): recomputed by this build: submitted = 1

Step 3. Model message (recorded):

The submission went through (result: submitted). I claimed strength 12.05 and exponent 2.5, from the law F = 12.05 * m / d^2.5. I used 4 of the 20 probes. A fifth probe (d=2, m=4) was rejected by the 4-calls-per-message limit and was not repeated. The run is complete.

The model ended the run with a message that requests no tool.

The model was the Claude model of the force-law lesson, in a Claude Code session on 2026-10-16, as a fresh sub-agent that knew nothing of that lesson’s run. It was given the system prompt, the task, and the tool descriptions, and nothing else, and chose every call after reading the readings the harness computed.

def score_table(experiment: Experiment, score) -> None:
    """The agent's run scored again from its tool calls; record the table."""
    experiment.table(
        "d3-score",
        [
            ("probes used", f"{score.probes}"),
            ("calls refused", f"{score.refused}"),
            ("claimed strength", f"{score.claim['strength']:.4g}"),
            ("claimed exponent", f"{score.claim['exponent']:.4g}"),
            ("held-out error", f"{score.held_out_error:.3g}"),
            ("tolerance", f"{TOLERANCE:g}"),
            ("outcome", score.outcome),
        ],
        [("quantity", "", ""), ("value", "", "")],
        "The agent's run, scored again by the build from its tool calls alone.",
    )

src/pbc/mechanics/experiments/modified_force_law.py, lines 129 to 144, read from the repository when this page was built.

The agent's run, scored again by the build from its tool calls alone.
quantity	value
probes used	4
calls refused	1
claimed strength	12.05
claimed exponent	2.5
held-out error	0.00417
tolerance	0.05
outcome	recovered

Numerically verified (one recorded run, one seed, the simulated modified central force): The agent’s claim is scored recovered by the protocol: it did not fall back on the inverse square, and its law predicts the held-out distances within the tolerance. The agent spent 4 probes, and one more call was refused because the message held more calls than the harness runs. It submitted a round exponent, as the agent of the force-law lesson did, but not the familiar one.

Is the round exponent what its readings support? Fit the free family and the familiar law to exactly the agent’s readings:

def the_agents_readings(experiment: Experiment, transcript, score):
    """The free family and the familiar law fitted to exactly the agent's
    readings; record the two tables and return the readings and the free
    fit."""
    own_data = probes(RUN, transcript)
    own_fit = fit_probes(RUN, own_data)
    own_familiar = familiar_law(
        RUN, own_data, "the agent's readings, inverse-square law"
    )
    exponent, strength = float(own_fit.params[1]), float(own_fit.params[0])
    experiment.table(
        "d3-agent-readings",
        [
            ("free fit, exponent", exponent, float(own_fit.stderr[1])),
            ("free fit, strength", strength, float(own_fit.stderr[0])),
            (
                "agent's claimed exponent minus fit, in standard errors",
                (score.claim["exponent"] - exponent) / float(own_fit.stderr[1]),
                None,
            ),
            (
                "agent's claimed strength minus fit, in standard errors",
                (score.claim["strength"] - strength) / float(own_fit.stderr[0]),
                None,
            ),
        ],
        [("quantity", "", ""), ("value", "", ".4g"), ("standard error", "", ".2g")],
        "The agent's own readings: the free fit with its standard errors.",
    )
    experiment.table(
        "d3-agent-familiar",
        [
            (
                own_familiar.name,
                "accepted" if own_familiar.success else "rejected",
                own_familiar.reason or "fit consistent with the noise",
            )
        ],
        [("fit", "", ""), ("verdict", "", ""), ("reason", "", "")],
        "The familiar law fitted to the agent's own readings.",
    )
    return own_data, own_fit

src/pbc/mechanics/experiments/modified_force_law.py, lines 147 to 188, read from the repository when this page was built.

The agent's own readings: the free fit with its standard errors.
quantity	value	standard error
free fit, exponent	2.493	0.013
free fit, strength	11.99	0.0049
agent's claimed exponent minus fit, in standard errors	0.5261
agent's claimed strength minus fit, in standard errors	12.31

The familiar law fitted to the agent's own readings.
fit	verdict	reason
the agent's readings, inverse-square law	rejected	chi-square 2.22e+03 for 3 degrees of freedom: the inverse-square law does not describe the measurements

Even four readings reject the inverse square. The agent’s exponent lies within one standard error of the fit to its readings. Its strength does not: the agent read it off its readings at a single distance, and it lies many standard errors from the fit. The difference is small against the tolerance, which is why the claim still recovers the law, but it is a difference between the claim and its evidence, and the score alone would not show it. The comparison of the agent with the baselines:

def beside_the_baselines(experiment: Experiment, score, results) -> None:
    """The agent and the baselines on the modified world; record the
    table."""
    rows = comparison((score,), results, TOLERANCE)
    experiment.table(
        "d3-comparison",
        [
            (
                r["who"],
                r["probes"],
                r["held_out_error"],
                "yes" if r["recovered"] else "no",
                r["note"],
            )
            for r in rows
        ],
        [
            ("method", "", ""),
            ("probes", "", "d"),
            ("held-out error", "", ".3g"),
            ("recovered", "", ""),
            ("note", "", ""),
        ],
        "The agent and the baselines on the modified world, with the same budget.",
    )

src/pbc/mechanics/experiments/modified_force_law.py, lines 191 to 215, read from the repository when this page was built.

The agent and the baselines on the modified world, with the same budget.
method	probes	held-out error	recovered	note
agent, seed 0	4	0.00417	yes	recovered
grid + least squares	20	0.00295	yes	fit consistent with the noise
random + least squares	20	0.00127	yes	fit consistent with the noise
grid + sparse identification	20	0.158	no	fit consistent with the noise

The signature of the modification

def draw_unlearning(results, familiar, own_data) -> Figure:
    """The signature of the modification: the readings of the grid and of
    the agent, the free fit, and the inverse-square fit."""
    grid = grid_probe(environment(RUN))
    free = {b.name: b for b in results}["grid + least squares"].claim
    r = np.linspace(1.0, 4.0, 100)
    fig, ax = plt.subplots(figsize=(6.4, 3.6))
    ax.plot(
        [s["distance"] for s in grid.settings],
        signature(grid),
        "o",
        color="#9aa4ae",
        markersize=4,
        label="grid baseline readings",
    )
    ax.plot(
        [s["distance"] for s in own_data.settings],
        signature(own_data),
        "o",
        color="#93003a",
        markersize=7,
        label="agent's readings",
    )
    ax.plot(
        r,
        free["strength"] * r ** (2 - free["exponent"]),
        color="#00429d",
        label="free fit",
    )
    ax.axhline(
        familiar.claim["strength"],
        color="#57606a",
        linestyle="--",
        label="inverse-square fit",
    )
    ax.set_xlabel("distance (m)")
    ax.set_ylabel("force × distance² per unit mass (N m²/kg)")  # noqa: RUF001
    ax.legend(fontsize=8)
    return fig

src/pbc/mechanics/experiments/modified_force_law.py, lines 226 to 264, read from the repository when this page was built.

A chart of force times distance squared per unit mass against distance from one to four metres. A horizontal line is the inverse-square fit. A falling curve is the free fit. Grey dots of the grid baseline and four larger dots of the agent lie along the falling curve and cross the horizontal line, above it at short distance and below it at long distance. — Figure 1: simulated The force times the distance squared, per unit mass, against the distance: under the inverse square it would be the horizontal line the familiar fit draws; the readings of the grid and of the agent fall with distance, along the free fit.

The readings are simulated and recomputed by this build. A horizontal row of points would have meant the inverse square; the points fall instead, and the familiar fit, the best horizontal line, misses both ends. The free fit follows them, and the agent’s exponent agrees with it within one standard error.

Run the agent yourself

This needs your own key and costs a few cents. The build does not run it, so the commands are marked.

Not verified. The build of this site did not run or check the material in this box.

uv sync --locked --no-dev --extra openai
export OPENAI_API_KEY="..."
cd site/lessons/agents-llm/07-a-modified-force-law
uv run --locked --no-dev --extra openai python -m pbc.discovery.force_lab modified

A live run may fall back on the inverse square. If it does, the score says so, and so should anyone who reports it.

Limits

One run, and it succeeded. The page shows that one agent, once, did not fall back on the familiar law. It does not show that agents generally unlearn it, and a different model or prompt may not.
The modification is of one kind. The law is still a power of the distance, and the claim is a power. A modification of another form, a force that changes sign or depends on the velocity, cannot be stated as a claim here.
A round claim and a fit agreed, in part. The agent’s exponent was round and lay within one standard error of its readings; its strength did not. The score does not distinguish a well-rounded measurement from a lucky guess; the force-law lesson shows the same.
Simulated, not measured. Nothing here says that any force in nature is modified, or that the inverse square fails anywhere.
The recording is not an API run. It is a Claude Code session, told by the session to answer in the harness’s format and to use no tool of its own; the session reported no token count or price.

Self-check. Suppose the agent had submitted the inverse square, with a strength that fits the near readings, while least squares recovered the modified law. How should the page report the run?

As a partial success, since the agent found the right strength at short distance.

The protocol scores the claim at the held-out settings, and there the familiar law is far off. A partial success is a reading the score does not give.
As a failure: the claim is not recovered at the held-out settings, the familiar law fails the chi-square, and the baseline shows that the readings allowed the modified law to be found.

Right. The baseline is what makes the failure informative: it shows that the information was there.
By lowering the tolerance’s strictness, since the modified law is unusual.

The tolerance was fixed before the run. Widening it after seeing the result would turn a failure into a success by changing the test.

Solution to the self-check

As a failure, with its score. The protocol’s verdict depends on the held-out error alone, and the familiar law misses there by far more than the tolerance. The baseline’s recovery from the same budget shows that the failure was the agent’s and not the experiment’s: the readings contained the modified law. A lesson that explained the failure away, by changing the tolerance or by crediting the right strength at short distance, would be reporting what it wished had happened.

Exercises

Exercise 1. How small a modification can the familiar-law test detect with this budget and noise? Make worlds of your own with strength 12 and exponents 2.002, 2.005, 2.01, 2.02, and 2.05, fit the inverse-square law to the grid of each, and compare the verdict of the chi-square with the held-out error.

Solution to exercise 1

def small_modifications(experiment: Experiment) -> None:
    """Exercise 1: the inverse-square law fitted to worlds with slightly
    modified exponents; record the table."""
    rows = []
    for exponent in (2.002, 2.005, 2.01, 2.02, 2.05):
        world = CentralForceEnvironment(strength=12.0, exponent=exponent, budget=BUDGET)
        result = fit_familiar(world)
        rows.append(
            (
                exponent,
                "accepted" if result.success else "rejected",
                result.held_out_error,
                "yes" if result.held_out_error <= TOLERANCE else "no",
            )
        )
    experiment.table(
        "small-modifications",
        rows,
        [
            ("exponent of the world", "", ".4g"),
            ("chi-square test", "", ""),
            ("held-out error", "relative", ".3g"),
            ("within the tolerance", "", ""),
        ],
        "The inverse-square law fitted to worlds of your own with slightly"
        " modified exponents.",
    )

src/pbc/mechanics/experiments/modified_force_law.py, lines 267 to 293, read from the repository when this page was built.

The inverse-square law fitted to worlds of your own with slightly modified exponents.
exponent of the world	chi-square test	held-out error (relative)	within the tolerance
2.002	accepted	0.00418	yes
2.005	rejected	0.0105	yes
2.01	rejected	0.0212	yes
2.02	rejected	0.0428	yes
2.05	rejected	0.111	no

The chi-square rejects the familiar law already at modifications whose held-out error is still within the tolerance. With twenty readings of this precision the test sees a modification before it matters for prediction at the held-out distances. The smallest modification the test catches is set by the noise and the budget; with noisier readings or fewer of them, a modified world could pass as the familiar one.

Exercise 2. Use the ratio formula of the force-law lesson on two of the agent’s readings: the ones at 1 m and 4 m with a mass of 1 kg. What exponent do they give, and how does it compare with the agent’s claim?

Solution to exercise 2

def two_point_exponent(experiment: Experiment, own_data, own_fit, score) -> None:
    """Exercise 2: the exponent from the agent's readings at 1 m and 4 m with
    a mass of 1 kg, beside its claim and the fit; record the table."""
    by_setting = {
        (s["distance"], s["mass"]): v
        for s, v in zip(own_data.settings, own_data.values, strict=True)
    }
    near, far = by_setting[(1.0, 1.0)], by_setting[(4.0, 1.0)]
    two_point = math.log(near / far) / math.log(4.0)
    experiment.table(
        "two-point",
        [
            ("from the two readings", two_point),
            ("agent's claim", score.claim["exponent"]),
            ("free fit to all its readings", float(own_fit.params[1])),
        ],
        [("estimate of the exponent", "", ""), ("value", "", ".4g")],
        "The exponent from two of the agent's readings, beside its claim and the fit.",
    )

src/pbc/mechanics/experiments/modified_force_law.py, lines 296 to 314, read from the repository when this page was built.

The exponent from two of the agent's readings, beside its claim and the fit.
estimate of the exponent	value
from the two readings	2.486
agent's claim	2.5
free fit to all its readings	2.493

Two readings at the ends of the range already give an exponent close to the claim. The fit to all four readings is close to both, and its standard error says how much the noise allows. The agent’s round number is consistent with both estimates.

Replay this experiment

This page was built from commit 6417e2ff90258bd2d2ba21d217eb7666fd24f680 of the repository.

In a clone at that commit, with the project’s Python packages installed, this command runs the experiment of this page:

uv run --locked --no-dev python -m pbc.mechanics.experiments.modified_force_law results

Files, tests, and every command

The files behind it:

site/lessons/agents-llm/07-a-modified-force-law/index.qmd: this page, with every code cell.
src/pbc/discovery/force_lab.py: code the experiment runs.
src/pbc/discovery/environment.py: code the experiment runs.
src/pbc/discovery/protocol.py: code the experiment runs.
src/pbc/discovery/baselines.py: code the experiment runs.
src/pbc/mechanics/experiments/modified_force_law.py: code the experiment runs.
src/pbc/experiment.py: code the experiment runs.
tests/unit/test_force_lab.py: tests of that code.
tests/unit/test_discovery.py: tests of that code.
tests/unit/test_modified_force_law_experiment.py: tests of that code.

With Git and uv installed (see the setup page), run:

git clone https://github.com/taspinar/physics-by-construction.git
cd physics-by-construction
git checkout 6417e2ff90258bd2d2ba21d217eb7666fd24f680
uv sync --locked --no-dev
uv run --locked --no-dev python -m pbc.mechanics.experiments.modified_force_law results

The experiment writes every figure of this page into results/, as <label>.png, and every number and table of the page, as the page shows it, into results/results.json. Its numbers match this page to the digits shown, and its figure files are the ones the page shows; digits beyond those can differ between machines, because floating-point libraries differ. Change the code of the experiment and run it again to see what changes.

The experiment replays the recorded run of the agent: it runs every tool call of the recording again and needs no model and no key.

Not verified. The build of this site did not run or check the material in this box.

Optional: the live run. It asks a language model to run the experiment again, with your own account. It needs an OpenAI account and its API key in the variable OPENAI_API_KEY, and the optional dependency openai; a run costs a few cents. The key stays in your shell, never in a file.

# needs: an OpenAI account, its API key, and the extra openai
uv sync --locked --no-dev --extra openai
export OPENAI_API_KEY="..."   # this shell only
uv run --locked --no-dev --extra openai python -m pbc.discovery.force_lab modified