Three reviewers for one piece of code

Three review agents, one for the physics, one for the numerical implementation, and one for the evidence, each with a short list of checks from the scientific validation contract, review a correct stepper and a faulty one; where they disagree, the build recomputes every check and the failed one stands.

Course

Classical mechanics, lesson 17 of 28

Slot

AI4, number 5 of the 9 slots of the path Physics reasoning and scientific agents

Methods

Simulation, LLM agents

Difficulty

Advanced, level 3 of 3

What you’ll learn

• Give each of several review agents its own bounded list of checks over the same code
• Read a disagreement between reviewers as a difference in what each one checked
• Decide a review from recomputed check results, not from the words of a reviewer

Prerequisites

An agent runs an experiment, Verifying scientific code.

Before you begin

• Python: reading a short function and a dictionary

Related

none

Previous

A counterexample hunter

Next

An agent that designs the measurements

With scripts, this page keeps a record in your browser of whether you completed the lesson. Nothing is stored without them.

With scripts, this place suggests what to do next, with the reason, from the progress saved in your browser. Without them, follow the learning path.

A piece of simulation code is handed over with a sentence about what it does: here, that it is a first-order method for Newton’s second law and that the energy of an oscillator it steps stays in a band. Someone has to decide whether to believe the sentence. The lesson on verifying scientific code gave that someone five checks. This lesson gives the checks to language models, three of them, with different jobs, and keeps the one thing a model should not be trusted with out of their hands: the verdict.

Each reviewer is one run of the loop of the first agent lesson. One looks at the physics, one at the numerical implementation, one at the evidence. Each has its own short list of tools, and each tool runs a check of the contract on one of two pieces of code. One piece is the symplectic Euler stepper of the course. The other is a variant with a fault that a reader could miss. The reviewers do not know which is which, and they do not see each other’s work.

What the page shows is what the build recomputed. The reviewers’ words are a recording; every check they asked for is run again, and the verdicts below come from those runs.

Assumptions

1. The claim comes first. Both pieces claim first order and a bounded energy of the oscillator. A reviewer is asked whether the code does what it claims, and a claim written after the checks would be a description.
2. The checks are the contract’s, and they are shared. The three reviewers use the same functions with the same tolerances, fixed before any piece was run. They differ in which checks they may run, not in what a check means. If they could choose their own tolerances, a disagreement might only be a disagreement about tolerances.
3. A reviewer’s words are data. A model may write that code passes when it does not, or the reverse. The verdict of this page is read from the results of the tools, never from the text.
4. A pass covers what was run. A reviewer that ran three checks says nothing about the other two. Where reviewers differ, the cause is usually that they ran different checks, and the resolution rule is written for that case.
5. The oscillator is a fair reference, and only one. A fault that matters for a force that depends on the velocity could pass every check here. The contract says what to add for a new model, and the evidence reviewer is asked to say what its evidence does not cover.
6. A recording is a record. It shows what one model once did with this task. The numbers in it are checked; the reasoning is not.

Three jobs, one rule

The reviewers are three runs of the same loop with different system prompts and different allowlists:

• the physics reviewer may run the checks limits and bounded-energy: does the code reproduce the motions with a closed form, and does it keep the quantity it says it conserves;
• the numerics reviewer may run clock, purity, and convergence-order: does a step behave as a step of a method of the claimed order must;
• the evidence reviewer may reproduce any of the five checks in the order of the contract, and may ask for the raw error against the exact oscillator at a number of time steps, so that it can write the evidence in the contract’s form, with the name of the check and the call from its own calls, and say what is not covered.

There is no new machinery. The boundary of the first agent lesson holds unchanged: a model can only name a tool of its list and give numbers within bounds, the loop stops after a fixed number of messages, and the harness has no code that evaluates what a model writes. A reviewer has even less to ask for than the experimenter of the first lesson: its tools take the number of a piece of code, 1 or 2, and the number of a check.

Disagreement and resolution

The three runs are compared by the build, not by a fourth model. Each tool result carries the numbers of a finding in the contract’s form (rule 3): the measured value, the tolerance, and whether it passed. The name of the check and the call that reproduces it are in the reviewer’s own tool call, and the claim is the one both pieces make; the build recomputes the numbers, not these. The rule that combines them is one sentence:

A check that failed when it was recomputed stands, whatever the other reviewers said. A reviewer that did not run that check has no standing against it.

So a disagreement is not settled by a vote. When two reviewers recompute the same check, they get the same number, because the check is deterministic; if they did not, the build reports a conflict. When one reviewer found a failure and another did not, the second ran other checks, and the list of what each ran says which.

Code

The allowlists of the three reviewers, as the models see them:

The physical assumptions reviewer

• check_limits: No force gives uniform motion; a constant force gives the exact velocity.
  • candidate: the piece of code to check: 1 or 2; allowed values 1 to 2
• check_bounded_energy: The energy band of the oscillator over 100 periods, last 5 periods against first 5; measured is the growth factor.
  • candidate: the piece of code to check: 1 or 2; allowed values 1 to 2

The numerical implementation reviewer

• check_clock: A step of length dt advances the time by dt.
  • candidate: the piece of code to check: 1 or 2; allowed values 1 to 2
• check_purity: A step depends only on its arguments, not on earlier steps.
  • candidate: the piece of code to check: 1 or 2; allowed values 1 to 2
• check_convergence_order: The fitted order of the error against the exact oscillator; measured is the order, tolerance the allowed distance from the claim.
  • candidate: the piece of code to check: 1 or 2; allowed values 1 to 2

The evidence reviewer

• reproduce_check: Run one check of the contract on a candidate and return its evidence: measured value, tolerance, and passed (1 or 0).
  • check: the check, by its place in the contract: 1 clock, 2 purity, 3 limits, 4 convergence-order, 5 bounded-energy; allowed values 1 to 5
  • candidate: the piece of code to check: 1 or 2; allowed values 1 to 2
• oscillator_error: The largest distance of the candidate from the exact oscillator over two periods, for a number of time steps per period.
  • candidate: the piece of code to check: 1 or 2; allowed values 1 to 2
  • steps_per_period: time steps in one period of the oscillator; allowed values 20 to 800

Every tool returns the evidence of the contract as numbers. The verdicts of the page are read from them, and the text of a reviewer is not read at all:

def verdicts(reviewer: Reviewer, transcript: Transcript) -> list[Verdict]:
    """The checks a reviewer ran, from the recomputed results of its tools.

    The text of the model is not read. A call that was refused, and a tool
    that is not a check (``oscillator_error``), gives no verdict.
    """
    found = []
    for step in transcript.steps:
        for call, result in zip(step.message.tool_calls, step.results, strict=True):
            if not result.ok:
                continue
            if call.name == "reproduce_check":
                check = CHECKS[call.arguments["check"] - 1]
            elif call.name.startswith("check_"):
                check = call.name.removeprefix("check_").replace("_", "-")
            else:
                continue
            found.append(
                Verdict(
                    reviewer.id,
                    call.arguments["candidate"],
                    check,
                    result.content["measured"],
                    result.content["tolerance"],
                    bool(result.content["passed"]),
                )
            )
    return found

src/pbc/agents/review_lab.py, lines 270 to 297, read from the repository when this page was built.

The resolution is the rule above and a list of the checks no reviewer ran:

def resolve(all_verdicts: list[Verdict], candidate: int) -> Resolution:
    """Combine the verdicts on ``candidate`` by one rule.

    A failed recomputed check stands, whoever else passed the code: a reviewer
    that did not run the check has no standing against it. A check that two
    reviewers recomputed with different outcomes is a conflict (the checks
    are deterministic, so it means the reviewers did not run the same check).
    """
    mine = [v for v in all_verdicts if v.candidate == candidate]
    outcomes: dict[str, set[bool]] = {}
    for verdict in mine:
        outcomes.setdefault(verdict.check, set()).add(verdict.passed)
    flagged = sorted({v.reviewer for v in mine if not v.passed})
    clear = sorted({v.reviewer for v in mine} - set(flagged))
    return Resolution(
        candidate=candidate,
        failed=tuple(c for c in CHECKS if False in outcomes.get(c, ())),
        flagged_by=tuple(flagged),
        clear_by=tuple(clear),
        not_covered=tuple(c for c in CHECKS if c not in outcomes),
        conflicts=tuple(c for c in CHECKS if len(outcomes.get(c, ())) > 1),
    )

src/pbc/agents/review_lab.py, lines 322 to 343, read from the repository when this page was built.

Worked examples

The two pieces of code

Piece 1 is the stepper of the course. Piece 2 is a variant that looks like it: it moves the particle with the old velocity, so the two updates are in the wrong order. Both claim the order and the bounded energy of Claim(order=1, bounded_energy=True).

def symplectic_euler_step(state: State, acceleration: Acceleration, dt: float) -> State:
    """Advance ``state`` by one symplectic Euler step of length ``dt`` (s).

    The velocity changes first, with the acceleration at the old state;
    then the *new* velocity moves the particle. One change of order against
    explicit Euler, with the same cost; it is the simplest method that keeps
    the energy of an oscillator bounded.
    """
    v = state.v + acceleration(state.t, state.x, state.v) * dt
    return State(t=state.t + dt, x=state.x + v * dt, v=v)

src/pbc/mechanics/integrators.py, lines 16 to 25, read from the repository when this page was built.

def explicit_euler_as_symplectic(
    state: State, acceleration: Acceleration, dt: float
) -> State:
    """A "symplectic Euler" that moves the particle with the old velocity:
    the two updates are in the wrong order, which is explicit Euler."""
    x = state.x + state.v * dt
    v = state.v + acceleration(state.t, state.x, state.v) * dt
    return State(t=state.t + dt, x=x, v=v)

src/pbc/verification/faulty.py, lines 76 to 83, read from the repository when this page was built.

The second function is wrong on purpose: it is in the gallery of faulty steppers of the earlier lesson, the only module outside the lessons and tests that imports it is the one that sets up this review.

The three recorded runs

The cell replays the three recordings. The model’s messages are the recorded ones; each tool result is computed by this build, now.

    replays = replay_fixtures(LESSON)
    review = ReviewDisplay(
        {name: transcript for name, (_, transcript) in replays.items()}
    )
    growth = next(
        v.measured
        for v in review.verdicts
        if v.candidate == 2 and v.check == "bounded-energy"
    )

src/pbc/mechanics/experiments/scientific_review.py, lines 153 to 161, read from the repository when this page was built.

Recorded run. The model messages below were recorded from claude-sonnet-5-5 (run by Claude Code in a session, not through a provider API) on 2026-10-10. This build did not call a model. It ran every tool call again; the results shown are the ones it computed.

Step 1. Model message (recorded):

Both pieces claim first order and a bounded oscillator energy. The physical claims I can test are the limits (no force, constant force) and the energy band. I run both checks on both pieces.

• check_limits(candidate=1): recomputed by this build: measured = 7.10543e-15, tolerance = 1e-12, passed = 1
• check_limits(candidate=2): recomputed by this build: measured = 7.10543e-15, tolerance = 1e-12, passed = 1
• check_bounded_energy(candidate=1): recomputed by this build: measured = 1.00318, tolerance = 1.5, passed = 1
• check_bounded_energy(candidate=2): recomputed by this build: measured = 9.13717e+49, tolerance = 1.5, passed = 0

Step 2. Model message (recorded):

Limits: both pieces reproduce uniform motion and the exact velocity under a constant force (errors about 7e-15 against a tolerance of 1e-12). Energy: piece 1 keeps its energy band almost unchanged between the first and the last five periods (growth factor 1.003 against an allowed 1.5). Piece 2 fails: the band grows by a factor of 9e49. I accept piece 1 and reject piece 2. My acceptance rests on the oscillator being a fair reference: a force that depends on the velocity is not covered, and neither is a step size other than 1/32 of a period.

The model ended the run with a message that requests no tool.

Recorded run. The model messages below were recorded from claude-sonnet-5-5 (run by Claude Code in a session, not through a provider API) on 2026-10-10. This build did not call a model. It ran every tool call again; the results shown are the ones it computed.

Step 1. Model message (recorded):

I check the clock, the purity of a step, and the order of convergence, first on piece 1.

• check_clock(candidate=1): recomputed by this build: measured = 0, tolerance = 1e-12, passed = 1
• check_purity(candidate=1): recomputed by this build: measured = 0, tolerance = 1e-12, passed = 1
• check_convergence_order(candidate=1): recomputed by this build: measured = 1.03558, tolerance = 0.4, passed = 1

Step 2. Model message (recorded):

Piece 1 passes all three. The same checks on piece 2.

• check_clock(candidate=2): recomputed by this build: measured = 0, tolerance = 1e-12, passed = 1
• check_purity(candidate=2): recomputed by this build: measured = 0, tolerance = 1e-12, passed = 1
• check_convergence_order(candidate=2): recomputed by this build: measured = 1.13394, tolerance = 0.4, passed = 1

Step 3. Model message (recorded):

Both pieces pass all three checks. The clock and the purity errors are 0 against 1e-12. The fitted order is 1.04 for piece 1 and 1.13 for piece 2, both within 0.4 of the claimed order 1. As numerical methods of first order I find no fault in either, and I would accept both. My checks cannot see whether the energy stays bounded: I did not run that check, so this is no statement about it.

The model ended the run with a message that requests no tool.

Recorded run. The model messages below were recorded from claude-sonnet-5-5 (run by Claude Code in a session, not through a provider API) on 2026-10-10. This build did not call a model. It ran every tool call again; the results shown are the ones it computed.

Step 1. Model message (recorded):

I reproduce the five checks in the order of the contract, piece 1 first.

• reproduce_check(check=1, candidate=1): recomputed by this build: measured = 0, tolerance = 1e-12, passed = 1
• reproduce_check(check=2, candidate=1): recomputed by this build: measured = 0, tolerance = 1e-12, passed = 1
• reproduce_check(check=3, candidate=1): recomputed by this build: measured = 7.10543e-15, tolerance = 1e-12, passed = 1
• reproduce_check(check=4, candidate=1): recomputed by this build: measured = 1.03558, tolerance = 0.4, passed = 1

Step 2. Model message (recorded):

Now the last check of piece 1 and the first three of piece 2.

• reproduce_check(check=5, candidate=1): recomputed by this build: measured = 1.00318, tolerance = 1.5, passed = 1
• reproduce_check(check=1, candidate=2): recomputed by this build: measured = 0, tolerance = 1e-12, passed = 1
• reproduce_check(check=2, candidate=2): recomputed by this build: measured = 0, tolerance = 1e-12, passed = 1
• reproduce_check(check=3, candidate=2): recomputed by this build: measured = 7.10543e-15, tolerance = 1e-12, passed = 1

Step 3. Model message (recorded):

The last two checks of piece 2, and the start of the error table of piece 1.

• reproduce_check(check=4, candidate=2): recomputed by this build: measured = 1.13394, tolerance = 0.4, passed = 1
• reproduce_check(check=5, candidate=2): recomputed by this build: measured = 9.13717e+49, tolerance = 1.5, passed = 0
• oscillator_error(candidate=1, steps_per_period=50): recomputed by this build: error = 0.0700937
• oscillator_error(candidate=1, steps_per_period=100): recomputed by this build: error = 0.0332445

Step 4. Model message (recorded):

The rest of the error table.

• oscillator_error(candidate=1, steps_per_period=200): recomputed by this build: error = 0.0161626
• oscillator_error(candidate=2, steps_per_period=50): recomputed by this build: error = 1.19284
• oscillator_error(candidate=2, steps_per_period=100): recomputed by this build: error = 0.483329
• oscillator_error(candidate=2, steps_per_period=200): recomputed by this build: error = 0.218151

Step 5. Model message (recorded):

Piece 1 passes all five checks. Piece 2 passes the first four and fails the fifth, bounded-energy: measured growth 9.1e49 against a tolerance of 1.5. The error table agrees with a first-order method for both: the error of piece 1 is 0.0701, 0.0332, 0.0162 at 50, 100, 200 steps per period (ratios 2.1 and 2.1), that of piece 2 is 1.19, 0.483, 0.218 (ratios 2.5 and 2.2). The table also shows what the order check does not: at 50 steps the error of piece 2 is 17 times that of piece 1 although both have order 1. The evidence supports accepting piece 1 and rejecting piece 2, for the oscillator, steps of 20 to 800 per period, and a force that does not depend on velocity. It does not cover other models, a second reference implementation, or the source of either piece.

The model ended the run with a message that requests no tool.

Where they agree, and where they do not

This is the part of the page that is decided. Every line is built from the recomputed results of the tools, and no text of a reviewer is in it.

• Piece 1
  • physical assumptions: limits passed (measured 7.11e-15, tolerance 1e-12); bounded-energy passed (measured 1, tolerance 1.5)
  • numerical implementation: clock passed (measured 0, tolerance 1e-12); purity passed (measured 0, tolerance 1e-12); convergence-order passed (measured 1.04, tolerance 0.4)
  • evidence: clock passed (measured 0, tolerance 1e-12); purity passed (measured 0, tolerance 1e-12); limits passed (measured 7.11e-15, tolerance 1e-12); convergence-order passed (measured 1.04, tolerance 0.4); bounded-energy passed (measured 1, tolerance 1.5)
  • resolution: not rejected by the checks that were run.
• Piece 2
  • physical assumptions: limits passed (measured 7.11e-15, tolerance 1e-12); bounded-energy failed (measured 9.14e+49, tolerance 1.5)
  • numerical implementation: clock passed (measured 0, tolerance 1e-12); purity passed (measured 0, tolerance 1e-12); convergence-order passed (measured 1.13, tolerance 0.4)
  • evidence: clock passed (measured 0, tolerance 1e-12); purity passed (measured 0, tolerance 1e-12); limits passed (measured 7.11e-15, tolerance 1e-12); convergence-order passed (measured 1.13, tolerance 0.4); bounded-energy failed (measured 9.14e+49, tolerance 1.5)
  • resolution: rejected, because of bounded-energy. The reviewers disagreed: evidence and physics found a failure that numerics did not see, because the checks they ran do not include it.

The numerics reviewer ran three checks and passed both pieces, and was right about what it ran: the faulty piece has the correct order, a correct clock, and no hidden state. It did not run bounded-energy. The physics reviewer and the evidence reviewer did, and the recomputed check failed for piece 2. By the rule, the failure stands.

That is the reason to want several reviewers, and the reason not to count them. Had the numerics reviewer been the only one, the review would have passed a method whose energy grows by a factor of 9.1e+49 over a hundred periods.

A reviewer cannot talk its way past the build

The recorded reviewers wrote honest summaries. A model need not. The cell below plays a reviewer that calls the check on piece 2 and then says the piece passes every check; the verdict is read from the result of the call.

class Boaster:
    model = "a reviewer that does not tell the truth"
    secrets = ()

    def __init__(self):
        self.messages = [
            ModelMessage(
                "I will check the energy.",
                (ToolCall("1", "check_bounded_energy", {"candidate": 2}),),
            ),
            ModelMessage("Piece 2 passes every check, with a wide margin."),
        ]

    def complete(self, system, conversation, tools):
        return self.messages.pop(0)

src/pbc/mechanics/experiments/scientific_review.py, lines 50 to 64, read from the repository when this page was built.

def boasting_reviewer():
    """Exercise 1: a reviewer that runs one check and then claims the
    opposite of its result."""
    physics = REVIEWERS[0]
    transcript = run_agent(
        Boaster(), physics.allowlist, physics.system, physics.task, MAX_STEPS
    )
    print("what it said: ", transcript.steps[-1].message.text)
    for v in verdicts(physics, transcript):
        print(
            f"what it ran:   {v.check} on piece {v.candidate}:",
            "passed" if v.passed else "FAILED",
            f"(measured {v.measured:.1e}, tolerance {v.tolerance})",
        )

src/pbc/mechanics/experiments/scientific_review.py, lines 67 to 80, read from the repository when this page was built.

what it said:  Piece 2 passes every check, with a wide margin.
what it ran:   bounded-energy on piece 2: FAILED (measured 9.1e+49, tolerance 1.5)

The replay keeps the page honest

The replay compares every recomputed result with the recorded one, to a relative and absolute difference of 1e-06. If a check of the contract changes, a recording is no longer a recording of what the code does. Here the tolerance of the exact checks is made tighter than round-off, and the physics reviewer is replayed.

def stricter_check(replays):
    """Exercise 2: the replay of the physics reviewer against a check whose
    tolerance has changed."""
    physics = REVIEWERS[0]
    original = checks.EXACT
    checks.EXACT = 1e-30
    try:
        replay_reviewer(physics, replays["physics"][0])
    except ReplayMismatch as mismatch:
        print(mismatch)
    finally:
        checks.EXACT = original

src/pbc/mechanics/experiments/scientific_review.py, lines 83 to 94, read from the repository when this page was built.

step 1, tool check_limits: passed was 1.0 when recorded and is 0.0 now (tolerance 1e-06). If the change is intended, record the run again with: uv run --extra openai python -m pbc.agents.review_lab --record (with OPENAI_API_KEY set)

The build would fail with that message. The maintainer then runs the reviewers live, reads the new runs, and commits the new recordings.

What the evidence reviewer saw

The evidence reviewer asked for the error against the exact oscillator at three step counts. The figure shows what the build recomputed, with a fresh scan of the same two pieces.

def draw_review_evidence(replays) -> Figure:
    """The error of the two pieces against the steps per period, with the
    experiments the evidence reviewer ran."""
    counts = np.array([20, 40, 80, 160, 320, 640])
    colors = {1: "#00429d", 2: "#93003a"}
    fig, ax = plt.subplots(figsize=(6.4, 3.6))
    for piece, build in CANDIDATES.items():
        ax.loglog(
            counts,
            [oscillator_error(build, n) for n in counts],
            color=colors[piece],
            label=f"piece {piece}",
        )
        asked = {}
        for step in replays["evidence"][1].steps:
            for call, result in zip(step.message.tool_calls, step.results, strict=True):
                if (
                    result.ok
                    and call.name == "oscillator_error"
                    and call.arguments["candidate"] == piece
                ):
                    asked[call.arguments["steps_per_period"]] = result.content["error"]
        ax.loglog(list(asked), list(asked.values()), "o", color=colors[piece])
    ax.set_xlabel("time steps per period")
    ax.set_ylabel("error")
    ax.legend()
    return fig

src/pbc/mechanics/experiments/scientific_review.py, lines 97 to 123, read from the repository when this page was built.

Figure 1: simulated Error against the exact oscillator over two periods, against the number of time steps per period, for pieces 1 and 2. Lines are a fresh scan; dots are the values the evidence reviewer asked for, recomputed by this build.

Both lines fall with a slope near $-1$: both are first-order methods, which is why the order check passes both. The line of piece 2 sits higher, and the figure does not show the thing that is wrong with it: an energy that grows without bound. A convergence plot shows how fast the error falls at a fixed time, and the energy test shows what happens to a long run. The contract has both for that reason.

Run the reviewers yourself

This needs your own key and costs a few cents. The build does not run it, so the commands are marked.

Not verified. The build of this site did not run or check the material in this box.

# once: install the optional provider package
uv sync --extra openai

# in this shell only: the key stays out of code and files
export OPENAI_API_KEY="..."

uv run --extra openai python -m pbc.agents.review_lab

It runs the three reviewers one after the other and prints each message and tool call. A live run will differ in its words and perhaps in the order of its calls; the numbers of the checks it asks for are the numbers on this page, and the verdicts follow from them. The maintainer records the fixtures of this page with --record, reads the files, and commits them.

Exercises

Exercise 1. Run all five checks of the contract on both pieces yourself, without a reviewer. Which check fails, and which of the three reviewers could have found it with its own list of tools?

Solution to exercise 1

def every_check():
    """Exercise 3: every check of the claim on both pieces, without a
    reviewer."""
    for piece, build in CANDIDATES.items():
        for finding in verify(build, CLAIM):
            print(
                f"piece {piece}  {'pass' if finding.passed else 'FAIL'}"
                f"  {finding.check:18} {finding.detail}"
            )

src/pbc/mechanics/experiments/scientific_review.py, lines 126 to 134, read from the repository when this page was built.

piece 1  pass  clock              a step of 0.125 s advanced the time by 0.125 s
piece 1  pass  purity             the same step, repeated after other steps, differs by 0.0e+00
piece 1  pass  limits             uniform motion off by 8.9e-16 m, velocity under constant force by 7.1e-15 m/s, position by 9.8e-01 m (allowed 9.8e-01 m)
piece 1  pass  convergence-order  fitted order 1.04, claimed 1
piece 1  pass  bounded-energy     the energy band is 1.1e-01 in the first 5 periods and 1.1e-01 in the last 5
piece 2  pass  clock              a step of 0.125 s advanced the time by 0.125 s
piece 2  pass  purity             the same step, repeated after other steps, differs by 0.0e+00
piece 2  pass  limits             uniform motion off by 8.9e-16 m, velocity under constant force by 7.1e-15 m/s, position by 9.8e-01 m (allowed 9.8e-01 m)
piece 2  pass  convergence-order  fitted order 1.13, claimed 1
piece 2  FAIL  bounded-energy     the energy band is 4.1e+02 in the first 5 periods and 3.7e+52 in the last 5

Only bounded-energy fails, for piece 2. The physics reviewer has it in its list, and so has the evidence reviewer. The numerics reviewer does not, so its review, however careful, could not have failed the piece. Which reviewer is given which checks is part of the design of a review, and it should be written down before the reviewers run.

Exercise 2. Two reviewers recompute the same check on the same piece and report different verdicts. Build the verdicts by hand, resolve them, and say what the build reports and what it means.

Solution to exercise 2

def a_conflict():
    """Exercise 4: two reviewers that disagree on one check."""
    found = [
        Verdict("physics", 1, "limits", 7e-15, 1e-12, True),
        Verdict("evidence", 1, "limits", 3e-3, 1e-12, False),
    ]
    resolution = resolve(found, 1)
    print("rejected:", resolution.rejected, "| conflicts:", resolution.conflicts)

src/pbc/mechanics/experiments/scientific_review.py, lines 137 to 144, read from the repository when this page was built.

rejected: True | conflicts: ('limits',)

The failed check stands, so the piece is rejected, and the check is reported as a conflict. The checks are deterministic, so two honest reviewers get the same number from the same check on the same code. A conflict means they did not run the same check or the same code, and it is the reviewers’ set-up that has to be looked at before anything is concluded about the piece.

Replay this experiment

This page was built from commit 6417e2ff90258bd2d2ba21d217eb7666fd24f680 of the repository.

In a clone at that commit, with the project’s Python packages installed, this command runs the experiment of this page:

uv run --locked --no-dev python -m pbc.mechanics.experiments.scientific_review results

Files, tests, and every command

The files behind it:

• site/lessons/agents-llm/03-three-reviewers/index.qmd: this page, with every code cell.
• src/pbc/agents/review_lab.py: code the experiment runs.
• src/pbc/verification/checks.py: code the experiment runs.
• src/pbc/verification/faulty.py: code the experiment runs.
• src/pbc/mechanics/experiments/scientific_review.py: code the experiment runs.
• src/pbc/experiment.py: code the experiment runs.
• tests/unit/test_review_lab.py: tests of that code.
• tests/integration/test_review_replay.py: tests of that code.
• tests/unit/test_scientific_review_experiment.py: tests of that code.

With Git and uv installed (see the setup page), run:

git clone https://github.com/taspinar/physics-by-construction.git
cd physics-by-construction
git checkout 6417e2ff90258bd2d2ba21d217eb7666fd24f680
uv sync --locked --no-dev
uv run --locked --no-dev python -m pbc.mechanics.experiments.scientific_review results

The experiment writes every figure of this page into results/, as <label>.png, and every number and table of the page, as the page shows it, into results/results.json. Its numbers match this page to the digits shown, and its figure files are the ones the page shows; digits beyond those can differ between machines, because floating-point libraries differ. Change the code of the experiment and run it again to see what changes.

The experiment replays the recorded run of the agent: it runs every tool call of the recording again and needs no model and no key.

Not verified. The build of this site did not run or check the material in this box.

Optional: the live run. It asks a language model to run the experiment again, with your own account. It needs an OpenAI account and its API key in the variable OPENAI_API_KEY, and the optional dependency openai; a run costs a few cents. The key stays in your shell, never in a file.

# needs: an OpenAI account, its API key, and the extra openai
uv sync --locked --no-dev --extra openai
export OPENAI_API_KEY="..."   # this shell only
uv run --locked --no-dev --extra openai python -m pbc.agents.review_lab