Collaborator handbook

Start here: Tannistha and Narendra

This page is the entry point, and it is deliberately short. docs/STATUS.md is a running log of tens of thousands of words rather than an introduction; do not start there.

What is done

• 33 controller runs, scored and committed across four arms. Decisions, per-step diagnostic panels and — for the 13 runs that had a model in them — full transcripts are in results/. The other 20 correctly have no transcript: there is no language model in them, so there is nothing to transcribe, and that absence is itself the C2 point.
• The disclosed replication landed. llm-rep2-seed{1,4,9}, registered before launch, reported beside the originals and never pooled. Same depth on 2 of 3, same schedule on 1.
• The reproduction gate passed — GATE PASSED, 8 metrics at 0% relative difference, with the driver’s sha256 recorded in results/repro-gate-verdict.json.
• The contamination probe cleared — no evidence of recitation over 14 scored probes — and survived three rounds of audit of its own scoring, every one of which had leaned toward the verdict the study wants.
• The realisation-dropout audit is committed: 22 of 2,700 realisation-runs across 27 runs, 0.815%.
• The repository installs from a clone on Python 3.12 and the whole analysis regenerates from committed data with one command. On 3.13 or 3.14 the install stops at resdata==6.2.9.
• The bibliography is verified against registry records, field by field.

What is open

• 4 author notes in the paper, at paper/main.tex lines 49, 355, 917 and 934. The gate is make -C paper check — the Makefile is paper/Makefile and there is none at the repository root — and it exits 1 until all 4 are gone.
• The open decisions in docs/WORKPLAN.md. D1 is decided and D2 was resolved by the data. D8 is complete. Still open: D3 (which comparison is Section 5.2’s primary result, with a recommendation on record and no decision), D4 (Norne in or out), D5 (PetroGraph baseline), D6 (author order and affiliations), D7 (buy the EAGE pitfalls paper), D9 (the localisation arm), D10 (the escalation suite), D11 (the failing Vercel project dkailab/agentmatch: its premise was that this repository had nothing for Vercel to build, which this branch changes — the decision is now whether to keep the project and let it serve this site, not whether to disconnect it).
• The three reviews have not happened. docs/reviews/ does not exist yet. Nothing here has been reviewed end to end by a human.
• The container rebuild. agentmatch:opm and agentmatch:ert must be rebuilt from env/ with the patched resdata pin before anything is published as runnable. The image on the box predates the bump.
• Two full texts are unobtained: the EAGE pitfalls paper, 10.3997/2214-4609.202539039, and SPE-229413-MS, which is cited from its deposited abstract and says so.
• Stale documents that contradict this site. README.md still says the project is pre-experimental and that no results are claimed, which stopped being true 33 scored runs ago. It must not be linked as the project’s framing until it is rewritten.

Tannistha: the science, and the EOR half

First, regenerate everything yourself and read the paper against your own output

Do not read the paper against the numbers in any document, this one included.

git clone https://github.com/deepkapha/agentmatch && cd agentmatch
python3.12 -m venv .venv && . .venv/bin/activate    # 3.12 only, see below
pip install -r requirements.txt
python3 tests/test_crps.py && python3 tests/test_regressions.py
bash scripts/finalise_analysis.sh

The pins in requirements.txt are the ones that produced every result in the paper, and they resolve only on Python 3.12 — on 3.13 or 3.14 pip stops at resdata==6.2.9 with “no matching distribution”, because the wheels for that version end at 3.12. Use 3.12 rather than relaxing the pin: relaxing it changes the environment the numbers came from.

The last command refuses to certify an incomplete sweep unless you pass --allow-partial, which stamps the figures provisional. Then open paper/figures/f2-stats.tex, paper/figures/f4-stats.tex, paper/figures/f5-stats.tex, paper/figures/f6-stats.tex and paper/figures/f7-stats.tex. Those macro files are the only place a number enters the paper — and, by way of public/data/stats.json, the only place one enters this site. If the prose and the figure ever disagree, one of them was hand-typed and it is a defect. The standard you are holding the paper to is docs/preregistration/2026-08-27-controller-adjudication-rule.md: anything the paper claims that the registration does not license gets flagged.

Second, make the EOR scope call

Open paper/main.tex lines 347–356. The list of quantities that pre-EOR history never constrains — residual oil saturation, the imbibition branch of relative permeability, the miscibility transition and minimum miscibility pressure, the residual resistance factor, adsorption and the Todd-Longstaff mixing parameter — is described in its own author note at lines 355–356 as the scientific core of the EOR half of a paper that currently has no EOR half. There are three options and you own the science of the choice: a forward pointer, a two-sentence limit, or a cut. Tarry owns the scope.

Third, write the scientific review and pre-answer the two referee questions

Create docs/reviews/ and put a written verdict in it: does each of C1, C2 and C3 follow from the artifacts in results/? Two questions are already flagged in the paper and both are yours. The first, at paper/main.tex lines 863–865, is whether a small trained classifier on the same diagnostic panel would match the language model. We did not run that arm and cannot answer it, so the honest response needs writing rather than measuring. The second is the physical plausibility of the posteriors; public/data/ carries the mean and spread fields for the three seed-1 posteriors and results/controllers/ carries the scored panels. One further item is substantive and belongs to you: pre-registration rule 6 requires every metric in both observation families to be reported, and the paper currently reports one.

Narendra: reproducibility and the open-source release

First, do the review as a stranger would

A fresh machine, README.md, and nothing else. Every step that makes you ask us a question is a defect, and it is logged rather than quietly fixed.

git clone https://github.com/deepkapha/agentmatch && cd agentmatch
python3.12 -m venv .venv && . .venv/bin/activate    # 3.12 only, see below
pip install -r requirements.txt
python3 tests/test_crps.py && python3 tests/test_regressions.py
bash scripts/finalise_analysis.sh
bash scripts/fetch_data.sh
python3 scripts/verify_data.py

The pins in requirements.txt are the ones that produced every result in the paper, and they resolve only on Python 3.12 — on 3.13 or 3.14 pip stops at resdata==6.2.9 with “no matching distribution”, because the wheels for that version end at 3.12. Use 3.12 rather than relaxing the pin: relaxing it changes the environment the numbers came from.

verify_data.py is the authority on whether the fetch worked, not the fetch script’s own exit code. Note before you start that README.md is stale: it still says the project is pre-experimental and that no results are claimed. Rewriting it is part of this job.

Second, rebuild the container and close the release blocker

docker build -f env/Dockerfile     -t agentmatch:opm .
docker build -f env/Dockerfile.ert -t agentmatch:ert .

The base image must be built first. The image on the compute box still carries resdata 4.2.4, subject to GHSA-pr85-w493-9w3x; requirements.txt and env/Dockerfile pin 6.2.9, and the bump was checked rather than trusted, re-scoring every committed run whose posterior is still on disk to numbers identical at 1e-12. Nothing is published as runnable until the image is rebuilt from the patched pin.

Third, assemble the release

It ships as one package with the paper, because the pieces cite each other. It needs: a GitHub release tagged to the commit the arXiv source was built from; a CITATION.cff, which does not exist yet and waits on author order (D6); a Zenodo DOI cited in the paper’s Reproducibility section; and ert/LICENSE-DATA.md with the mandatory UNISIM-I-H citation carried where anyone taking the files will see it — as it is in the footer of every page here, and one place it is still wrong: scripts/fetch_data.sh prints the obligation as “Avansi & Schiozer”, the incomplete two-author form that ert/LICENSE-DATA.md exists to correct. It is the first licence notice anyone fetching the data reads. Fix it to the three-author citation before the release, and grep the tree for the short form. Two smaller items are yours and are quick: LICENSE line 189 still carries the unfilled Apache placeholder, so no copyright holder is asserted anywhere, and the internal compute-box hostname and root path are exposed in tracked docs.

Four traps already found here, so nobody finds them twice

1. Verify that a thing works, not that it exists. A waiter process was checked with pgrep and reported alive for four days while its guard errored on every poll, so the arm it existed to launch never started. Alive was true. Alive was not the property that mattered.
2. Never state a measurement you are not re-measuring. A number in a comment is a claim that rots. Assert it in a test so it is re-measured, or state the property and delete the figure.
3. A filter that drops data must be justified out loud. The memorisation probe’s first extractor dropped answers of n <= 0 as “no number given”. Zero is an answer — it is what a well produces before breakthrough — and the filter discarded the model’s worst predictions and flattered it.
4. The memorisation probe’s scorer is a suspect instrument. Three separate defects have been found in it and every one leaned toward the verdict the study wants. Treat any future change to it as suspect until you have checked which way it moves the verdict.

One number to hold on to while you read: 13 of 20 of the llm and heuristic runs finished under Σ 1/α = 1, and the depth mismatch that makes the primary comparison unadjudicable falls on seeds 1, 4 and 9. Both were recomputed from public/data/runs.json when this page was served.