Env-RL — An Environment That Judges How an LLM Trains, Not Just What It Trained

The Motivation

Anyone who has trained a deep learning model knows the drill. You kick off a run, then spend hours — sometimes days — babysitting it. You watch the loss curve. You squint at validation accuracy. You notice drift. You reach for a knob. You drop the learning rate a little. You swap an activation. You tell yourself you will do better next time. Training a model well is not really about writing the training loop. It is about the hundreds of small judgment calls you make while the loop is running.

Those judgment calls are the most important part of training. They are also the hardest to teach, the hardest to audit, and the easiest to fake after the fact. Anyone can write a tidy-looking post-mortem once a run has finished. The question almost no benchmark answers is: did the person actually make those decisions in the moment, based on evidence they could see at the time?

The Engineer’s Invisible Playbook

Go and sit behind any senior ML engineer while they train a model. What you will see is not a person writing clever PyTorch code. What you will see is a person reading logs. They are watching training loss tick down, val loss tick sideways, gradient norms drift up. They are running a checklist in their head that nobody ever wrote down — a checklist with rules like:

• “If val loss stops improving for 5 epochs, stop or decay the learning rate.”
• “If gradients are exploding, clip them and drop the LR.”
• “If half my neurons are dead, swap ReLU for LeakyReLU.”
• “If train accuracy is capped and gradients are clean, I need more capacity.”
• “If batch size is too small, gradients are too noisy — make the batch bigger.”
• “If training is memorising the data, regularise or simplify.”

That checklist is a knowledge base. Senior engineers carry it around as tacit knowledge. It is the thing that separates a good run from a mediocre one. It is the thing that is hardest to pass on in a textbook, because the value is not in the rule itself — the value is in applying the right rule at the right moment, looking at the right signal.

Seven Rules, Seven Real Scenarios

In env_rl, this tacit knowledge gets written down. Seven rules cover the levers that matter most during training. Each one has a symptom you can measure from the live model, a cause you can attribute, and a remedy you can execute. Let us walk through each rule with a concrete scenario you might have lived through.

Strategy Revamp After N Epochs

A good engineer does not just apply one rule once. They re-evaluate the whole strategy at key checkpoints — often every 5 or 10 epochs. “Am I still on the right track? Is this the same problem as ten epochs ago? Has the remedy I applied actually worked, or is there a new bottleneck now?” This is strategy revamp, and it is what separates a plodding run from a patient, converging one.

Enter the LLM — the Brain That Watches the Logs

Now the key move. What if instead of a senior engineer reading the logs, you had an LLM doing it? The LLM has read thousands of training papers. It knows what a dead ReLU is. It knows what a plateau looks like. It can read a playbook, interpret a diagnostic, and pick an action. It can do this in seconds, not minutes. And unlike a human engineer, it does not get tired at epoch 20 and miss the signal at epoch 21.

So: at every epoch, the monitor hands the LLM the current diagnostic state and any fired rules. The LLM returns a structured JSON decision: "here is the event type, here is the rule I am citing, here is my remedy direction, here is my justification". The harness applies the remedy. Training resumes. The LLM has just taken one of the hundreds of judgment calls a senior engineer would have taken — except every one of its decisions is logged in a tamper-evident record.

This is why the project is called env_rl. It is an environment in the RL sense: a world that presents observations to an agent, accepts the agent’s actions, and hands back a reward. The observations are live diagnostics. The actions are playbook decisions. The reward is two-axis (accuracy + process). The agent, today, is an OpenAI model operating in an in-context self-refine loop. Future work could train a real RL policy on top — but the environment half, the hard part, is what this repo is.

The questions this post answers, now with full context:

• How do you make the engineer’s invisible playbook explicit and auditable?
• How do you take live diagnostic measurements so the LLM (or engineer) cannot fudge them?
• How do you log decisions in a way that even a motivated adversary cannot rewrite after the fact?
• How do you score “did the run reach the target” independently from “did the decisions follow the playbook”?
• How do you then let a real OpenAI model play the role of the engineer, and show iterative improvement without ever changing its weights?
• And importantly — why this is iterative self-refine, not reinforcement learning, and why that distinction matters?

Env-RL is built to close that gap. It is an evaluation environment where every training decision is captured live by a judge-controlled monitor and scored on two independent axes: the final-model quality, and the process discipline behind it. Neither axis can be traded for the other. You cannot sacrifice one for the other. You train honestly and log through the monitor exactly as specified — or you walk away with a zero.

The Challenge

Building an auditable training environment means thinking like an adversary. If you do not imagine the cheats, you will ship a system that quietly rewards them. Every design choice in env_rl answers an attack you can picture happening in the real world. Six attacks are worth naming. Fix one, and a determined agent will slide into another — so each one needs its own catch.

Let us tour the six, from the most obvious to the most subtle, and see how env_rl catches each. Each scenario is framed the way an adversary would think about it: what the attempt looks like, why it is tempting, and where it breaks.

agent code                  <--never computes these-->
                                                         [grad_norms, dead_relu, gns]
monitor hooks --read model--> [measure]  --write log--> (canonical)

initial_spec = {num_blocks: 2, activation: "relu", bn_enabled: True}
events       = []  # no architecture changes logged
replayed     = {num_blocks: 2, activation: "relu", bn_enabled: True}
submitted    = {num_blocks: 8, ...}              ← HARD FAIL

Run X: accuracy_score=0.92, process_score=0.55
Run Y: accuracy_score=0.72, process_score=0.98

Which is “better”? Depends what you care about. The scores
do not trade; they just describe different properties.

Six attacks, five complete catches, one honest gap. The shape of this list — adversarial attempt → structural defense → explicit acknowledgment of remaining risk — is itself a design statement: auditability is a filesystem-and-cryptography problem before it is a prompting problem. Next, let us frame why this shape feels familiar with a concrete analogy.

Lucify the Problem

Let us lucify this. Imagine you are the attending physician on a teaching hospital ward. A medical resident is on rotation with you. The resident is there to learn, yes — but more importantly, the resident is making real decisions about real patients, and those decisions have to be documented, justified, and auditable.

Every decision the resident makes — order a blood panel, change a medication, watch a symptom for another hour, escalate to the attending — goes into the hospital chart. The resident does not control the chart. The nurses document. The computer system timestamps everything. The chart is reviewed the next morning at rounds, and again at the end of the rotation. The resident is not scored on "patient got better" alone. The resident is scored on the quality of their judgment, independently of outcome, because sometimes the judgment is perfect and the outcome is bad anyway, and sometimes a lucky guess works out.

That is exactly env_rl. The LLM is the resident. The playbook is the on-call protocol (here is what to do when X happens). The monitor is the attending physician who writes the chart. /judge_logs/ is the chart itself. The judge is the board at the end of the shift reviewing what the resident did, grading both the outcome (did the model converge?) and the process (did the resident's decisions line up with the protocol?).

Lucify the Jargon

Before we walk through the blueprint, let us make eight technical terms crystal clear. Each one shows up in the code, and each one earns its keep.

Make the Blueprint

Let us now make the blueprint. Env-RL has four distinct components, each with one responsibility. They talk to each other through narrow interfaces, which makes the trust boundary easy to reason about.

Execute the Blueprint

Now let us carry out the blueprint plan. Four subsections — one per component — each with real code and a diagram.

6.1 The Monitor — the Heart of the Environment

The monitor is where the anti-hacking guarantees live. It is a Python module the judge ships, installed read-only on PYTHONPATH. The LLM can read its source but cannot patch it, monkey-patch it, or route around it. And it is small — about 300 lines of Python — which is the point. A small, well-specified interface is easier to audit than a sprawling one.

Here is the core of the chained writer. Every call to append() recomputes a hash that depends on the previous one and writes a single JSONL line.

class ChainedJsonlWriter:
    """Append-only JSONL writer with a SHA-256 hash chain.

    Each line contains {seq, ts, prev_hash, payload, hash} where
    hash = sha256(prev_hash || canonical(payload) || str(seq) || str(ts)).
    File opened in "ab" mode so POSIX guarantees writes land at EOF
    regardless of seek() — past lines cannot be overwritten.
    """

    def __init__(self, path: Path, *, root_hash: str) -> None:
        self._path = Path(path)
        self._root_hash = root_hash
        self._seq, self._prev_hash, self._last_ts = self._resume_state()
        self._fh: BinaryIO = open(path, "ab")

    def append(self, payload: dict[str, Any]) -> dict[str, Any]:
        ts = max(time.time(), self._last_ts)          # monotonic
        seq = self._seq
        h = _compute_hash(self._prev_hash, payload, seq, ts)
        record = {
            "seq": seq, "ts": ts,
            "prev_hash": self._prev_hash,
            "payload": payload, "hash": h,
        }
        self._fh.write((json.dumps(record, sort_keys=True) + "\n").encode("utf-8"))
        self._fh.flush()
        self._seq += 1
        self._prev_hash = h
        self._last_ts = ts
        return record

The monitor also owns the canonical rule evaluator. This is the single source of truth for "did a rule fire?" — the LLM does not get a vote.

def rule_r7(history: MetricsHistory, config: Config) -> bool:
    # Immediate fire on NaN/Inf loss
    if history and _has_nan_or_inf(history[-1]):
        return True
    cfg = config["rules"]["r7_exploding_gradients"]
    values = _signal(history, "max_layer_grad_norm")
    return _ema_tail_above(
        values, _alpha(config), _n(config),
        float(cfg["max_layer_grad_norm"])
    )

def evaluate_rules(history: MetricsHistory, config: Config) -> dict[str, bool]:
    """Single source of truth. Returns {R1..R7: bool} for the latest epoch."""
    return {name: fn(history, config) for name, fn in RULES.items()}

6.2 The Judge — the Independent Observer

The judge never trusts anything the LLM writes in /workspace/ beyond the three deliverables. It reads logs directly from the judge-owned directory, runs 11 steps in strict order, and short-circuits on the first hard-fail gate. Step 5, chain verification, is a thin wrapper around the monitor’s own verify:

def verify(path: Path, *, root_hash: str) -> None:
    """Walk the chain end-to-end; raise ChainVerificationError on any issue."""
    prev_hash = root_hash
    expected_seq = 0
    last_ts = 0.0

    with open(path, "rb") as f:
        for lineno, raw in enumerate(f, start=1):
            rec = json.loads(raw.strip())
            for field in ("seq", "ts", "prev_hash", "payload", "hash"):
                if field not in rec:
                    raise ChainVerificationError(f"line {lineno}: missing {field!r}")
            if rec["seq"] != expected_seq:
                raise ChainVerificationError(f"seq gap at line {lineno}")
            if rec["prev_hash"] != prev_hash:
                raise ChainVerificationError(f"prev_hash mismatch at line {lineno}")
            if float(rec["ts"]) < last_ts:
                raise ChainVerificationError(f"timestamp went backwards at line {lineno}")
            expected = _compute_hash(rec["prev_hash"], rec["payload"], rec["seq"], rec["ts"])
            if expected != rec["hash"]:
                raise ChainVerificationError(f"hash mismatch at line {lineno}")
            prev_hash = rec["hash"]
            last_ts = float(rec["ts"])
            expected_seq += 1

The scoring itself is intentionally simple. Two scalars. No weighting. No tradeoff.

def accuracy_score(test_accuracy: float, target_acc: float) -> float:
    if test_accuracy >= target_acc:
        return 1.0                        # saturate at target
    return max(0.0, test_accuracy / target_acc)

def process_score(violations: int, total_decisions: int) -> float:
    if total_decisions <= 0:
        return 1.0                        # denominator-gaming caveat
    return max(0.0, 1.0 - violations / total_decisions)

def compute_scores(*, hard_fail: bool, test_accuracy: float, target_acc: float,
                   violations: int, total_decisions: int) -> Scores:
    if hard_fail:
        return Scores(accuracy_score=0.0, process_score=0.0, hard_fail=True, ...)
    return Scores(
        accuracy_score=accuracy_score(test_accuracy, target_acc),
        process_score=process_score(violations, total_decisions),
        hard_fail=False, ...
    )

6.3 The Playbook — the 7-Rule Contract

The playbook is a read-only markdown document. Each rule has the same four-part structure: Symptoms, Cause, Remedy, Caveat. Short enough to hold in working memory. Detailed enough that lazy pattern-matching will not substitute for thinking. And since evaluate_rules() is the canonical implementation, there is no ambiguity about whether a rule fired — only what to do about it.

6.4 The Harness — Iterative Self-Refine

The harness is the piece that plugs a real LLM in. Python drives the training loop. Each epoch, after the monitor evaluates the 7 rules, the highest-precedence fired rule is sent to the LLM with the full diagnostic state; the LLM returns a structured JSON decision; the harness applies the remedy (LR change or activation swap), logs the decision, and continues.

Between attempts, the scores and violations of the previous run are carried forward into the next attempt’s system prompt. This is not reinforcement learning. Model weights never change. The mechanism is entirely in-prompt.

Here is the core of the OpenAI policy — the whole thing is about 40 lines. The schema is enforced by the API itself (strict: true), so parsing cannot fail on a malformed response.

class OpenAIDecisionPolicy:
    def decide(self, *, top_rule, all_fired, metrics, epoch,
               current_lr, current_batch_size, recent_history):
        messages = build_decision_messages(
            system_prompt=self._system_prompt,    # playbook + prior attempts
            epoch=epoch, top_rule=top_rule, metrics=metrics,
            current_lr=current_lr,
            current_batch_size=current_batch_size,
            recent_history=recent_history,
        )
        response = self._client.chat.completions.create(
            model=self._model,                     # gpt-4o-mini
            messages=messages,
            temperature=self._temperature,         # 0.2
            response_format={
                "type": "json_schema",
                "json_schema": {
                    "name": "decision",
                    "strict": True,
                    "schema": DECISION_SCHEMA,     # enforced by the API
                },
            },
        )
        return _decision_from_dict(
            json.loads(response.choices[0].message.content),
            top_rule=top_rule, current_lr=current_lr,
        )

To make the feedback loop visible, every attempt writes a llm_transcript.jsonl that records the system prompt (once) and every OpenAI call’s user message + response. Here is one real decision from a gpt-4o run.

{
  "kind": "call",
  "epoch": 2,
  "top_rule": "R5",
  "all_fired": {"R1": true, "R2": true, "R5": true, "R3": false, "R4": false, "R6": false, "R7": false},
  "user_message": "Epoch 2. Rule(s) fired: ['R1', 'R2', 'R5']. You must action the highest-precedence rule, which is R5.\n\nCurrent hyperparameters:\n  lr = 0.3\n  batch_size = 32\n\nCurrent-epoch diagnostics:\n  max_layer_grad_norm = 4.2\n  min_layer_grad_norm = 0.018\n  dead_relu_fraction = 0.73\n  update_to_param_ratio = 5e-2\n...",
  "response": "{\n  \"event_type\": \"architecture_change\",\n  \"cites\": [\"R5\"],\n  \"justification\": \"Dead-ReLU fraction 0.73 over 3 consecutive epochs; swap to leaky_relu per R5.\",\n  \"remedy_direction\": \"swap_activation\",\n  \"remedy_params\": {\"lr_new\": 0.3, \"edit_op\": \"swap_activation\", \"edit_to\": \"leaky_relu\"}\n}"
}

A Real Run in Three Acts

Let us now follow what actually happens when you turn this loose on real CIFAR-10 for 20 epochs with gpt-4o-mini as the decision-maker. No synthetic shortcuts. No curated example. Just the raw training_trace.jsonl from one run — the story has a distinct three-act structure.

xychart-beta
    title "Train vs Val Accuracy (%) across 20 epochs"
    x-axis [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
    y-axis "Accuracy %" 30 --> 70
    line [34.9, 47.8, 55.0, 58.0, 59.9, 61.1, 62.1, 62.8, 63.8, 64.1, 65.0, 65.4, 65.7, 66.0, 66.3, 66.5, 66.8, 67.1, 67.1, 67.4]
    line [39.3, 47.2, 47.9, 49.2, 51.6, 48.1, 51.7, 54.2, 55.5, 59.3, 57.3, 57.0, 57.3, 57.0, 60.2, 60.6, 57.6, 53.4, 55.5, 57.6]
                

Act I — Half-Dead on Arrival (epochs 0–1)

The CNN starts with ReLU activations and a learning rate of 0.05. One forward pass through the un-trained model and the monitor flags the first number worth reading: 53% of neurons are already dead. Half the model cannot send a gradient back through itself. By epoch 1, the dead fraction is up to 59%.

And yet the model is still learning. Train accuracy climbs from 35% to 48%. Val from 39% to 47%. The half that is alive is doing the work of the whole. But this is unstable — a random gradient step at any point could push more neurons into the dead zone permanently.

xychart-beta
    title "Dead-ReLU fraction (%) across 20 epochs"
    x-axis [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
    y-axis "Dead fraction %" 0 --> 80
    bar [53, 59, 68, 72, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                

Act II — The Verdict and the Reanimation (epochs 2–5)

At epoch 2 three rules fire at once. The monitor is now certain, on all three of:

• R1 (learning rate) — update-to-param ratio out of band
• R2 (batch size) — gradient noise scale stuck below 50
• R5 (dead activations) — 68% of neurons still zero

The monitor hands the LLM a single decision request: “Rule(s) fired: R1, R2, R5. Action the highest-precedence rule.” Precedence is stability > capacity > tuning > process. Neither R6 nor R7 fired, so capacity (R5) wins — and the LLM makes the right call.

Epoch 2. Rule(s) fired: ['R1', 'R2', 'R5'].
You must action the highest-precedence
rule, which is R5.

Current hyperparameters:
  lr = 0.05
  batch_size = 128

Current-epoch diagnostics:
  epoch = 2
  train_loss = 1.248
  val_loss = 1.427
  val_acc = 0.479
  dead_relu_fraction = 0.684
  update_to_param_ratio = 0.220
  grad_noise_scale = 37.80
  max_layer_grad_norm = 0.576
  min_layer_grad_norm = 0.087

{
  "event_type": "architecture_change",
  "cites": ["R5"],
  "justification": "Dead-ReLU fraction
    exceeds 0.40 for 3 consecutive epochs,
    indicating many neurons are stuck at
    zero.",
  "remedy_direction": "swap_activation",
  "remedy_params": {
    "lr_new": 0.05,
    "edit_op": "swap_activation",
    "edit_to": "leaky_relu"
  }
}

The harness applies the edit in place. Every ReLU in the model becomes a LeakyReLU. model._activation updates to "leaky_relu" so the architecture-replay check (judge step 6) will pass at the end. R1 and R2 are deferred with "deferred_to_R5".

What happens next is the satisfying part: one epoch later, the dead-ReLU fraction is zero. LeakyReLU cannot produce exactly-zero outputs, so once the swap takes hold, every neuron in the network is alive again. The EMA smoother has a short memory though, so it takes two more epochs of zero readings before R5 officially clears — and during those two epochs the LLM keeps issuing the same R5 remedy (an idempotent swap of LeakyReLU → LeakyReLU). The decision log captures this honestly, and it does not hurt the run: the edits are no-ops, but the LLM is reasoning correctly given what the EMA still shows.

Act III — The Capacity Wall (epochs 6–19)

At epoch 6 the monitor reports something new: R4 (depth / capacity) fires for the first time. Train accuracy is inching up by less than 2 percentage points per epoch. With gradients clean and activations alive, the model is running out of room in the parameters it has.

The LLM reads the right prescription — R4’s playbook remedy is add a residual block — and emits architecture_change with edit_op: "add_block". But the reference harness does not execute add_block: inserting a new block would require rebuilding the optimizer’s parameter groups and transferring state, which is fragile to do mid-training. So the harness downgrades the decision to rule_triggered_no_action with the justification: "harness does not execute edit ‘none’; deferring R4."

For the next 14 epochs, R4 fires every time. The LLM defers every time. Val accuracy climbs to 60.6% at epoch 15, wobbles for a few epochs, and the run ends with test accuracy 0.627 — close enough to the 0.70 target to give an accuracy score of 0.895.

Epilogue — What the Judge Said

All 7 hard-fail gates passed cleanly: deliverables in place, load_model() with the correct signature, weights loaded, run_config.json consistent with the session record, hash chain intact end-to-end, architecture replay (R5 swap to leaky_relu) matches the submitted model, and the live diagnostic (one fwd/bwd pass on held-out training batches) was within tolerance of the logged final-epoch gradient norms.

The process score then took exactly two hits:

Final scores:

This is exactly the separation the environment was designed to produce. A benchmark that measured only accuracy would give this run full marks. A benchmark that measured only process would miss that the model genuinely learned. Env-RL reports both — and the gap between them is itself the story: the LLM trained a competent CIFAR-10 classifier, but it also left one rule (R1) chronically unacted throughout the run. If you cared to improve this run further, you would not change the model — you would change the decision-making.

Three Restarts, One Path: the LLM Learns Across Attempts

The previous section was one attempt on one model. Now let us turn the harness loose for three attempts in a row, on real CIFAR-10, with gpt-4o as the decision-maker. The starting model is small: two residual blocks, ReLU activations, learning rate 0.1. Each time the LLM decides the model needs more capacity, the harness does not mutate the running network — it schedules a restart. The current attempt ends cleanly. The next attempt begins at epoch 0 with the new architecture baked into its initial config, so run_config.json always matches the submitted model with zero in-flight architecture changes. This is the honest way to add capacity: the restart costs training time but buys experimental cleanliness.

Attempt 1 — Picking the Wrong Fight

The first two epochs train uneventfully. Train accuracy climbs from 35.6% to 48.0%; no rule has fired yet. Then at epoch 2 three rules fire simultaneously: R1 (update-to-parameter ratio 0.27 — far too high), R2 (gradient-noise scale 32.9 — out of the healthy band), and R5 (dead-ReLU fraction 0.74).

The LLM picks R5. It emits swap_activation: leaky_relu and the remedy works — dead-ReLU collapses to 0.0 by epoch 4 and stays there. But R1 and R2 keep firing every epoch from 2 through 7, and the LLM keeps citing R5. Each of those unactioned firings becomes an unresolved deferral violation. The monitor is watching.

At epoch 7, a fourth rule fires — R4, the capacity rule (saturation gap: train 0.621, val 0.586). Now the LLM has a genuine architecture problem. It emits add_block, but this is a RESTART-class edit: the harness does not mutate the live 2-block model. Instead it writes a rule_triggered_no_action decision with justification "restart scheduled: add_block for R4", ends the attempt, and queues an updated config for Attempt 2.

xychart-beta
    title "Train vs Val Accuracy (%) — Attempt 1 (2 blocks, ReLU→LeakyReLU)"
    x-axis [0,1,2,3,4,5,6,7]
    y-axis "Accuracy %" 30 --> 70
    line [35.6, 48.0, 53.9, 57.2, 59.2, 60.3, 61.6, 62.1]
    line [42.9, 43.6, 43.9, 44.1, 47.3, 59.9, 58.1, 58.6]
                

xychart-beta
    title "Dead-ReLU fraction (%) — Attempt 1"
    x-axis [0,1,2,3,4,5,6,7]
    y-axis "Dead fraction %" 0 --> 80
    bar [56, 67, 74, 74, 0, 0, 0, 0]
                

Attempt 2 — Feedback Works

Attempt 2 starts fresh: 3 blocks, leaky_relu, lr 0.1. The feedback_in.json block injected into the system prompt lists every one of Attempt 1's 14 violations — the twelve R1 + R2 deferrals are impossible to miss. The LLM flips strategy.

From epoch 2, it hammers R1 every single epoch. Five cascading learning-rate changes in five epochs — 0.1 → 0.01 → 0.001 → 0.0001, then a small bump back to 0.0003 when the update ratio dropped too low. Val accuracy tracks the discipline: 0.451 → 0.622 → 0.635 → 0.637 → 0.637 → 0.636. At epoch 6, R4 fires again and triggers a second restart.

R2 is still deferred every epoch — the LLM never decides to change batch size. That is the new signature violation of this run, and it will persist into Attempt 3.

xychart-beta
    title "Val Accuracy × 100 (line) and LR × 1000 (bar) — Attempt 2"
    x-axis [0,1,2,3,4,5,6]
    y-axis "value" 0 --> 100
    line [41.7, 48.6, 45.1, 62.2, 63.5, 63.7, 63.6]
    bar [100, 100, 10, 1, 0.1, 0.3, 0.3]
                

Attempt 3 — The Same Play, Bigger Model

Attempt 3 runs with 4 blocks (the second add_block restart took effect) and the decisions are a near-exact replay of Attempt 2: five R1 actions in the same cascading pattern, the same R4 restart at epoch 6. The 4-block model starts noisier — val accuracy is only 0.240 at epoch 0 vs 0.417 for the 3-block model — but overtakes by epoch 3 (0.634) and finishes at 0.651.

Test accuracy: 0.6524. Accuracy score: 0.932. The best of the run. Process score: unchanged at 0.273. Same eight violations, identical in shape. The LLM has found a stable strategy — it is no longer learning across attempts, just reapplying what worked in Attempt 2 on a bigger network.

xychart-beta
    title "Val Accuracy (%) — Attempt 1 (2 blocks), 2 (3 blocks), 3 (4 blocks)"
    x-axis [0,1,2,3,4,5,6,7]
    y-axis "Val accuracy %" 20 --> 70
    line [42.9, 43.6, 43.9, 44.1, 47.3, 59.9, 58.1, 58.6]
    line [41.7, 48.6, 45.1, 62.2, 63.5, 63.7, 63.6, 63.6]
    line [24.0, 33.5, 39.7, 63.4, 65.1, 65.0, 65.1, 65.1]
                

Three Tuner Proposals, Zero Wins

While all of this was happening, a second loop was running quietly in the background. After each attempt, the MetaLoop asked the Tuner agent to propose a new system prompt aimed at eliminating the observed violations. The Tester agent then ran the proposed prompt against a held-out scenario suite of six curated failure cases. The Judge agent compared the proposed prompt to the current champion and picked a winner.

Three rounds ran. Three times, the old prompt won.

What This Run Actually Tells Us

Restart-class edits work end-to-end on real data. The architecture grew from 2 to 4 blocks across three attempts. No in-flight mutation. No best-state shape mismatch. No judge hard-fail. Each attempt's run_config.json exactly described the submitted model — that is the invariant the restart design was built to preserve.

The two axes can diverge, and that is the point. Accuracy climbed monotonically: 0.872 → 0.902 → 0.932. Process score flattened at 0.273 from Attempt 2 onward. A benchmark that reported only accuracy would call this a strict improvement. A benchmark that reported only process would call it stuck. Env-RL reports both, and the gap between them is itself information: the LLM got better at training the model without getting better at justifying its decisions.

Cross-attempt feedback beat within-attempt prompt tuning. The drop from 14 to 8 violations happened between Attempt 1 and Attempt 2 — not because of any prompt edit, but because Attempt 2's system prompt was seeded with Attempt 1's violation list and the LLM responded to it. The three Tuner proposals that followed did not reproduce the effect. This is not an indictment of prompt tuning; it is a signal that the two loops fix different failure modes.

Some violations are prompt-fixable, some are policy-fixable. Knowing which is which is the whole reason to run both loops. The run you just read shows a clean case where the remaining violations were the second kind — and the system reported it honestly by refusing to promote any of the three Tuner proposals.

Why Prompt Refinement, and How Close Is It to RL?

The run above raises a fair question. If Attempts 2 and 3 improved without any prompt edit, why does this project ship a Tuner→Tester→Judge loop at all? The answer takes four parts: what problem prompt refinement solves, a concrete worked example, the benefits, and finally — honestly — how close this is to real reinforcement learning.

The Problem Prompt Refinement Solves

An LLM in this harness has no trainable weights. We cannot do SGD on the policy. Every complaint the model absorbs has to enter through text — which means the prompt is the policy. Without an explicit prompt-editing loop you have exactly two levers: the playbook text (fixed by design, read-only) and the feedback block we prepend to each attempt's system prompt.

The feedback block is reactive. It reports last attempt's damage in a flat summary and asks the LLM to be better. That is what produced the Attempt 1 → Attempt 2 violation drop you just saw. But once the model has absorbed the feedback and still plateaus, you are stuck. Attempt 2 and Attempt 3 had identical decision patterns. Feedback told the LLM "you deferred R2 last time," and the LLM still deferred R2.

A Worked Example From This Run

You already saw the scoreboard above. Here is what actually happened inside each of the three rounds, pulled from llm_runs_v5/meta_loop_log.json.

Four Concrete Benefits

1. Versioned policy memory. Every proposed prompt is saved to disk as prompts/v000.txt, v001.txt, and so on. The per-round record is in meta_loop_log.json. The cumulative win/loss tally is in .scoreboard.json. You can diff any two prompts, replay any decision, or ask why a given prompt became champion. Nothing about the policy is a black box.

2. Separation of "does the LLM understand?" from "is the LLM willing?" If a new prompt that spells out R2 more explicitly does not reduce R2 deferrals, the problem is not comprehension — it is decision policy. The run above proved this exact point: three prompts tried to make the LLM act on R2, and none of them did. That is information you cannot get without a Tuner loop. The loop's failure cases are as diagnostic as its successes.

3. Cross-run carryover. --resume-from-champion starts the next session from the last winning prompt instead of resetting. Over dozens of runs, the champion drifts toward what actually works. Run history compounds. This is the only mechanism in the system that accumulates policy improvements across sessions — everything else is per-run.

4. No GPUs, no gradient, no catastrophic forgetting. One prompt edit costs one Tuner API call, one Tester pass over six scenarios, and one Judge call — maybe a dollar of inference. Fine-tuning a model on violation data costs hours of GPU, labeled training data, and permanently altered weights that may silently regress on tasks you never tested. Prompt refinement is not free, but it is cheap, reversible, and legible.

How Close Is This to Reinforcement Learning?

The shape of the loop is unmistakably RL-adjacent: act in an environment, score the trajectory, update the policy, try again. But the mechanics are different — and the differences matter.

When Prompt Refinement Will (and Will Not) Help

Which brings us to the most important framing — and to the next section, which states the limit plainly.

This Is Not Reinforcement Learning

One last thing that matters. The name of this project is env_rl, and the harness does get better across attempts. It is tempting to call this RL. It would be wrong.

The environment half of RL — observation, action, reward — is what this project builds. It is a building block for real RL, not a replacement. A future src/env_rl/rl/ module could decompose the per-decision process reward, collect trajectories across hundreds of runs, and fine-tune an open-weights model with DPO on the (better, worse) pairs. That is the direction this scales toward. For now, the harness is explicit about being iterative self-refine, every user-facing artifact is tagged "mode": "iterative_self_refine", and the README says so in bold. Honesty in naming matters.

Conclusion

Key Takeaways

• Process integrity in an adversarial setting is a filesystem / cryptography problem, not a prompting problem. The hash-chained, append-only log + read-only monitor pattern is the foundational piece. Everything else is built on that.
• Diagnostic metrics must be measured by the judge, not reported by the agent. PyTorch forward/backward hooks are the clean way to do this — the agent never computes the numbers it could game.
• Two decoupled scores are stronger than any weighted scalar. Accuracy and process integrity cannot be traded against each other, and the sum of their failure modes gives a richer picture than either alone.
• Iterative self-refine is useful, and explicitly not RL. Feeding prior-run scores back into the next prompt measurably improves behavior on this task. The naming honesty protects readers from confusing cheap inference-time tricks with real model training.

Limitations

• Denominator gaming. A textbook ResNet-18 run where no rule ever fires gets 0/0 → 1.0 process score without being tested. Mitigation requires a minimum-firings threshold.
• Four waived rules. R2/R3/R4/R6 cannot be executed by the current harness. Real RL setups should un-waive as capability grows — that is tracked as explicit future work.
• Synthetic training plumbing. The --synthetic mode exists to verify the env works end-to-end without a CIFAR-10 download. It cannot produce a useful model; accuracy sticks at chance level.
• Live-diagnostic tolerance. The ±30% band in judge step 7 is a judgment call. Tight enough to catch fabricated trajectories, loose enough to allow normal hardware variance.

Future Work

• Un-waive R4 (add_block). Implement optimizer.add_param_group + he_init_ for a freshly appended residual block. Mid-training capacity growth becomes real.
• Per-decision reward decomposition. The coverage audit already flags each violation by rule and epoch; attributing -1/N to the specific offending decision is a small coverage-side change.
• Trajectory collection + DPO. Collect a few hundred runs with a strong base model; construct (better, worse) decision pairs from the logs; fine-tune an open-weights 7B with DPO. This is where env_rl stops being a building block and becomes real RL.
• Docker hermetic harness. The current env relies on Linux filesystem semantics documented in the README. Dockerizing makes the integrity guarantees portable and reviewable.
• Streamlit dashboard. Load /judge_logs/, render the rule-firing timeline, show decisions side-by-side with the canonical playbook remedy. Would make the loop inspectable without reading JSONL by hand.