The first real retraining of the model's core: two levers, two well-understood failures, one chosen pause

Beginner track

Where we left off

Chapter 18 ended on a real decision, taken on 28 July 2026 by the project's owner: after

eighteen attempts to repair, from the outside, the way the main model (ebwm.pt) judges "is

there a tree close by?", all of them falling into the same brightness trap, it was time to try

something else — to touch the very core of that model for the first time, rather than keep

stacking crutches around it without ever opening the bonnet. Two precise, quantified ideas were on

the table, found in the previous chapter: widen the vocabulary of gestures the model has learned to

recognise, and change the anti-collapse ingredient that stops it "cheating" by learning to confuse

everything.

This chapter tells what happened when those two ideas were tried for real. Both failed — but

each for a precise reason that was understood, not left vague. And the decision that follows is

not an abandonment: it is a chosen pause.

Lever 1: teaching the model more gestures

Recall from Chapter 18: a test had shown that in the runs where the expert chops trees, the model

has almost never seen certain gestures — jumping, strafing sideways, looking up or down. Out of

seventeen possible gestures, only eight really appeared in the original training data.

The fix was concrete: re-read the recorded runs and recover those gestures, which had been

there all along but which the data-preparation program did not yet know how to read correctly. Once

fixed, almost every gesture appears (fifteen out of seventeen instead of eight). The model was then

retrained, starting from its current weights — not from scratch — with a very cautious learning

rate (the equivalent of adjusting in small steps rather than rebuilding everything), across five

complete passes over the data.

Result: it did not work. The test measuring whether the model properly distinguishes "tree

close" from "no tree" failed both before AND after this retraining, with no gradual improvement

across the five passes. The gesture vocabulary did widen, but the way the model judges a scene did

not improve for it.

Why? An investigation, not a guess

Faced with this failure, the team asked the right question before moving on: why did it not work?

A first, simple explanation came to mind — perhaps by adding the "jump" label, the program had

accidentally erased the "attack" label on the same frames (a bit like ticking a new box on a form

and mistakenly unticking one already filled in). That explanation was checked twice, with exact

counts — and it is false: the number of frames labelled "attack" stayed rigorously identical,

frame for frame, before and after.

What really changed: about 5 frames in 100 changed label, and the overwhelming majority of that

change is frames where the expert was moving forward AND jumping at the same time (a "bunny hop", a

genuinely real movement technique in Minecraft) — a real gesture finally being seen, not an error.

The most solid explanation (but not yet proven, and the team is honest about that): those rare

gestures accounted for roughly 1% of the model's training attention before the fix, and jumped

suddenly to around 4% after — up to nine times more on certain gesture combinations. It is a bit as

if a student who had barely ever practised the piano suddenly received nine times more lessons than

usual on that instrument: even if their other subjects do not change, the shock of that new learning

unbalances their general habits. A more cautious follow-up (introducing those gestures more

gradually) was considered but not attempted — the user preferred to move straight to the second

lever.

Lever 2: changing the anti-collapse ingredient

The second lever touched something deeper: the ingredient that stops the model "cheating". Recall

from Chapter 2: a JEPA model can learn to represent all images the same way (a collapse) because

that artificially minimises its error without learning anything useful. This project has used an

ingredient called VICReg from the start to prevent that. Two recent scientific publications claim

that a newer ingredient, called SIGReg, is more reliable in theory.

Before writing any code, an examination of the existing code revealed a surprise: the version of

VICReg actually used in this project had, from the very beginning, only ever been half the full

recipe — just one of the two mechanisms the theory prescribes. That simplified the swap to SIGReg:

no need to rebuild an entire architecture, just to exchange one ingredient for another in the same

place.

The new ingredient was tested with the most cautious recipe of the whole campaign: an even

lower learning rate than Lever 1, a maximum of only three passes over the data, and above all — a

new safeguard built specifically for this test, able to detect a kind of collapse the old

monitoring tool cannot see.

A different kind of collapse, caught before it did too much damage

Here is what that new danger is, with a simple picture. The project's old monitoring tool

(batch_var, used since Chapter 2) measures whether the model's representations have all become

identical — a collapse "in height", as if an entire class of students gave exactly the same answer

to every question. But there is a second kind of collapse, more insidious: the representations

do stay different from one another overall (so batch_var sees nothing wrong), but everything

making them different concentrates onto a handful of details, whereas the model previously had more

than four thousand different ways to vary. It is as if the whole class kept giving varied answers,

but those answers now only covered 4 or 5 topics instead of several thousand — an illusory richness.

That is exactly what happened, from the very first pass over the data, and the new safeguard

detected it and stopped training before continuing pointlessly: the effective number of "useful

directions" in the model's representations collapsed from 26.7 to 4.5 — an 83% drop — while the old

monitoring tool displayed a perfectly healthy figure, as good as any previous good training run.

Without that new safeguard, nobody would have noticed in time.

On that single model snapshot saved before the full shipwreck, the central test ("does it properly

distinguish tree-close from no-tree") gave a result even worse than Lever 1 — and the test on

the model's original task (chopping trees in the simple environment) very nearly collapsed too.

This new ingredient, as tried here, broke the model more broadly than it repaired it.

Important: that failure is not counted as a seventh confirmation of the famous brightness trap

from the previous chapters. It is a problem of a different nature — a collapse of the

representations, not a confusion between colour and distance. The two problems look alike in their

symptoms ("the model fails the test") but not in their cause.

The decision: a pause, not an abandonment

Faced with two levers, two failures, each now well understood (not just "it did not work" but

"here is why"), the user chose to pause the campaign rather than immediately move on to a third

lever. This is an important decision not to misread: it does not mean that retraining the

model's core is impossible. It means that these two precise implementations, the cheapest available

for each idea, each failed for a reason that has now been identified — not for lack of trying, but

because of a real technical problem, spotted and explained.

What still stands and works, as the project's positive baseline: the rule preventing the agent from

abandoning its plan too soon, the search pushing the agent to explore the terrain rather than going

in circles, and the anti-drowning reflex. None of those three mechanisms was touched by this

attempt — nor was ebwm.pt itself, which remains identical to before, verified byte for byte.