A real research journal, in progress
Watch a JEPA agent learn to play Minecraft — and understand how it works, chapter by chapter
Mine-JEPA is a lightweight JEPA (Joint-Embedding Predictive Architecture) trained directly on game images, built to plan its own actions rather than reacting blindly. This site tells the story of how it was built, in the order it actually happened — including the parts that didn't work. Each chapter is written twice: once for a curious beginner, once for someone who already knows machine learning. Pick a track, switch anytime.
Start the journey
MineRLTreechop-v0: the agent, guided by its JEPA world model,
plans its approach to a tree and chops it down to obtain a log. This is the best recorded
episode among those evaluated — the actual success rate of this policy is between 1-in-4 and
1-in-2, not every single attempt.
The Tech Tree
Chapters follow the project in the order it actually unfolded: each builds on the ideas and results of the previous one — like a crafting recipe where each slot only unlocks once its ingredients are gathered.
-
What is JEPA, and why train a program to play Minecraft with it?
No prerequisites — this is the starting point of the path.
-
The shortcut the model always tries to find (and how we prevent it)
Prerequisites: Chapter 1, "What is JEPA, and why train a program to play Minecraft with it?"
-
Teaching the model to imagine what comes next
Prerequisites: Chapter 1, "What is JEPA, and why train a program to play Minecraft with it?" and Chapter 2, "The shortcut the model always tries to find (and how we prevent it)"
-
Trying 512 futures in its head before pressing a button
Prerequisites: Chapters 1 to 3 — the encoder, anti-collapse safeguards, and world model.
-
The leap to real Minecraft: four failures, then an agent that chops trees
Prerequisites: Chapters 1 to 4 — encoder, anti-collapse safeguards, world model, and imagination planner.
-
Learning to craft: the rule is understood, the first tree remains out of reach
Prerequisites: Chapters 1 to 5 — encoder, anti-collapse safeguards, world model, planner, and the first success in real Minecraft.
-
Broken curiosity, then two patches that half-work
Prerequisites: Chapters 1 to 6 — up to the wall of finding the first tree solo, which this chapter attacks for the first time.
-
The wall is behavioral: one real success, then four proofs that refining the signal is not enough — and a fifth that confirms the diagnosis
Prerequisites: Chapters 1 to 7 — including the first two failed attempts to find the first tree solo.
-
After tracks A and C: the learned policy was promoted, then tested (following Chapter 10)
Prerequisites: Chapters 1 to 8 — including attempt #8 (avenues A and C, now tested and NO-GO, see Chapter 8). This chapter distinguishes acquired results from an unexecuted plan (learned policy and two refinements).
-
The learned track was tested: it too is not enough — but it eliminates two explanations
Prerequisites: Chapters 1 to 9 — including the learned policy by behavioral cloning (Proposal B), promoted to priority 1 in Chapter 9.
-
The campaign takes a pause: the compass did not turn off, it points backwards
Prerequisites: Chapters 1 to 10 — including the diagnostic by elimination of Chapter 10, which points to the mechanism tested here.
-
Two paths from the menu put to the test: repairing the compass fails a third time, memory of visited places produces the second real success of the campaign
Prerequisites: Chapters 1 to 11 — including the first two paths of the menu opened in Chapter 11 (repairing the compass, a memory of visited places), tested here for real. Compass repair fails a third time (the brightness shortcut lives in the frozen encoder); memory of visited places produces the second non-zero result of the entire cold-start campaign, without behavioral pathology.
-
The drowning alarm works perfectly; the rescue action, however, does not know where dry land is
Prerequisites: Chapters 1 to 12 — including the topological frontier memory (attempt #12), for which a review of the logs here reveals that 12 of the 20 episodes in its own confirmation batch actually ended in drowning. The pixel-based drowning detector built in response is accurate and well-calibrated; the wired escape action does not know where dry land is — same lesson as attempt #5 in Chapter 8, on a completely different mechanism.
-
A web giant put to the test: CLIP does spot the forest — but it confuses that with brightness too
Prerequisites: Chapters 1 to 13 — including the brightness shortcut confirmed three times. An off-the-shelf 400M-image model (CLIP), never trained on Minecraft, is asked whether it already spots a forest correctly: it does, and it is caught by exactly the same brightness confound. A direct retraining of the world model with photometric augmentation then produces a mixed result, not a clean fix.
-
A fifth confirmation for the brightness shortcut, a false alarm that wasn't one, and a 144-point riddle still unanswered
Prerequisites: Chapters 1 to 14 — a hand-built colour heuristic is rejected before being written, then a narrower version is tested anyway and fails the same gate. The sharpest interpretation of the campaign: dark forests versus bright fields is this domain's real scene composition, so no single-frame photometric feature can separate them. Separately, the drowning fix holds at N=20.
-
The closest thing to H-JEPA yet predicts the future, not the image — and still loses to a plain average
Prerequisites: Chapters 1 to 15 — the first mechanism built under the 'no photometric scoring' constraint: predict how much new ground a heading will uncover, from motion and visit history only. Brightness genuinely does not dominate the target (the campaign's most reproducible clean result), but no per-trial model is learnable at this sample size.
-
Even a detector that cannot learn anything falls into the same shortcut: the problem is in the eye, not in the learning
Prerequisites: Chapters 1 to 16 — two direct attacks on the never-repaired compass. A detector with no gradient, no loss function and nothing to learn falls into the same brightness trap as five trained systems before it, which relocates the defect from 'how we train' to the frozen representation itself.
-
The campaign's first victory did not survive a larger sample — and it was the team itself that caught it
Prerequisites: Chapters 1 to 17 — two new leads from recent literature. The first looks like the campaign's first genuine success and is retracted the same session once the hand-labelled sample is nearly tripled. The second surfaces a real, non-photometric gap: the two training sets do not exercise the same actions.
-
The first real retraining of the model's core: two levers, two well-understood failures, one chosen pause
Prerequisites: Chapters 1 to 18 — the first real retraining of the model's core in nineteen attempts. Both scoped levers fail, each with a diagnosed cause, and a purpose-built safeguard catches a collapse mode the old monitor cannot see: representations keep varying while their effective dimensionality drops by 83%.
-
Doing nothing predicted better: twenty attempts fixing the driver, when the problem was the engine
Prerequisites: Chapters 1 to 19 — the campaign's closing chapter and its single most important measurement. Nobody had ever checked whether the world model reacts to the agent's actions at all. It does — but conditioning on the true action predicts the future measurably worse than assuming the agent did nothing, on the training domain itself.
The journey stops honestly where writing stops today: no future slots have been added in advance.
Why this feels like a lab notebook, not a showreel
Mine-JEPA is a real, ongoing project, and this site is written directly from its research log — including attempts that failed, partially conclusive results, and problems nobody has solved yet. If a chapter reports a negative result, it remains a negative result here: no forced enthusiasm, no confetti. The honest version of this story is also the most interesting.