Broken curiosity, then two patches that half-work

Beginner track

The problem we are tackling: finding the first tree, alone

Chapter 6 ended on a well-identified wall: the agent knows how to craft planks perfectly once it has wood, but in a real randomly generated survival world, it never finds its first tree — 0 logs out of 5 tested episodes. This chapter tells the story of the first major campaign to attack this wall. Honest spoiler, as always in this project: the first idea fails cleanly, and the subsequent patches only provide partial progress — but each failure reveals something precise about the true nature of the problem.

Idea #1: Curiosity — rewarding the agent for what surprises it

The idea is called artificial curiosity. The principle: instead of only rewarding the agent for reaching a goal, give it a small reward every time it goes towards something surprising — a place its world model understands poorly, where its predictions are very wrong. The intuition: if the agent never finds a tree because it doesn't know which direction to look, pushing it towards what surprises it should naturally make it explore rather than standing still.

Practically, the project built a small committee of 5 "guessers" that each try to predict what will happen next in the embedding (the number-summary of the image, Chapter 1). When the 5 guessers agree, it is a familiar place — nothing surprising. When they disagree, it is a poorly understood place — potentially interesting to explore. This idea is called Plan2Explore and comes from a real research paper.

Before launching this experiment (and all the following ones in this chapter and the next), the project built a small team of three specialized software agents — one that proposes an experiment based on real scientific references, one that implements it without ever breaking what already works, and one that tests it honestly and reports the true numbers. This discipline prevents telling oneself stories about one's own results.

The result: no difference, and slower

Comparison over 20 episodes of Treechop (the "just chop a tree" mode from Chapter 5), with and without curiosity enabled: without curiosity, 30% success; with curiosity, 25% — a difference far too small to be reliable with only 20 episodes on each side, and above all, with curiosity, the game runs two and a half times slower (the committee of 5 guessers has to judge each of the 512 imagined stories at every instant, Chapter 4). Honest verdict: curiosity brought nothing, and it cost a lot in speed.

Why it failed: the committee agreed too fast

Looking at the training of the committee of guessers, the explanation becomes clear: by the third training pass, the 5 guessers all agreed with each other, and stayed in agreement for the rest of the training. A committee that always agrees can never again signal "this is surprising" anywhere — the curiosity signal flatlined everywhere, which explains exactly why enabling curiosity changed nothing (except slowing down the game).

The real cause, more interesting than the symptom: the real Plan2Explore method trains its committee of guessers continuously, on what the agent discovers while exploring itself — a naturally varied stream of data, since the agent constantly moves towards new places. This project, however, trained the committee once and for all, on frozen recordings of expert games where they always chop the same tree the same way. On such repetitive data, all 5 guessers find the same simple solution, and never again have a reason to disagree. The project reproduced the form of the method without reproducing the condition that makes it work.

Idea #2: Stop looking through the lens of curiosity

After this failure, the project stepped back and asked the question differently: what does the planner (Chapter 4) actually, mechanically do when no tree is visible? Two very concrete flaws appeared — neither of which requires machine learning to be fixed.

Flaw 1: The planner cannot even imagine the right gesture. To search for an invisible tree, the right gesture is "turn the camera in one direction for several instants in a row". But the planner draws its 512 imagined stories by choosing, at each instant, an action completely independent of the previous one — a bit like rolling a 17-sided die 12 times in a row without memory. The probability of such a draw producing the "turn" action 6 times in a row is practically zero. The problem is not that the planner chooses poorly — it's that the right gesture doesn't even exist in the list of things it considers.

The fix is called sticky sampling: at each imagined instant, instead of drawing a completely new action, repeat the previous action with a certain probability. This single rule jumps the consecutive repetition rate from 7% (independent draw) to 72% (70% sticky) — the 512 imagined stories now contain gestures held over time (turn-and-look, walk-towards, strike continuously).

Flaw 2: The agent doesn't even know it is lost. When a tree is visible, the 512 imagined stories get very different scores (some get closer, some move away). When nothing is visible, the 512 scores are almost identical — the final choice becomes a disguised coin flip. This gap between scores (their standard deviation) is a free signal, already calculated, never used before: high means "I see something useful"; close to zero means "I am lost". A small rule was added: after several consecutive turns with this signal close to zero, the agent temporarily abandons the planner and turns the camera methodically until the signal goes back up (a tree entered the field of view) or a budget of turns is exhausted.

Take this for what it is: this is not learning. It is a hand-written reflex — the kind of thing that heavier methods (out of reach for a single consumer GPU) learn on their own. But it makes no sense to teach an agent to explore if it cannot even turn its head properly.

The results: real progress on Treechop, zero on cold-start

On Treechop (the agent always starts in a forest), the sticky+reflex combination yields a nuanced result: at a moderate setting (50% sticky), the success rate remains within the normal range already observed in Chapter 5, but each success yields about twice as much reward — the stickiness helps to dig deeper (keep chopping a tree already found) more than to search wider (find more trees). At a stronger setting (70% sticky), the agent gets stuck too much on a single gesture and the result slightly worsens.

On the real cold-start (random spawn, no starting wood): zero logs chopped, in all tested configurations. Worse, at a certain threshold setting, the "turn head" reflex becomes pathological: the agent turns the camera 82 to 92% of the time, literally spinning in circles without ever stopping. The most revealing observation: in some episodes, the agent ends up surrounded by trees and still does not chop them. So it is not (only) a search problem — it is an approach and chop problem once the target is found.

A variant tested for understanding: give the agent two separate "brains" — the one that proved itself on Treechop drives the tree search, the new model takes over from the first log to craft. Result: still zero logs, but the behavior changes markedly (recognizable lumberjack gestures instead of confused wandering). One of the episodes shows the agent spawning in a rocky ravine without a single tree, and hitting stone instead — a reminder that a spawn point with no wood at all cannot be solved by any search reflex.

One last attempt before concluding: fine-tuning the model on more diversity

Last avenue tested in this campaign: if the "I am lost" signal is blurry on the crafting model (unlike the Treechop model where it is sharp), it might be because the expert demonstrations used to train it rarely show moments where the expert is "lost" — experts find wood fast, they never linger in the void. By adding about twenty short games played randomly (thus, statistically, often lost) to the training dataset, and then lightly retraining the model on this mix, the "lost vs. found" signal indeed becomes much sharper than before.

But the final result does not change one bit: still zero logs chopped, with or without this fine-tuning. Making the signal sharper did not make the behavior better. This discovery confirms what the two-brains experiment had already suggested: the wall is not in the perception (knowing one is lost), it is in the behavior (knowing what to do once something is found, or how to cover ground when nothing is found). The next chapter continues this investigation with even more direct tools.