What is JEPA, and why train a program to play Minecraft with it?

Expert track

Problem Statement

The backbone of Mine-JEPA is a lightweight JEPA (~15M parameters) trained end-to-end on pixel Crafter/MineRL data, chosen explicitly over a heavy frozen video foundation model (see rationale for rejection below). The architecture follows Yann LeCun's JEPA proposal (A Path Towards Autonomous Machine Intelligence, openreview BZ5a1r-kVsf, 2022) and inherits its encoder lineage from I-JEPA (Assran et al., arXiv:2301.08243, CVPR 2023).

Why Not Generative Pixel-Space Prediction

A generative model (autoencoder, diffusion, autoregressive pixel model) is trained to minimize a reconstruction or likelihood loss over the entire observation — every pixel carries equal weight in the objective. On a POV Minecraft/Crafter view, this means the loss is dominated by high-entropy, task-irrelevant signal (foliage texture, sky gradient, lighting noise) rather than the low-dimensional task-relevant structure (agent posture, nearby object identity, inventory state). This is precisely the generic failure mode JEPA is designed against: features useful for control are generally low variance, high influence in the pixel signal, not high variance — see Littwin et al. (arXiv:2407.03475) for the theoretical argument showing that JEPA's implicit bias favors precisely these high-influence predictive features over the high-variance noisy features pursued by reconstruction objectives (e.g. MAE).

Architecture

Three components, all operating on a per-frame latent s ∈ R^D (D=128 in Mine-JEPA's Phase 1 Crafter encoder):

x_t  ──→ [Context Encoder f_θ]  ──→ s_x
                                       │
                              [Predictor g] + a_t  ──→ ŝ_{t+1}
                                                             │
x_{t+1} ──→ [Target Encoder f_θ̄]  ──→ s_y                  │
                                       │                     │
                            L = ‖ŝ_{t+1} - s_y‖²  ←──────────┘

• Context encoder fθ: ResNet5 (~40K parameters) taking a 64×64 RGB image as input (docs/01jepa.md).
• Target encoder fθ̄: Architecturally identical, but its weights are an Exponential Moving Average (EMA) of fθ (θ̄{t+1} ← 0.99·θ̄t + 0.01·θt), with gradients blocked (@torch.nograd() in update step). This mechanism prevents both encoders from co-collapsing into a trivial constant — see Chapter 02 for full anti-collapse details.
• Predictor: A small MLP (or lightweight convolution) taking sx and a discrete action embedding (Embedding(nactions, 32) over Crafter's 17-action space), producing ŝ_{t+1}.

The loss is strictly L2 in latent space: LJEPA = ‖ŝ{t+1} - s_y‖². There is zero pixel reconstruction term in the objective — this structural distinction defines JEPA against generative/autoencoding world models, not a mere training detail.

Action Conditioning and Latent Rollouts

Because the predictor is conditioned on at, it encodes transition dynamics rather than marginal next-state statistics: ŝ{t+1} = g(st, at). This enables full latent unrolling without calling the render engine or environment:

s_1 = g(s_0, a_0);  s_2 = g(s_1, a_1); ...; s_k = g(s_{k-1}, a_{k-1})

This "latent imagination" is the mechanism leveraged by the planner (upcoming chapter on world model and planning) via MPC/CEM: sample candidate action sequences, unroll them in latent space, score them against a goal embedding, execute the first action of the best sequence, replan.

Architectural Choices & Rejected Alternative

The project explicitly rejected frozen V-JEPA 2 (Assran et al., arXiv:2506.09985, 30 authors, 2025) as the primary backbone: a 600M parameter ViT-H trained on ~1M hours of natural video is out-of-distribution (OOD) on stylized Minecraft POV, and is neither clonable nor end-to-end fine-tunable on a consumer GPU with 8GB VRAM. It remains available solely for secondary comparison via torch.hub, never as a planning substrate. The chosen backbone design — lightweight, trained directly on target domain, action-conditioned — closely follows LeWorldModel (Maes, Le Lidec, Scieur, LeCun, Balestriero, arXiv:2603.19312, 2026): ~15M parameters, single GPU, and a two-term objective (next-embedding prediction + variance/covariance regularizer) mirrored in Mine-JEPA's loss formulation (see Chapter 02).

Why This Over an LLM/VLM Policy for This Task

JEPA and language/vision-language models solve distinct problems and are not competing substitutes here: an LLM-driven agent ("computer use") reasons well over high-level instructions, but at ~1–10s per decision is far too slow for reactive control, and its "understanding" of scene dynamics goes through textual description rather than direct pixel-conditioned prediction. Mine-JEPA's JEPA backbone runs in under 100ms per action and predicts visual dynamics directly from pixels, at the cost of lacking high-level reasoning. The project framing: an LLM would specify what to do (high level), JEPA executes how to do it — this project builds and validates only the latter.

References (Verified, from docs/references/index.md)

• LeCun, A Path Towards Autonomous Machine Intelligence, openreview BZ5a1r-kVsf (2022) — original JEPA proposal.
• Assran et al., I-JEPA, arXiv:2301.08243 (CVPR 2023) — encoder lineage.
• Maes, Le Lidec, Scieur, LeCun, Balestriero, LeWorldModel, arXiv:2603.19312 (2026) — closest published architecture; source of ratio = valpred/valcopy gate used from Chapter 03.
• Assran et al., V-JEPA 2, arXiv:2506.09985 (2025) — rejected as primary backbone, comparison only.
• Littwin et al., arXiv:2407.03475 (2024) — theoretical foundation on why JEPA favors predictive over high-variance noisy features.