Emergence Is a Property of Opponents: Self-Play and the Curricula That Games Build

1The single-agent illusion

Reinforcement learning is usually taught as a loop between two objects: an agent and an environment. At every step the agent observes a state, picks an action, and receives a reward; the environment transitions deterministically or stochastically to the next state. The whole discipline is organized around that diagram, and the standard formulation writes the agent’s goal as maximizing discounted return from the state it is in:

$$V^{\pi}(s) \;=\; \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^{t} R(s_t, a_t) \;\middle|\; s_0 = s,\; a_t \sim \pi(\cdot|s_t)\right],$$

with the quiet assumption that $P(s_{t+1} | s_t, a_t)$ — the dynamics — are whatever they are and never move. The agent adapts to the environment; the environment does not adapt back. That is what makes a learned policy stable, and what lets a value function converge.

Multi-agent reinforcement learning takes that diagram and breaks the assumption in the most direct way possible: other agents are part of the environment, and they are learning too. The formal object is a stochastic game [6] — a tuple $\{S, \{A^i\}_{i=1}^N, P, \{R^i\}_{i=1}^N, \gamma\}$ — where the transition now depends on the joint action of all $N$ agents, and each agent’s return depends on everyone else’s choices:

$$V^{i}(s;\, \pi^i, \pi^{-i}) \;=\; \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^{t} R^{i}_t \;\middle|\; s_0 = s,\; a^i_t \sim \pi^i,\; a^{-i}_t \sim \pi^{-i}\right].$$

The superscript $-i$ is the whole problem in two characters: “all agents except $i$.” Your value function is a function of their policies, and their policies are moving. The environment is not a fixed obstacle course; it is a set of opponents who are watching you improve and improving in response. This is called non-stationarity, and it changes the nature of the learning problem entirely — the target you are trying to hit is moving because you are shooting at it.

Most of what follows is about what you can build once you stop fighting that fact and start using it.

2Self-play, the oldest curriculum

The trick of using a learner as the environment for another learner is almost as old as the field. Minimax and alpha–beta search are the pre-deep-learning version: the search assumes the opponent will play optimally against you, which is a crude model of a learner. TD-Gammon in 1992 trained a backgammon evaluation net by playing against itself, a clean two-player version of the idea, and reached strong human-level play without any book knowledge [7]. Backgammon happened to be a forgiving domain — luck-heavy, so self-play did not collapse into a single brittle strategy.

The modern self-play lineage runs through AlphaGo and its successors. AlphaGo needed supervised data plus a value network; AlphaZero removed the data, keeping only self-play, and achieved superhuman Go, chess, and shogi with a single architecture and a single reward: win or lose [8]. The important thing for this post is not the search machinery but the training loop: the opponent pool is generated by the agent itself, and its difficulty rises automatically as the agent improves. Nobody had to build a ladder of progressively stronger sparring partners. The ladder built itself out of snapshots of the learner.

That is the germ of the idea this post is about. AlphaZero’s self-play is a curriculum engine: every game is perfectly matched to the current ability of the learner, because the opponent is the learner. When you sit a human curriculum against that, the human version is usually worse — hand-built ladders guess at the right difficulty, and they get the ordering wrong as often as they get it right. The game, played against itself, knows exactly what the agent can and cannot beat yet.

But AlphaZero is still a two-player, zero-sum game with a winner and a loser, where “improve” has an unambiguous meaning. The interesting questions arrive when you loosen those constraints: more than two agents, no scoreboard, no single measure of who won. That is where emergence stops looking like “learned a better move” and starts looking like “invented a new game.”

3Hide-and-seek and the strategy ladder

In 2019, OpenAI published a deceptively simple demonstration: two teams of agents play hide-and-seek in a physics-simulated arena, and nothing else is specified. There is no reward for hiding, no reward for seeking, no reward for building, no instruction that walls exist or that tools can be moved [1]. The hiders get reward for being unseen at the end of the round; the seekers get reward for seeing them. Everything else — all structure, all tactics, all cleverness — has to be invented by the learning process.

The agents were trained with population-based training and a self-play loop, and over roughly 25 million games the recorded behavior evolved through four distinct phases:

• Epoch 1 — pursuit and evasion. Simple chase and dodge: seekers chase, hiders run. This is the zero-th level, what you’d expect from agents with no knowledge at all.
• Epoch 2 — fortification. Hiders learn to grab the arena’s movable boxes and build walls to block doors. This is the first behavior that is not in the reward: nothing says “use the boxes.” It is invented because it wins.
• Epoch 3 — box surfing. Seekers respond by learning to move a box to a ramp and ride it over the walls. A wall the hiders built was never in the reward function, and neither is the ramp exploit that defeats it. It is a countermeasure invented against a countermeasure.
• Epoch 4 — tool use. Hiders learn to lock the seekers’ boxes in place — using the ramp or other boxes to pin the very tool the seekers learned to surf on, and in some runs to lock a seeker out of the arena entirely.

Read that ladder again with the reward function in front of you. There are two reward terms in the whole system. Walls, ramps, surfing, locking, pinning — none of them are named. Each phase is a response to the strategy that defeated the previous phase. The agents didn’t just get better at the game; the game itself changed under them, because the strategy that wins round $k$ defines the failure mode that round $k+1$ must exploit.

The proper caution here is important. This is a technically impressive demonstration of emergence within a closed, small world, not a claim that arbitrarily complex behavior blooms from any self-play loop. The behaviors are simple by absolute standards; they emerge because the game is small enough that the full strategy space is traversable, and the episodes are cheap enough to run tens of millions of times. Scale this to the real world and the compute multiplier is the story — more on that in Section 6. But as a proof of concept, the ladder is complete: given an open enough game, a self-play loop with a bare reward will produce escalating, unscripted, multi-step strategies. That is a fact about the method, and it doesn’t depend on the toy being a toy.

4Beyond zero-sum: social dilemmas

Hide-and-seek has a scoreboard: at any moment, someone is winning. Emergence is often discussed through zero-sum games for exactly this reason — the feedback signal is crisp. But the same mechanism works when nobody is keeping score in a way that has a winner and loser, and that is where the results get stranger.

The canonical starting point is Axelrod’s iterated prisoner’s dilemma tournaments from the 1980s [9]: simple deterministic strategies (cooperate/defect rules) competing in repeated encounters, with no learning at all, and the simple tit-for-tat winning not by dominating anyone but by being robust against everyone. The lesson — cooperation can win without any altruism, purely as a self-interested strategy in a repeated interaction — is the intellectual ancestor of everything in this section.

The modern version replaces hand-written strategies with learned ones. DeepMind’s sequential social dilemmas put learned agents into games with the same incentive structure as the prisoner’s dilemma but with temporal and spatial structure — a foraging game where agents must gather apples that regenerate, with an option to “tag” (shoot a beam at) other agents to knock them out [3]. What emerges, repeatedly, is a mix of cooperation and spite that no reward names: agents that tag to steal others’ apples, then tag aggressively as a defense, and in some runs a division of the arena — a de facto territorial arrangement — that no rule mandates. DeepMind’s later study even showed that injecting inequity aversion into the reward — penalizing the agent for outcomes where the others do much worse — shifts the population toward stable cooperation, an intervention at the reward level that reshapes which equilibria the learning process lands in.

The deep point is that these are not “RL algorithms being smart.” They are equilibrium selection effects: multi-agent learning is a dynamical system that settles into some strategy profile, and the one it settles into depends on the reward, the update rule, and the population structure — not on anyone writing the strategy down. When you see “emergent cooperation” or “emergent conflict” in a multi-agent paper, you are watching the learning dynamics pick a point in strategy space. The designer’s real lever is not the strategy, it is the shape of the payoff landscape. This is the sense in which the opponent is the environment — the social structure of the game is doing the teaching.

5From adversaries to leagues

There is a trap in self-play, and it is the flip side of its power: training against a single opponent is training against a stationary environment again. A lone sparring partner, once beaten, becomes easy, and the learner optimizes for beating that partner — not for being good. The result is the classic failure mode of naive self-play: cycles of forgetting, where the agent beats its frozen self from three generations ago and loses to itself from last week. AlphaZero escapes this because the opponent is always the current network; the newest version, at its own level, is exactly the right difficulty. But in richer games — and in any game where the “opponent” is actually many opponents — a single self is not enough.

The production answer is to replace “the self” with a population. In population-based training, a set of agents is trained in parallel, each periodically evaluated against and imitated from the others, with hyperparameters mutated across the population [4]. The opponent is not the current network; it is a distribution of networks of different ages, skills, and styles. AlphaStar used this to beat top professional StarCraft II players, maintaining a league of agents — main agents trained to beat the league, and exploiters trained specifically to find and expose the main agents’ weaknesses [10]. OpenAI Five did the same for Dota 2: a pool of teams that played each other and against past checkpoints, so the training signal was “beat this whole crowd,” not “beat this one guy” [5].

Why does this matter so much for emergence? Because a population is the curriculum. When the opponent pool contains diverse strategies, the learner is forced to develop responses to strategies it has never seen — which is exactly the pressure that produces new behavior rather than refinement of old behavior. A single opponent produces specialization; a population produces coverage. And there is a subtlety worth stating plainly: the goal in multi-agent RL is not to beat anyone in particular — it is to beat the field, and the field is best represented by a population that includes past, present, and deliberately-weird versions of the learner itself.

6What emergence is, and isn't

Given the word’s cargo, it’s worth being precise about what these results do and do not show.

What emergence is, operationally. In all of the above, a behavior appears that no component of the training signal explicitly targets: wall-building, box-surfing, territorial division, cooperation. The behavior is not specified in the reward, the state representation, or the model architecture. It is a fixed point of the learning dynamics — the strategy profile the population converges to — and it is discovered, not designed. That is a clean, falsifiable definition: if you removed the reward terms one by one and the behavior still appears, it emerged.

What it is not. Nothing here demonstrates general-purpose creativity, or behavior transcending the game’s physics and rules. Hide-and-seek agents did not learn “tools” in the human sense; they learned game-specific manipulations that we label with tool words after the fact. The emergence is bounded by the world the agents live in, and the world is tiny. More importantly, emergence here is not a property of the agent; it is a property of the game being open enough. If the game’s state space is closed and small enough to be fully searched, the “emergent” behavior was always in the strategy set — the learning process just found it. The genuinely interesting question is what happens when the strategy set is too large for that — the regime where the behaviors are not merely undiscovered but unreachable in practice by any curriculum we can run.

That last point is the honest cost. The strategy ladder in hide-and-seek took roughly 25 million episodes — and that is in a toy physics world with a handful of agents. The moving-target problem multiplies sample complexity in ways that single-agent RL never faces: every policy update changes the environment for every other learner, so experience collected under an old opponent distribution is stale the moment policies move. The standard remedies — huge parallel rollouts, population training, evaluation against checkpoints — are all compute multipliers. Emergence, when it works, is bought with sample volume and opponent diversity. There is no free lunch, and the menu is written in GPU-hours.

7For the efficient-inference crowd

This site is normally about low-bit weights and memory-bound inference, and it is worth closing the loop: multi-agent RL is quietly one of the most inference-hungry workloads in applied ML, and it repays exactly the systems discipline this site keeps preaching.

Consider what a single training step of a modern MARL system does. For every learner in the population, you run the policy — the network forward pass — in thousands of parallel environments, at every timestep, generating experience to learn from. Then you run evaluation: the same networks, again, against opponent checkpoints and exploiters, to decide what to train next. In hide-and-seek’s 25 million episodes, the forward passes are measured in the trillions. The training is only partly a compute problem — it is substantially a memory traffic problem: policy networks, replay buffers, opponent checkpoints, and evaluation results all streaming through the same machines, with the ratio of inference to gradient steps stacked heavily toward inference. A 4.6 pJ multiply-add is nothing next to 20 pJ per bit of DRAM traffic — the arithmetic of the policy nets is almost never the bottleneck [11]; moving the weights and the experience is.

Which is to say: the standard MARL scaling advice — “more environments, more rollout workers, bigger populations” — is a systems claim wearing a research claim’s clothes, and the parts that make it affordable (quantized or low-bit policies for rollouts, weight-level caching of shared opponent checkpoints, batched evaluation) are the parts this site keeps writing about. The deeper echo is conceptual: NNUE solved chess evaluation by not re-reading what didn’t change [12], and MARL’s evaluation problem is the same shape — most of a multi-agent system’s work is re-reading state that barely changed between timesteps, in environments that are copies of each other. Every trick for cutting that traffic applies twice.

The genuinely load-bearing idea, though, is the one this whole post has been circling: in multi-agent learning, the environment is the opponent, the opponent is the curriculum, and the curriculum is the product. Get the population structure and the reward geometry right, and behaviors no designer wrote down will emerge on their own — the system teaches itself, and the only thing it needs from you is enough cheap inference to keep up with its own improvement.