No command
Reyn1 and Reyn2 · two level-5 CPUs
Intro
Last month, we introduced Faynt, a family of open-weight models that can play Super Smash Bros. Melee. We trained it on approximately 840,000 human replays and refined it through reinforcement learning (RL).
Faynt could compete in one-on-one (1v1) matches, but we could not steer it with commands. What if it could also play two-versus-two (2v2)?
And to take it even further, what if it could become a language-steerable player you could coordinate with in real time, in both 1v1 and 2v2?
- Would it still be able to fight back while following your commands?
- Would it be able to coordinate with a human player?
- What about friendly fire? How would it avoid hitting its teammate?
Enter Reyn 26C, our language-steerable evolution of Faynt. We started with Fox, expanded to four characters, and now steer one shared model across all 26. Reyn 4C remains a development baseline in the comparisons below. Our Fox team and combo lab are separate experiments.
In the opening video, we pit two Reyn agents against two in-game bots. Each agent receives its own steering commands, displayed above its character, and adapts its gameplay in real time. The second part is a separate combo-lab relay. Both are selected examples.
In this article, we’ll go in depth about how we created Reyn, including:
- Language steering: pre-training and RL, starting with 1v1 matches.
- Voice: a conversational pipeline turns speech into commands for Reyn and spoken replies about the game.
- 2v2: bringing steering to team games and dealing with friendly fire.
- Bonus: training a Wombo Combo 2v1 model, just for fun.
Steering: Reyn 26C on the original command suite against mirror level-9 CPUs. The team-game and teammate-damage results come from separate Fox experiments; the latter uses a human proxy.
The goal
What we are building
We want to:
- Build a Melee model you can steer with natural language.
- Push what a small model can do, and extend steering to all 26 characters.
- Keep the model able to fight back while following your commands.
What Reyn is not trying to do
We are not trying to build the most competitive win-at-all-costs Melee bot or a standalone voice companion. The goal is a teammate that can still fight while responding to a person's instructions.
We started with Fox to keep the first steering experiments simple, then expanded to four characters. To reach the full roster, we tried reusing that gameplay stack and teaching its text router more requests. That became Reyn 26C. It handles special-move requests across the roster, although quality still varies by character. The experimental 2v2 branch currently uses Fox.
The plan was to make Faynt steerable, bring that steering to 2v2, then build toward a partner for a person.
Starting with language steering
We started by adding language steering to Faynt, with as little loss of playing strength as possible.
Following commands while still fighting
A request to grab, move somewhere or stop attacking changes which actions are useful. We wanted those requests to change Reyn's behaviour without making it ineffective at the game. To teach those behaviours, we began with Faynt's weights and added language labels to human replays. Rules called predicates identify actions or situations and pair them with matching instructions. Given an instruction and the game history, Reyn learns to predict the controller action from the replay. This imitation stage trains the policy network, cross-attention adapter and original text encoder together.
Asked to Grab him.
, it tried more grabs, but few connected. We then used RL to reward grabs that landed.
RL also lets us reward a hit on the opponent or arrival at the requested platform. It gives the model practice with behaviours underrepresented in the labelled replays, such as moving to platforms. Imitation connects instructions to actions; RL improves what those actions achieve.
Post-training with RL
We developed the gameplay stack in Reyn 4C. After imitation learning, we froze the 10 million-parameter gameplay policy and trained its cross-attention adapter, command embeddings, state adapter and taunt head with RL. The state adapter reads 16 game features; the taunt head adds an input the frozen policy cannot produce. About 2.2 million parameters trained in this stage. With no command, the steering components are bypassed.
We train these parts with proximal policy optimization (PPO). Kullback–Leibler (KL) divergence measures how differently two policies assign probabilities to actions in the same game situation. We use a KL penalty to discourage large changes from the no-command policy while learning the requested behaviour. Specialists learn groups of commands, then distillation combines what they learned into one model. The Reyn 4C model then received a joint RL round across all six stages.
For Reyn 26C, we kept those gameplay weights fixed and fine-tuned only the text router on requests across the roster, including different ways to ask for the same move. It learns which supported command the words mean. This supervised training is separate from the RL that taught the gameplay response.
Scroll sideways to see the whole figure.
Scroll sideways to see the whole figure.
Designing the rewards
Before writing a reward, we decided what would count as following the command. A test needed an observable behaviour, a success threshold and a comparison with no command. We then automated the loop: propose a reward, train a specialist, check the benchmark and replays, and revise the reward. Fire Fox shows why that loop mattered. The other specialists went through similar iterations with rewards tailored to their commands.
Example: automatic reward tuning for Fire Fox
Fire Fox sends Fox flying, so a badly aimed launch can throw away a stock. Our first specialist landed hits but also self-destructed. We increased the self-destruct penalty, and it started avoiding using the Fire Fox move instead. This was reward hacking: avoiding the requested move was an easier way to avoid the penalty than learning to use it safely.
So we changed the reward to catch bad launches sooner. If a launch looked likely to send Fox too far past the stage edge, it got a penalty straight away and no reward for that launch, even if it landed a hit. We then made that check more cautious and rewarded aiming at the opponent. In the later test, the specialist still landed 5.2 Fire Fox hits per minute, with 0.08 self-destructs per minute.
| Step | What changed | What happened |
|---|---|---|
| First attempt | Reward Fire Fox use and penalize self-destructs. | 5.4 Fire Fox hits and 0.57 self-destructs per minute. |
| Bigger penalty | Make self-destructs more costly. Other game rewards changed too. | Fewer attempts and hits in the training log; we stopped this trial. |
| Catch risky launches sooner | Check where Fox is heading. Penalize launches likely to end too far past the stage edge, and give those launches no reward. | 4.7 Fire Fox hits and 0.42 self-destructs per minute. |
| More room for error and better aim | Make the safety check more cautious and reward aiming at the opponent. | 5.2 Fire Fox hits and 0.08 self-destructs per minute. |
Fire Fox is one specialist example. We distilled the specialists into Reyn 4C and refined it with joint RL. The later router update produced Reyn 26C, which meets at least the looser target in 386 of 424 original-suite cases: 267 meet the stricter target and 119 meet only the looser one. The comparison below shows both grades.
Natural language
To tell Reyn what to do, we first wanted a small, fast path from words to a command. During Reyn 4C development, predicate-labelled pre-training used a 0.6 billion-parameter text encoder beside a 10 million-parameter gameplay policy. Running that encoder for each request would add latency and make the language machinery much larger than the model playing the game.
A smaller router recognizes a request and selects a stored command embedding. Reyn 26C extends that approach with a router trained on more character moves and varied phrasing. It reuses the gameplay weights from Reyn 4C: the extra character coverage comes from the language path, rather than another round of gameplay RL. In the additional text-only wording test, it routed 100 % of 208 generated test rows correctly.
A smaller language path
We then tried more compact command representations and lighter routers. Every setup went through the same 424 test cases with held-out wordings. The question was practical: how much size and request time could we cut without losing command following?
In the Reyn 4C development experiment, the smaller-router setup cut the text-side files from 164.1 MB to 28.8 MB. On this held-out-wording benchmark, it met at least the looser command-following target in 88.2 % of cases, compared with 88.4 % for the original setup. Median text-request time fell from 13.84 ms to 4.09 ms. Compressing the command representation to a single token hurt command following.
Voice interaction
The router could turn “Move left” into a gameplay command quickly. With a larger language model, Reyn can go beyond “Moving left”: answering questions about the match, following the conversation, or offering a playful roast after a misplay.
For that exchange, the conversational pipeline replaces the router with a larger language model. Speech recognition supplies the words; the model reads them alongside conversation history and recent game context. It can issue a supported command with its reply, or just talk. Any command is checked for the current character and stage before it reaches Reyn’s gameplay policy.
Speech synthesis streams the reply back. “Moving left” says what Reyn intends to do; execution feedback supplies failures and observed results.
The extra flexibility costs time: the larger model adds latency. To reduce the wait, the pipeline starts preparing a reply while you speak, then waits for speech recognition to mark the request complete before acting or speaking.
The video below shows one of our earlier prototypes, using prepared synthetic speech, speech recognition and a router.
Reyn1: no command
Prepared synthetic speech
Reyn1 · CPU (CP)
From four characters to 26: what changed?
Next level — 2v2
The broader singles model leaves the team-play question open. In a separate 2v2 branch, we built on the 1v1 model and trained a Fox pair for team games. Each Fox had a partner it could accidentally hit, so fighting well also meant leaving that partner room to act. We trained against two copies of Faynt in melee-sim-light, a fast simulator, then checked the pair in the real game. Results below are real-game measurements unless labelled simulator.
Our 2v2 work is experimental and needs further refinement. The eventual goal is for a person to join the team.
Approach for team games
Adapters beside a frozen policy had worked for language steering. We tried the same idea for 2v2, aiming to make training faster and less expensive by updating fewer parameters. First came a team adapter that sees all four fighters, then a target selector. Neither produced a clear gain. The team adapter won 51.7 % in a preregistered simulator test; the selector rarely changed a decision. Training all policy parameters worked better: the resulting pair won 84.4 % of 160 real games against two Faynts, compared with 56.2 % for the untrained pair.
Pick an approach to see what it trains and what stays frozen.
Four ways to put Faynt in 2v2
what trains, what stays frozenFaynt sees itself and one opponent, with no direct input for a teammate or a second enemy. Its state encoder, causal Transformer and action head form one policy. In 2v2, each Fox must be told which enemy to look at.
Each Fox keeps one memory per enemy. The nearest-enemy rule selects a hidden state; the adapter combines it with the current team features and adds a residual before the frozen action head. Held close to Faynt by the KL penalty, this approach stayed at a coin flip.
Coordination as who attacks whom, with the fighting left to Faynt. The target head reads both enemy-specific histories and team features, then picks which candidate action to use. The choice barely moved from its starting prior and never changed a decision.
Our pair gets its own trainable copy of Faynt, so learning can change how each Fox fights near its partner. The frozen original is evaluated separately for opponents' play and for the KL reference. KL compares both policies on the learner's decisions. This is the approach that worked: the 2v2 pair in this post.
Explore · the win rate during training
Pick a mark on the curve to see the win rate at that point of training, in the simulator.
Comparing multi-agent RL algorithms
We compared four multi-agent RL algorithms using the same training setup and 19.7 million simulator frames each. They differ in policy sharing, critic inputs and how they reuse a batch of experience. The critic estimates expected return during training; each Fox acts from its own observation at play time. Alongside multi-agent proximal policy optimization (MAPPO) and independent proximal policy optimization (IPPO), we tested heterogeneous-agent proximal policy optimization (HAPPO) and a one-pass variant inspired by advantage actor–critic (A2C).
| Algorithm | The network | What the critic sees | How each batch of play is used |
|---|---|---|---|
| MAPPO | one network shared by both Foxes | both Foxes' views | clipped PPO, a few passes over each batch |
| IPPO | one shared network | its own Fox's view only | as MAPPO |
| HAPPO | a copy per Fox | both Foxes' views | the two copies update one after the other, the second seeing the first's change |
| One pass (A2C-style) | one shared network | both Foxes' views | one gradient step per batch, nothing reused |
Explore · how MAPPO, IPPO and HAPPO differ, drawn
Select an algorithm to see its critic, policy sharing and update order. The diagram uses the earlier adapter-only setup; the table above describes training the full policy.
Single-agent PPO: one Fox learns
- The critic
- One Fox and one critic, which judges how good the moment is from that Fox's state alone.
- What changes
- Faynt learned 1v1 this way, with every weight training. Put two copies in a 2v2 game this way and each treats its partner as part of the world.
- In our runs
- Faynt's own training, the starting point of every 2v2 run.
IPPO: each Fox judged on its own
- The critic
- The same local critic is evaluated separately for each Fox. It reads that Fox's hidden states and team features, but not the partner's hidden states. The policy parameters are also shared.
- What changes
- Compared with MAPPO, the critic lacks the partner's separately encoded history. The partner keeps changing, so its predictions keep shifting too.
- In our runs
- Every weight trained: 79.3 % on one seed, close to MAPPO. The adapter alone: 39.6 %, stopped early.
MAPPO: one critic for the team
- The critic
- A centralized critic reads both Foxes' hidden states and team features. Each policy still acts from its own inputs.
- What changes
- Compared with IPPO, only the critic changes. Its judgement of the team's situation should be steadier, so the signal that drives each update should be less noisy.
- In our runs
- Every weight trained: 82.8 % over three seeds, the highest of the four, level with IPPO within the spread between seeds. The adapter alone: 45.7 %, stopped early.
HAPPO: one copy per Fox, updated in turn
- The critic
- The same team critic as MAPPO, but each Fox has its own weights.
- What changes
- Each update chooses a random order, fits one Fox first, measures on every frame how much that Fox's moves changed, and scales the second Fox's update by it. The second Fox then learns given its partner's new behaviour.
- In our runs
- Every weight trained: 70.7 / 73.0 % on two seeds, about 11 points behind MAPPO. The adapter alone: 36.2 %, stopped early.
One pass: the same team critic, no reuse
- The critic
- Everything as in MAPPO.
- What changes
- The policy takes one gradient step on the whole batch; the critic is fitted separately. That policy step starts where PPO's clip cannot act, so this tests whether PPO's several passes help.
- In our runs
- Every weight trained: 67.2 / 68.8 % on two seeds, about 15 points behind MAPPO. The adapter alone: 53.1 %, the best of the adapter runs but not clearly above the adapter left at zero, 49.5 %.
Drawn for the adapter runs, where each Fox's backbone stays frozen and a small adapter trains, four passes a batch. When every weight trains, the whole network trains along with the adapter, with fewer passes; the differences between the algorithms stay the same. From our team-cohesion study.
Win rate by algorithm
against two copies of FayntSimulator, 19.7 million training frames each, every run read on the same games. A bar is an algorithm's mean, a dot one training seed; the dashed line is a coin flip.
Only the small adapter trains, in the real game. The runs stopped at about 2.7 million frames and the rates come from their own training games, so they show the early trend only.
MAPPO averaged 82.8 % over three seeds. IPPO scored 79.3 % on one seed, within MAPPO's 10-point seed range, so this comparison does not distinguish them reliably. HAPPO and the one-pass variant trailed MAPPO by about 11 and 15 points. In the earlier real-game adapter-only runs, no algorithm clearly beat leaving the adapter at zero.
Steering two agents
We added steering adapters to the 2v2 pair, allowing separate commands for each Fox. Against two Faynts in the real game, 10 of 12 tested commands were followed whether one or both Foxes were commanded. Without a command, the adapters are bypassed.
Each Fox's active command appears above it in the clip.
No command
Reyn1 and Reyn2 · two level-5 CPUs
Avoiding friendly fire
Separate commands let us direct each Fox and name an opponent to pursue. To build toward a partner for a person, we also had to reduce how often it hit its teammate. We built on the 2v2 policy, but adding a friendly-fire penalty made every penalised run passive: the bot attacked less overall.
Without a mask, it dealt 56.2–62.8 % damage per minute to its teammate in real games. In the simulator, a penalised bot beside an imitation-trained human proxy won less than half as often as the plain 2v2 policy beside the same proxies, and the teammate's share of the bot's damage stayed about the same.
We then added an action mask at play time, using Fox's hitbox data to reject a proposed move predicted to hit the teammate.
The bars compare partner configurations against two level-9 CPUs; the filter in their labels is the action mask. Training with the mask added no clear real-game benefit.
Late or moving hitboxes still get through, mainly the up-smash, back-air and Fire Fox. The mask checks only when a move starts.
Bonus: Wombo combos
We also trained a separate pair of bots to keep one opponent in a combo. We count moves that start or continue a linked combo, with follow-ups landing before the opponent can act again. Multiple hits from the same move and recent repeats by the same bot add nothing.
Combo lab
Idle target · normal damage · friendly fire off
The first long combos happened almost in one spot: both bots stood together and attacked. Getting them to pass the opponent back and forth took more work. Stricter passing rules reduced their scores, and reward changes alone did not get the bots to separate. We added a scripted teacher for parts of training, with an imitation loss alongside RL. After a hit, the teacher moved the attacker back to its position, helping the pair learn the handoff. Later teachers helped introduce moves that reward alone was too slow to teach, including a down-air from above. The teachers were used only in training; the bots played every scored game without them.
Mario and Yoshi developed a repeatable routine on Bowser under normal rules in both training seeds. But Bowser was doing nothing. Testing the pairs against opponents that could act exposed the limit. Compare the solid level-9 CPU bars with the inactive-opponent outlines: across all eight routines shown, the median best combo per game falls to half the inactive-opponent result or less. Switching to the scripted escaper shows the same pattern.
Training against an active opponent helped. Two Foxes trained against the escaper reached a median 11–11.5 counted moves per stock against a level-9 CPU they had never trained against. The inactive-opponent baseline retained only 5 of 13. The escaper-trained Foxes still lost many stocks chasing opponents offstage.
Conclusion
We started with a model that could fight and asked whether we could give it instructions, then play alongside it. Reyn now acts on many tested requests during a match. The separate Fox team experiments improved team results, and a play-time action mask reduced friendly fire. A partner's combo exposes the next problem: the policy rarely joins in. Reliable human teamwork remains the goal.
What we learned
- Labelled replay pre-training connected words to actions; RL improved the outcomes of tested moves.
- Comparing a command with the no-command baseline helped us design rewards for the requested behaviour.
- Adapters beside a frozen policy worked for steering. In our 2v2 comparison, training all policy parameters worked better than the adapters alone.
- Multi-agent proximal policy optimization (MAPPO) and independent proximal policy optimization (IPPO) were close in our 2v2 comparison.
- Friendly-fire penalties made our policies passive. A play-time action mask reduced teammate damage more effectively in these experiments.
- Replay review exposed reward hacks that reward totals concealed.
What we failed at
Fire Bird can still miss
Our test scores Fire Bird flights from on-stage starts. In a focused read of Reyn 4C on Final Destination against the level-9 CPU, a scored flight connected in 10 of 32 held-command tests. The clip shows the median hold: it attempts the move repeatedly, but no flight connects.
No command
Reyn (Reyn1 tag) · level-9 CPU (CP)
Off the edge
With the right-ledge request active, Reyn returns to the stage and then falls without another hit. This excerpt is from the other recorded game.
Reyn1: “Don't start any attacks.”
Heard: “Don't start any attacks.”
Reyn1 · CPU (CP)
Gaming the ledge reward
Earlier ledge rewards encouraged dropping off and repeatedly catching the ledge, rather than useful recoveries. Watching replays exposed behaviour the reward totals concealed.
Limitations
Reyn can take commands across all 26 characters, but support is still uneven. Special-move commands remain unreliable for characters such as Mewtwo and Zelda.
Some teamwork, mostly skill
There are some signs of teamwork, but much of the 2v2 gain still comes from each Fox's individual skill. Rewarding hand-offs did not help: both Foxes crowded one opponent.
Three moves still hit you
The up-smash, the back-air and the Fire Fox get past the action mask, which checks a move only when it starts.
Possible future work
- Following up a person's combos remains a goal. The partner rarely joins in when you start a combo, and follow-ups fell short of the bar we set.
- The pair needs to follow commands more reliably against two Faynts. Our hypothesis is that strong opponents leave too little time and space to act on a command.
- Improve uneven command quality across the roster, and bring broader character support into 2v2. The current Reyn 26C release and the Fox team branch remain separate.