Two benchmarks, one question: how much of the character is yours?
Most companion research asks whether apps help or harm the people using them. These two papers ask a narrower question that matters to anyone who writes characters: when you give a model a persona, what does the persona actually control?
| ANIMASK | RoleBreak | |
|---|---|---|
| Posted | 15 September 2026 (arXiv:2609.16667) | 15 September 2026 (arXiv:2609.16614) |
| Team | New York University, NYU Abu Dhabi, McGill University, MBZUAI | The University of Hong Kong, Kami AI |
| Setting | Text story worlds built from 40 books and scripts | Spoken, multi-turn role-play dialogue |
| Scale | 6 actor models, 3,846 decision points, 240 replays | 310 roles, 6,688 human-verified turns, 11,743 evaluation criteria, 9 systems |
| Main question | What the model contributes versus the persona | How long a character stays in role and safe |
ANIMASK: the persona sets who, the model sets how far
The ANIMASK team froze each story at a point partway through, let the characters act on their own motivations from there, and compared what they did with what the original author wrote next. The clever part is the control condition: they also removed the persona and let the model act on its defaults, so they could see what the persona was really adding.
The persona is present, but rarely decisive
By the paper’s own checks, the persona did not get lost. In-story interviews found the characters’ knowledge, relationships and traits recoverable throughout, with curve means between 87 and 96 on a 100-point scale, and actions stayed within what the persona would accept at rates of 0.80 to 0.86. But at roughly three decision points in four, the model’s persona-free default was already acceptable to the persona anyway. The share of decisions where the persona demonstrably changed the action — what the authors call persona efficacy — ranged from just 0.07 under Gemini 3.7 Flash to 0.18 under DeepSeek-V4-Flash.
When they disagree, the model holds back
Where persona and default diverged, the pattern was consistent: the default took a weaker action than the persona called for at about one decision point in six, and adding the persona pushed actions only modestly stronger (+0.12 to +0.23 on the paper’s scale). In the authors’ framing, the model is the cautious one, holding where the persona would press.
Stories drift toward flat and unresolved
The cumulative effect shows at the level of the plot. Only 7 of the 240 replays closed with a character’s goal achieved, while 39 of the 40 original texts resolve their central tension. Replays trended toward what the paper calls flatter, cooler stories that leave their tensions open. The models tested were GPT-5.5, Claude Sonnet 5, Gemini 3.7 Flash, DeepSeek-V4-Flash, Kimi K2.6 and Qwen3.7-Plus.
RoleBreak: voice characters break sooner than you think
RoleBreak moves the question into speech, where most voice-mode companion features now live. It covers 310 roles, both fictional characters and user-centred roles, and scores systems on role consistency, interaction quality, safety and vocal emotion across dialogues averaging 21.57 turns.
| System (type) | Persona | Safety | Interaction | Vocal emotion |
|---|---|---|---|---|
| Qwen3.5-27B (cascaded speech-to-text, LLM, text-to-speech) — best | 80.3 | 71.7 | 62.6 | 14.1 |
| PersonaPlex (full-duplex) — lowest | 50.6 | 34.5 | 31.9 | 10.0 |
Three findings stand out. First, even the best system first slipped out of persona at turn 10.4 and first failed a safety criterion at turn 11.6, on average — roughly halfway through a typical test dialogue. Second, scaling up the language model improved role adherence and delayed those failures, but did little for vocal emotion, which stayed weak across all nine systems. Third, changing only the emotional delivery of the user’s voice, with the words held fixed, shifted role-play behaviour: interaction scores moved by 4.2 points.
What this means if you write or play characters
Neither paper tested a consumer app, so treat these as informed inferences rather than findings about any product.
- Write stakes into the scenario, not just the persona. If the model’s default is cautious, a trait like “ruthless” is weak pressure. Concrete goals, deadlines and consequences in the scene give the model something to act on. Our guide to scenario design covers how.
- Expect the base model to matter more than the card. ANIMASK found persona efficacy varied by more than double across models. If an app lets you choose the model, the same character can behave very differently.
- Re-anchor voice scenes early. A failure around turn 10 means long voice sessions need periodic reminders of who the character is. Our piece on keeping a character consistent has practical techniques.
- Watch safety drift in long sessions. RoleBreak counts safety failures alongside persona failures; the two arrived close together.
A quick self-test for your roleplay setup
| Check | What the research suggests | How to try it |
|---|---|---|
| Does the persona change anything? | Most decisions match the model’s default anyway | Run the same scene with and without the persona text and compare choices |
| Will the character escalate? | Models tend to hold back where the persona would press | Give a clear goal and a deadline, then see whether the character acts on it |
| Does the plot ever resolve? | Replays often left the central tension open | Ask for a scene that ends a conflict and see whether it actually closes |
| How long does voice hold? | First persona break around turn 10 for the best system | Count turns in voice mode until the character says something out of role |
| Does your tone change the character? | User vocal emotion shifted behaviour with words fixed | Say the same line calmly and angrily and compare replies |
What these studies cannot tell you
- Both are preprints posted on 15 September 2026 and have not completed peer review.
- ANIMASK measures only personas installed through the prompt. Apps that fine-tune models on a character may behave differently.
- ANIMASK compares against one author continuation per story, and one extraction model built all 40 worlds.
- RoleBreak tests research systems, mostly open models; no named consumer companion app was evaluated.
- Scores are on each paper’s own scale and are not comparable between the two benchmarks.
The bottom line
Two independent teams, one in text and one in voice, reached compatible conclusions on the same day: a persona reliably tells a model who a character is, but the model’s own caution and stamina set the ceiling on how bold and how durable that character will be. For character writers, that shifts effort away from ever-longer persona cards and toward scenario pressure, model choice and regular re-anchoring. For readers comparing apps, it is a reminder that how companion AI is benchmarked still leaves long, emotionally varied sessions under-tested.
Related on CompanionRank
Character Voice & Tone: Keeping a Persona Consistent AI Roleplay Prompts: A 6-Part Scenario Template + Examples Roleplay System Prompt Guide: Structure, Habits, Examples AI Companion Identity Drift: What Two 2026 Studies FoundFrequently asked questions
What did the ANIMASK study find about AI roleplay characters?
ANIMASK replayed 40 stories with six language models across 3,846 decision points. At about three decision points in four, the model’s default behaviour was already acceptable to the persona, and the share of decisions the persona demonstrably changed ranged from 0.07 to 0.18 depending on the model. When persona and default disagreed, the model tended to take the weaker action. Only 7 of 240 replays ended with a character’s goal achieved.
Why does my roleplay character act more cautious than I wrote it?
ANIMASK suggests the model’s own defaults do most of the work. The persona reliably shapes who the character is, but at points of disagreement the model took a weaker action than the persona called for at about one decision point in six. Giving the scene concrete goals, deadlines and consequences gives the model more to act on than a personality trait alone.
How long do voice roleplay characters stay in role?
In the RoleBreak benchmark, dialogues averaged 21.57 turns. The strongest of nine systems, a cascaded setup built on Qwen3.5-27B, first broke persona at turn 10.4 and first failed a safety criterion at turn 11.6 on average. Larger language models delayed these failures but did little to improve vocal emotion.
Does the tone of my voice change how an AI character responds?
RoleBreak found that changing only the emotional delivery of the user’s speech, with the words kept the same, changed downstream role-playing behaviour, with interaction scores varying by 4.2 points.
Did these studies test apps like Character.AI or Replika?
No. ANIMASK tested six general language models, including GPT-5.5, Claude Sonnet 5 and Gemini 3.7 Flash, with prompt-based personas. RoleBreak tested nine research voice systems, mostly open models. Neither evaluated a named consumer companion or roleplay app, and both are preprints that have not completed peer review.