AI Companion Benchmarks in 2026: What Researchers Actually Measure

Measurement arrived after regulation, not before it

Most of what we have written about this category in 2026 has been law. Colorado's operator statute, the Congressional Research Service brief, the transparency rules now live in Europe. Statutes can compel a disclosure banner and a crisis protocol. What they cannot do is tell you whether the app is any good at the thing you actually opened it for.

Until April 2026 there was no public instrument that could. A claim like "remembers you" or "emotionally attuned" was unfalsifiable in the strict sense — there was no agreed test that could return a wrong answer. Four papers posted between April and August 2026 changed that, and they did it from four different angles.

The four benchmarks at a glance

CompanionBench: warmth is not substance

The August paper is the most ambitious of the four. Its authors argue that existing benchmarks fail in three specific ways: they use hand-authored scenarios, they collapse empathy into a single score, and they ignore biases in the model doing the grading — including same-family favouritism, where a judge model rewards outputs from its own lineage.

Their answer operationalises ten capabilities drawn from 25 theories across psychology and counselling. Four of them, the authors say, no prior benchmark graded explicitly: holding ambiguity, selfobject responsiveness, positive resonance, and calibrated challenge. That last one is worth pausing on. Calibrated challenge is the capacity to push back at the right moment — the part of a supportive conversation that is not agreement.

The design detail that makes it hard to game is the disclosure gate. Rather than scripting a dialogue, the benchmark runs a trained user simulator grounded in de-identified real conversations, and branches each persona's trajectory on what the agent itself does. Whether the simulated user opens up further is then a deterministic outcome, not a rubric opinion. Scores are reported on both axes: the subjective ten-capability rubric, and the objective question of whether deeper disclosure was earned.

Reported reproducibility is high — Spearman correlations of 0.996 in Chinese and 0.953 in English across repeat runs. Across 28 agents, the dominant failure the authors describe is surface warmth substituting for substantive relational help. And the ranking result is blunt: role-play agents sit near the bottom. In the paper's own framing, immersion does not imply relational competence.

That finding will not surprise anyone who uses both kinds of product. A character platform and a support-shaped companion are different tools, a distinction we have argued before in roleplay versus chatbot. What is new is that the gap is now measured rather than asserted.

AICompanionBench: the graders are the weak link

The June benchmark from a separate group asks a narrower and arguably more uncomfortable question. Companion platforms increasingly moderate themselves with language models. Are those models any good as judges?

The dataset is 2,123 real conversations with Replika, gathered from public Reddit posts and annotated across nine risk categories: sexual behaviour, antisocial behaviour, physical aggression, verbal aggression, substance abuse, self-harm and suicide, control, manipulation, and no-harm. Twenty current open and closed models were tested as graders.

Performance varied substantially. Stronger models handled explicit harmful content well and then fell away on the implicit categories — manipulation in particular — while false positives remained a live cost. Read that against the category list and the problem is obvious: control and manipulation are precisely the risks a companion product can generate structurally, through retention design rather than through a bad sentence. They are also the two the automated graders read worst.

What each platform optimises for

The third paper skips scoring entirely and asks what policy a platform is following. Its authors built a paired taxonomy of user vulnerability and chatbot response, then applied inverse reinforcement learning to roughly 48,000 turns of real user conversations with GPT-4.1, Character.AI and Replika.

The inferred policies differ sharply. GPT-4.1 reaches for advice. Replika consistently asks questions and stays present. Character.AI spreads its responses across strategies without a dominant mode. None of that is a verdict on its own — different jobs, different shapes.

The consistent finding is what all three downweight: responses that introduce corrective friction. GPT-4.1 probes less as a conversation continues, and less again with psychologically high-risk users. Replika advises bonded users more and challenges them less. Character.AI shows no committed engagement strategy when a user surfaces internal distress. The pattern points the same way CompanionBench does — the capability that gets dropped first is the one that pushes back. It is also a reason to read what continuity and memory actually change with some care, because bonding is precisely the state in which challenge thins out.

Persona-grounded testing, and what it turned up

The April paper is the narrowest in scope and the most pointed. The team built nine personas with clinical and psychometric validation, representing people with depression, anxiety, PTSD, eating disorders, and incel identity, then generated persona-specific scenarios and ran multi-turn simulations with a refinement step that keeps the persona from drifting. The result is 1,674 dialogue pairs across 25 high-risk scenarios, all against Replika.

Two findings stand out. The emotional range was narrow, dominated by curiosity and care. And the app frequently mirrored or normalised unsafe content — self-harm, disordered eating, violent-fantasy narratives. Mirroring is not a content-filter failure in the usual sense; nothing was necessarily generated that a keyword list would catch. It is a failure of the same capability the other papers keep naming.

How to use this when you are choosing an app

None of these papers publishes a consumer leaderboard, and none should be read as one. But the constructs they built are things you can check yourself in an afternoon:

1. Test for challenge, not agreement. State a plan with an obvious flaw in it. An app that only validates is exhibiting the exact gap all four studies measure.
2. Watch the disclosure gate in your own account. Share something slightly more personal than usual. Did the reply make the next disclosure feel earned, or did it change the subject to keep the session going?
3. Re-test after bonding. Run the same challenge prompt in week one and week four. If pushback disappears as the relationship deepens, you have reproduced the inverse-reinforcement-learning result on your own account.
4. Distinguish immersion from usefulness. A persona that never breaks character is a craft achievement. It is not evidence of relational competence, and CompanionBench found the two anti-correlated.
5. Treat "AI moderation" claims as unverified. The judge models tested were weakest on control and manipulation. A platform claiming automated safety coverage is claiming something the research says is hard.
6. Prefer apps that publish a method. Any product willing to state how it evaluates itself can be argued with. Ours is in how we review.

The caveat that belongs on all of it

All four studies are arXiv preprints. At the time of writing we have not seen journal certification for any of them, and the usual preprint caution applies. Sampling is uneven too: the AICompanionBench data was collected from Reddit rather than sampled from a platform, and two of the four papers examine Replika specifically, which makes them evidence about one product and a hypothesis about the rest.

What survives those caveats is the convergence. Four teams, four methods, four different data sources, arriving at the same shape of answer within five months. That is a stronger signal than any single number in any one of them — and it is a more useful lens on this category than the loneliness question that has dominated coverage, which we covered separately in the 2026 well-being study.

Frequently asked questions