Study: LLMs generate social comparison but lack ability to detect it
A new study shows that large language models (LLMs) can generate text that evokes social comparison in readers, yet lack the ability to reliably identify these signals in text.
What happened?
Researchers have introduced a new benchmark called XHS-SCoRE (Xiaohongshu Social Comparison Reader Elicitation) to assess whether text-generated posts evoke upward, downward, or neutral social comparison from the reader's perspective. The study utilised prompt-based LLM classifiers and Chinese encoder models to analyse content from the social media platform Xiaohongshu (RedNote). The results show that while LLMs can effectively create content that triggers social comparison, they fail to consistently detect this effect in existing text.
Key facts
| Benchmark namn | XHS-SCoRE |
|---|---|
| Plattform för studie | Xiaohongshu (RedNote) |
| Typ av jämförelse | Uppåtgående, Nedåtgående, Neutral |
”We introduce Xiaohongshu Social Comparison Reader Elicitation (XHS-SCoRE), a reader-grounded benchmark for detecting if a text-only Xiaohongshu (RedNote) post elicits UPWARD, DOWNWARD, or NEUTRAL/no clear social comparison from a first-person reader perspective.”
”Across prompted LLM classifiers and supervised Chinese encoder baselines, we find a consistent mismatch between generation fluency and reliable detection ability: the signal is textually learnable in-domain, but not robustly accessible to prompt-based classification.”
”A controlled pilot further shows that LLM-generated Xiaohongshu-style posts can shift perceived standing and comparison-related affect even w”
Why it matters
This discrepancy between generative capability and detection capability highlights a significant limitation in current LLMs. The ability to generate text that influences the reader's self-perception and emotions is central to the understanding of online social interaction. The fact that LLMs cannot identify such signals could lead to unintended effects when used in social media or other platforms where text can impact user well-being.
Who is affected?
The study primarily affects developers and researchers within AI and machine learning working with LLMs and natural language processing. Users of social media platforms may be indirectly affected by content generated by LLMs, especially if it evokes social comparison without the platform's systems being able to moderate it. Companies using LLMs for content creation or moderation need to take these limitations into account.
Impact on the EU
Not applicable for EU status. The study focuses on technical limitations of LLMs and Xiaohongshu, a Chinese platform.
What else you should know
The controlled pilot study showed that LLM-generated posts in the style of Xiaohongshu, even if not directly detected by the models themselves, can still influence readers' perception of social standing and comparison-related emotions.
Quick answers about this story
Vad har hänt?
När hände det?
Varför spelar det roll?
Vilka bolag berörs?
The link opens in a new window and leads to the publisher's own site.
Källan har spårats automatiskt från utgivaren via Aheadlines signalkedja.
AI-verktyg i artikeln
Topics
Get similar news straight to your inbox
The reader's room
Send in a question or an addition. The newsroom reads everything before it's published and replies when relevant. No AI-generated text – just people.
Sign in to submit a comment or question.
Read the article through your role
- Decide whether this affects strategy over 6–12 months or is just noise.
- Discuss with leadership: do we own the right question or does ownership need to move?
- Ask: what risk are we taking by NOT acting on this this quarter?
Generated angle — not editorial analysis of "Study: LLMs generate social comparison but lack ability to d"