Study reveals AI's ability to estimate individual knowledge from chat logs
A new study from arXiv examines whether large language models (LLMs) can estimate an individual's domain knowledge based on communication logs from platforms such as Slack. The results show varying levels of accuracy across different models.
What happened?
Researchers have investigated whether large language models can measure individual domain knowledge by analysing communication logs from Slack. The study encompassed 27,188 messages from 43 users and evaluated seven different LLMs, including versions of Gemini, Claude, and GPT. The models' estimates were compared against self-reported expertise from 27 participants.
Key facts
| Publiceringsdatum | 27 maj 2026 |
|---|---|
| Antal meddelanden analyserade | 27 188 |
| Antal deltagare | 43 (användare), 27 (självrapportering) |
| Antal utvärderade LLM-modeller | 7 |
| Bästa modell (MAE) | Gemini 2.5 Flash (21,13%) |
”Gemini 2.5 Flash achieved the lowest error (MAE 21.13%), while GPT models showed significantly larger discrepancies.”
Why it matters
A lack of clarity regarding 'who knows what' within organisations often leads to productivity losses. This study addresses the possibility of automatically mapping expertise, which could streamline resource allocation and collaboration. The findings highlight both the potential and current limitations of automating this process.
Who is affected?
The study primarily impacts companies and organisations that use communication platforms extensively, as well as developers of AI tools for knowledge management. Users of these platforms may be indirectly affected as their communication data could be analysed to identify areas of expertise.
What else you should know
The study's estimation accuracy showed a weak dependence on message volume, indicating that more text alone does not necessarily improve inference. This highlights the need for richer and more structured representations of human knowledge to enhance AI analysis.
Quick answers about this story
Vad har hänt?
När hände det?
Varför spelar det roll?
Vilka bolag berörs?
The link opens in a new window and leads to the publisher's own site.
Källan har spårats automatiskt från utgivaren via Aheadlines signalkedja.
Get similar news straight to your inbox
The reader's room
Send in a question or an addition. The newsroom reads everything before it's published and replies when relevant. No AI-generated text – just people.
Sign in to submit a comment or question.
Read the article through your role
- Decide whether this affects strategy over 6–12 months or is just noise.
- Discuss with leadership: do we own the right question or does ownership need to move?
- Ask: what risk are we taking by NOT acting on this this quarter?
Generated angle — not editorial analysis of "Study reveals AI's ability to estimate individual knowledge "