PromptNCE: LLMs estimate mutual information without training
A new method, PromptNCE, enables large language models (LLMs) to zero-shot estimate pointwise mutual information (PMI) with high correlation to human evaluations, without the need for task-specific training.
What happened?
Researchers have developed PromptNCE, a method that allows large language models (LLMs) to estimate pointwise mutual information (PMI) solely based on prompts and elicited probabilities. Normally, a task-specific critic must be trained for this, which is a limitation in low-data situations. PromptNCE employs a contrastive task with an explicit "OTHER" category to recover the true conditional probability P(y|x), transforming a contrastive prompt into a generic zero-shot probability estimator.
Key facts
| Metod | PromptNCE |
|---|---|
| Resultat | Upp till 0,82 Spearman-korrelation med mänsklig PMI |
| Antal dataset | 3 |
”Estimating mutual information from text usually requires training a task-specific critic, which limits its use in low-data settings.”
”We show theoretically that adding OTHER recovers the true conditional P(y | x) rather than just a ranking over listed candidates, turning a contrastive prompt into a general-purpose zero-shot probability estimator.”
”PromptNCE is the best zero-shot method on all three datasets, reaching Spearman correlation up to 0.82 with human-derived PMI.”
Why it matters
PMI measures how much two events (for example, words) are expected to co-occur. The ability to estimate PMI zero-shot with LLMs eliminates the need for extensive datasets and training-specific models, broadening the application area for PMI estimation methods, particularly in domains with limited access to training data. The method achieves up to 0.82 Spearman correlation with human-derived PMI, indicating high accuracy.
Who is affected?
Researchers and developers in natural language processing (NLP) working with low-data resources or requiring rapid estimates of association strength between text units are primary stakeholders. Academic institutions and companies exploring new applications for LLMs also stand to benefit.
What else you should know
The researchers have also introduced a benchmark with human-derived ground-truth data for PMI to evaluate methods in this field, facilitating future research and comparison. This benchmark covers three publicly available datasets.
Quick answers about this story
Vad har hänt?
När hände det?
Varför spelar det roll?
Vilka bolag berörs?
The link opens in a new window and leads to the publisher's own site.
Källan har spårats automatiskt från utgivaren via Aheadlines signalkedja.
AI-verktyg i artikeln
Topics
Get similar news straight to your inbox
The reader's room
Send in a question or an addition. The newsroom reads everything before it's published and replies when relevant. No AI-generated text – just people.
Sign in to submit a comment or question.
Read the article through your role
- Decide whether this affects strategy over 6–12 months or is just noise.
- Discuss with leadership: do we own the right question or does ownership need to move?
- Ask: what risk are we taking by NOT acting on this this quarter?
Generated angle — not editorial analysis of "PromptNCE: LLMs estimate mutual information without training"