LLMScholarBench Tests 22 Models and Finds an Accuracy-Representation Trade-Off

An AI-generated list of experts can become more factually reliable while becoming less representative. Complexity Science Hub researchers found that retrieval-augmented generation improved the technical quality of scholar recommendations, particularly factual accuracy, while prompt instructions could improve representation; neither approach, alone or combined, improved every measured dimension at once.

That finding sits behind LLMScholarBench, a benchmark for auditing language models when they recommend academics. The team applied it to 22 models—20 open-weight and two proprietary—from families including Gemini, GPT, Llama, Qwen, DeepSeek, Grok, Mistral and Gemma. The researchers say omissions can carry real consequences when chatbots are used to identify speakers, supervisors or potential hires.

Measuring more than whether a name is real

The benchmark uses nine measures across technical quality and social representation. Its technical side covers factuality, validity, duplicate answers, consistency and refusals; its representation side covers connectedness, bibliometric similarity, diversity and parity. Parity asks whether recommended scholars track group proportions in the reference database.

In an initial physics evaluation of six models, scholars were placed in the wrong subfield about 40% of the time on average. Across models, recommended names matched real scientists in the reference database in roughly 63% to 82% of cases; seniority and subfield judgments were less accurate.

Representation scores

Diversity

0.440.69

Diversity scores ranged from 0.44 to 0.69 across the evaluated models.

Paritypercent

54%60%

Parity scores ranged from 54% to 60%; the measure compares recommendation shares with group proportions in the reference database.

Interventions move different parts of the system

The trade-off came from tested user-controlled interventions, not a finding that every model inherently improves one dimension at another’s expense. Retrieval-augmented generation has the model gather web information rather than rely only on training data, and it improved technical quality. Prompt engineering steered outputs toward the requested social representation. Combining them still did not improve all metrics simultaneously.

Tests of Gemini 2.5 Pro and Gemini 2.5 Flash with web-search retrieval made the constraint more concrete: factual accuracy improved, while diversity and parity fell. A project researcher attributed that result to gaps in web coverage of scholarly communities, which retrieval can import rather than correct.

The prompt can alter who appears

The models tended to favor male, senior, highly cited and U.S.-based scholars. They often recommended fewer women than women’s share of the reference database, while Black and Latino researchers were often absent. In a study across six disciplines, a user’s geographic location changed the scholars recommended; prompt language and specified role did not.

LLMScholarBench is an open audit framework rather than a final ranking of the model market, and some tested models have since changed. Its first study used a database of more than 450,000 scientists who published in American Physical Society journals between 1893 and 2020. That scope makes the benchmark a structured way to test scholar recommendations, not proof that every expert-search task behaves identically.