UC3M warns AI medical-report rewrites still need human oversight

Making a medical report easier to read can also strip out information patients need. In a November 10 account, Universidad Carlos III de Madrid highlighted errors across four AI model families tested on Spanish clinical texts. Its warning accompanies research published August 11 in Scientific Reports—not a newly released study—and calls for human oversight.

The work addresses a gap in medical-language research: Spanish clinical text has received less attention than English, particularly for systematic plain-language adaptation. The researchers argue that results from English or general-purpose Spanish tests cannot simply be carried over to clinical communication. Preserving medical meaning requires evaluation in the language and setting where a tool would be used.

The team built MEDICLARO, a collection of 50 clinical notes, each paired with three independently written plain-language versions. The notes came from clinical cases supplied by the Andalusian Society of Family and Community Medicine. Cognitive accessibility experts produced the 150 adaptations, giving the researchers human-written references against which to assess the AI-generated texts.

Those experts followed ISO 24495-1:2023, an international plain-language standard. The framework treats accessibility as more than replacing difficult words or shortening sentences. It also covers relevance, structure and guidance for the reader. In a medical document, the task therefore includes organizing information coherently and making relationships explicit—not merely producing a shorter version of the original.

Using three references per note also recognizes that there can be several valid ways to explain the same information. The benchmark compared fine-tuning—additional training to adapt a model to a task—with prompt-based approaches, which guide it through instructions. The original notes and expert adaptations provided the material for training and evaluation.

The tested systems were Llama 3.2 1B and 3B, Salamandra 2B and 7B, RigoChat-v2 7B, and Medical-mT5 3B. Researchers assessed simplification, similarity of meaning, factual consistency, readability and environmental impact. The paper also included human evaluation and qualitative error analysis, rather than relying only on automated measures of how closely an output resembled an expert version.

Two different strengths

01Most balanced profile

Llama-3.2-3B

Researchers found the strongest balance across efficiency, robustness and overall performance.

02Quality and readability

RigoChat-v2-7B

This model stood out when output quality and readability took priority.

The analysis identified three recurring types of failure across all four model families. They affected both the medical content and the clarity of the rewritten text:

• Oversimplification removed important clinical details. A version could become less demanding to read while leaving out information the patient needed to know.
• Clinical hallucinations introduced invented information or changed data from the original report. These were failures to preserve the source, not simply differences in wording.
• Incoherence produced fragmented sentences or abrupt jumps between ideas. That undermined readability—the very quality the adaptation was intended to improve.

Study co-author Paloma Martínez linked the need for human oversight directly to those errors, particularly in healthcare. The benchmark’s findings concern the quality of rewritten documents. Its evaluation targets were linguistic quality, preservation of facts and environmental impact, rather than whether patients subsequently followed treatment instructions more reliably.

The team has made its research materials public so other researchers can reproduce and extend the work. Martínez says the available resources include the dataset, fine-tuned models and prompts, hosted through repositories including GitHub and Zenodo. The paper also releases the benchmarking framework and code for its adapted evaluation measures, providing tools for further comparisons of Spanish clinical-text simplification.