AI speeds cancer side-effect review in study, but severity grading remains difficult

Cancer research staff reviewed treatment side effects faster and agreed more often when AI supplied structured suggestions and supporting evidence. A study published October 29 in Nature Cancer found that assistance saved an estimated 2.71 minutes per note. But assigning exact severity remained difficult, and performance was weaker when the system encountered new clinical documentation.

Jack Gallifant and colleagues tackled a documentation bottleneck: immune-related adverse events from cancer immunotherapy often appear in free-text notes rather than readily searchable health-record fields. Manual review is laborious and inconsistent. Their study followed an AI system through retrospective testing, three months of silent deployment and a randomized crossover evaluation with research staff.

Breaking a clinical note into smaller judgments

The system assessed six toxicities: pneumonitis, hepatitis, colitis, myocarditis, thyroiditis and dermatitis. It extracted whether an event was present, current or historical, its severity, whether the clinician attributed it to immune checkpoint inhibitor therapy, and the certainty of that attribution.

Rather than asking one model to settle everything at once, the researchers split the work into specialized stages. Detection and grading each ran three times independently, with a judge agent resolving disagreements. Later stages assessed attribution and certainty. The output paired structured labels with supporting passages from the note, giving reviewers evidence to inspect.

Development used 26 expert-annotated notes, followed by testing on 263 held-out notes. GPT-4.1-mini in the agentic configuration performed best overall, with estimated inference costs of $0.021 per note. Repeated runs mattered: removing that self-consistency step reduced detection F1 from 0.92 to 0.78, while lowering cost to about $0.011 per note.

Severity and new documentation expose the limits

In retrospective testing, the system scored 0.92 on F1 for detecting current events, but only 0.66 for exact severity grading. Severity followed CTCAE, a standardized adverse-event grading system. Intermediate grades 2 and 3 caused most errors; simpler divisions between lower and higher grades performed better.

A diagnosis alone may not supply enough information to assign severity. Grading colitis can require stool-frequency counts; hepatitis can require laboratory values and trends. Those details may be missing from a single note, creating a problem for both the model and human reviewers.

The fixed system then processed 884 notes in silent deployment from May through July 2025. Across severity, attribution and certainty labels, binary F1 was 0.72 for current events and 0.79 for past events. This phase encountered a broader case mix and changed documentation practices. The authors argue for continuous monitoring and periodic recalibration, not one-time validation.

More agreement is not the same as correct grading

The crossover study involved 16 clinical research coordinators and one research nurse reviewing 20 unique notes. As OncoDaily detailed, complete-match accuracy required every current-event grade across all six toxicities in a note to match the reference standard. That demanding measure rose from 19.4% with manual review to 25.9% with AI assistance.

Reviewer agreement improved sharply: Krippendorff’s alpha, an agreement measure, rose from 0.22–0.51 without assistance to 0.82–0.85 with it. Yet the study did not determine whether some convergence reflected reviewers anchoring on AI suggestions. The authors identify that risk as a question for future investigation. No phase evaluated patient outcomes, so faster, more consistent review does not establish safer treatment.