Outside expert warns against using IdeaLens to judge individual authorship

An AI detector that examines ideas rather than wording should not judge individual authorship, Cornell Tech computer scientist Nikhil Garg cautioned in Nature’s October 13 reporting. His warning accompanies a proposed use for IdeaLens: study co-author Rishanth Rajendhran says it could help gauge AI involvement in conference and journal submissions.

Garg called the detector impressive, expressing surprise that outlines alone could reveal AI-written arguments. But he warned that it could become inaccurate on material unlike its training data. A pattern associated with AI in one field might represent good human writing in another.

Rajendhran, a University of Maryland computer scientist, described conferences and journals as overwhelmed by artificially generated content. He argued that AI-assisted scientific writing is acceptable only when the core idea remains the researcher’s own. That is his proposed dividing line, not a universal publication policy.

Strip out the sentences, keep the argument

Researchers from the University of Maryland, Google DeepMind and Simon Fraser University developed IdeaLens. Their preprint, submitted to arXiv on October 5, describes a system that reduces documents to outlines. Each item pairs a paraphrased description with its role in the argument, minimizing overlap with the original wording.

The team trained it on one million FineWeb documents carrying human-or-AI labels supplied by Pangram, a detector of AI-written prose. These machine-assigned labels were proxies, not direct records of idea authorship. The researchers acknowledged that noisy labels need attention and larger datasets would be desirable.

Two detectors, two different questions

The clearest distinction emerged when researchers deliberately separated planning from writing. As AI models received increasingly detailed human plans, IdeaLens’s AI-flag rate fell from 95% to 7%. Pangram 4 still flagged 92% of the resulting prose.

The contrast reflects different detection targets, rather than a simple contest to catch the most AI use. IdeaLens aims to distinguish the source of a document’s ideas from whoever supplied its sentences. Human phrasing and human planning are separate contributions in these tests.

On fully human or fully AI documents, researchers reported 91% detection of AI-generated work and a 0.6% false-positive rate on human writing. IdeaLens was less accurate than some conventional detectors in that setting. The team also found weaker detection on text from some newer AI models, requiring further testing.

The authors released models and labeled datasets for further research. The project also lists code at github.com/RishanthRajendhran/IdeaLens and a public demo at ideadetector.ai.