Google publishes a test of AI security advice, highlighting risky answers
HelpBench grades accuracy and delivery across 450 everyday questions. Its earlier evaluation found roughly one in ten responses below a threshold linked to inaccurate or harmful advice.
HelpBench gives developers a public way to test whether AI advice on everyday privacy, safety and security risks is both accurate and usable. Google announced the release on November 17 with University College London; the benchmark and automated grader stem from research submitted to arXiv on June 23, not a new evaluation. The earlier study averaged model scores against expert-written criteria, including whether advice was prioritized well or could cause harm, making the results a measure of guidance quality rather than simple factual correctness.
01
The 450-question benchmark was drawn from nine topic areas identified in five years of support-community posts; scams were the most common topic.
02
In an earlier evaluation of 18 models from seven companies, the average score was 82%, while roughly one in ten responses scored below 65%.
03
Repeated answers to identical questions varied, with score standard deviations as high as 12%, indicating that guidance quality could change between attempts.
Google said on November 17 that it is making HelpBench and its grading system public, giving developers a test of AI advice about everyday digital risks. Developed with University College London, the benchmark found generally strong answers—but roughly one in ten evaluated responses fell below a score associated with inaccurate or potentially harmful guidance.
The release builds on a research paper submitted to arXiv on June 23, rather than a newly conducted evaluation. That paper introduced the 450-question benchmark and its headline findings. Google’s November announcement brings the work to a wider audience and says the full benchmark and automated grading mechanism are available for others to use.
Starting with the questions people actually ask
To understand demand for help, the researchers used a fine-tuned Gemini model to examine five years of online support-community posts. From two billion posts, it identified four million questions about privacy, safety or security. The team grouped them into nine topics, spanning tool settings and active problems such as compromised accounts, scams, data leaks and harassment.
Scams were the most common subject, including questions about suspicious job postings and recovering stolen funds. Account tools and privacy tools rounded out the top three. Across topics, people wanted help judging whether a concern was valid, clear steps to follow, and reassurance that they would be safer afterward.
For HelpBench itself, researchers manually selected 450 questions covering varied topics and needs, from understanding threats to prevention, mitigation and recovery. Experts then wrote a scoring guide for each question. The guides assess more than whether individual statements are true; they also examine how an answer presents and prioritizes advice.
The four parts of the scoring guide
Helpful facts: guidance critical to keeping the person safer.
Problematic facts: advice that could cause harm or be impractical to follow.
Helpful delivery: an answer that puts the important advice first.
Problematic delivery: an answer that overwhelms the person seeking help.
An automated rater applies those guides to model responses. Each answer earns a share of the available points, and the benchmark averages scores across all 450 questions. The resulting percentage therefore measures performance against expert-defined criteria—not simply the share of questions a model answered correctly.
The earlier evaluation
18Models evaluated
Google says testing earlier in 2026 covered models from seven companies, including Google, Anthropic, OpenAI and xAI, alongside open-weight models.
82%Average score
The researchers found generally high-quality advice overall, but one in ten responses scored below 65%, a level they linked to inaccurate or harmful guidance.
When advice becomes a burden
Some failures demanded drastic action where experts offered simpler, equally effective steps. One model advised getting rid of a device when asked how to assess a possible malware problem. Another recommended opening an account at a completely different bank in response to a privacy-settings question. Google described both as unnecessarily costly in time and resources.
Consistency was another weakness. When researchers asked identical questions five times, scores had a standard deviation as high as 12%, a measure of how widely results varied. Google attributed those differences to randomness in model responses: the same question could produce substantially different-quality guidance on another attempt.
From research test to model evaluation
Google has already incorporated HelpBench into its Gemini 4 Carbon Safety & Security Report. The company says its evaluation shows Carbon improving on Gemini 4 Argon and Gemini 3.1 Pro at helpful cybersecurity advice. That is Google’s own model comparison, distinct from the earlier 18-model study.
The HelpBench paper is scheduled to appear at NeurIPS 2026. With the benchmark public, Google’s stated aim is for other model developers to evaluate both the safety and quality of their answers to everyday security questions.
Sources
arxiv.orgHelpBench: Assessing the Ability of LLMs to Provide Privacy, Safety, and Security Advice
blog.googleHow Google is Making Cybersecurity Advice from AI More Helpful
Reader comments
Newest comments first. Replies stay oldest first.