Read the claim. Check the evidence.

These are fictional scenarios for practice, not claims about particular products or studies. No sign-in is needed. Your choices are not saved or included in Pulse research. This exercise does not measure expertise or changes in ability.

Scenario 1 / Evidence review

A productivity claim without a comparison

A company reports that 40 employees who volunteered to try an AI assistant completed work 25% faster than they did last month. The tasks changed between months, and there was no comparison group. The company says the assistant caused the improvement.

Which interpretation is best supported by this study design?Comparing the same employees before and after is enough to attribute the improvement to the assistant, although it may not apply to other teams.The volunteers completed the newer tasks faster, but the design cannot separate the assistant's effect from task changes and other factors.The missing comparison group only limits how widely the result applies; it does not affect whether the assistant caused the improvement in this team.

Scenario 2 / Evidence review

A model leaderboard with different test conditions

Model A scores 86% on a task set using one attempt per question. Model B scores 91% on the same set using five attempts and a process that selects the best answer. A headline calls B the better model.

What would you need before accepting that headline?Using the same task set makes the scores a comparison of model capability; extra attempts only affect the cost of reaching that capability.A larger task set using the current procedures would settle whether the five-point gap reflects a better model rather than extra attempts.Results under comparable attempt limits, selection rules and resource budgets; the current scores compare different evaluation setups.

Scenario 3 / Evidence review

High accuracy on a rare-event task

In a test set of 1,000 messages, 10 contain a dangerous request. A detector labels every message safe and is reported as 99% accurate. You are considering it as a safety filter.

What matters most about this result for that use?It missed all 10 dangerous requests; the aggregate accuracy hides failure on the cases the filter is meant to catch.Since dangerous requests are rare, getting 99% of the test set right demonstrates that the filter catches the risk expected in deployment.The score supports using it as a first filter, provided a human reviews every message that it flags as dangerous.

Scenario 4 / Evidence review

A real citation supporting a stronger claim

An AI summary says that AI tools improve productivity across occupations. Its cited paper studied 60 experienced developers performing short coding tasks. The paper found a benefit for those tasks; it did not study other occupations.

What is the most defensible edit to the summary?Keep the broad claim with a citation and a sample-size caveat, since a measured productivity benefit is evidence for the effect across occupations.Limit the claim to the developers and tasks studied, and make clear that the result does not establish a benefit across occupations.Replace the measured benefit with 'AI may help some people' because a narrow study cannot support a more specific statement about productivity.

Scenario 5 / Evidence review

A change in a reader poll

In September, 55% of 200 opt-in readers said they trusted AI for work. In October, 65% of 800 opt-in readers chose that answer after the poll was promoted to a different audience. The report says trust among workers rose by 10 percentage points.

How should that change be described?Report a workforce increase with a wider uncertainty range, since October's larger sample compensates for differences in recruitment.Describe the increase as a trend among all readers, since keeping the question wording fixed makes the monthly audiences comparable.The share choosing that answer rose in these reader samples; different recruitment and respondents could explain the change, so a broader workforce trend is not established.

Bring this habit to the next AI headline.

Check what was measured, who was included, how the comparison was run, and what the source actually supports.

Back to Pulse questions ↗

Superpower Daily

Your reading space