Amazon says its AI interview scoring matched experts in 89.3% of reviewed decisions
The Connect Talent evaluation measures agreement on behavioral evidence, not hiring outcomes. Its design separates the interviewer from the scoring process.
Loading page…
The Connect Talent evaluation measures agreement on behavioral evidence, not hiring outcomes. Its design separates the interviewer from the scoring process.
Listen to this story
Amazon’s November 19 account of AWS Connect Talent reports that its scoring process matched independent psychologists on 89.3% of 252 behavioral signals across 36 interviews, with Cohen’s kappa of 0.79. The result tests consistency in interpreting interview evidence—not whether the system selects better hires or predicts job performance. For teams evaluating AI hiring tools, the notable design choice is that the interviewer gathers evidence while a separate process scores transcript-based behaviors, flags insufficient evidence, and leaves scorecards editable and reviewable by hiring teams.
Agreement rose to 97.6% when candidates gave clear evidence and was 92.9% for signals indicating potential poor role fit.
Separate content validation found 91% of competencies traceable to job descriptions, 98% of questions targeting intended competencies, and 94% of rubric signals measuring them.
Using only first responses lowered scores by as much as 1.14 points on a five-point scale; full-conversation assessments tracked experts more closely.
Amazon is making a case for AI-led hiring interviews built around evidence, not overall impressions. In findings published November 19, AWS said Connect Talent agreed with independent experts on 89.3% of behavioral scoring decisions. That result covers 252 rubric signals from 36 interviews—a test of scoring agreement, not proof of better hiring outcomes.
An independent industrial-organizational psychology firm reviewed the interviews for specific strengths and concerns in candidates’ answers. Connect Talent assessed the same transcripts. The comparison concerned individual rubric signals—the behaviors used to support a rating—rather than whether experts and AI would hire the same applicants. AWS does not name the firm, and the findings are presented in Amazon’s own research account.
Agreement was higher when candidates supplied clear evidence. AWS also reported a Cohen’s kappa of 0.79, a measure of agreement that accounts for chance. The company described that as substantial agreement with expert scoring. Those measures address consistency with reviewers; they are not measurements of how candidates subsequently performed at work.
Amazon also commissioned an independent review of generated interview content before human editing. Psychologists found that 91% of generated competencies were clearly traceable to job descriptions, 98% of questions targeted their intended competencies, and 94% of rubric signals clearly measured the intended competency. This tested whether the interview material stayed focused on the job, a separate question from scoring candidates’ answers.
Amazon’s reported agreement with expert decisions across the reviewed behavioral signals.
Reported agreement when candidates provided clear evidence.
Reported agreement on behaviors that could indicate a poor fit for the role.
Connect Talent builds interviews from job requirements and occupational frameworks, using scorecards that define specific strengths and concerns. Its AI interviewer gathers evidence through questions and adaptive follow-ups, but does not score the candidate. A separate evaluation process identifies demonstrated behaviors and cites the transcript passages supporting each finding, according to Amazon’s description of the design.
A fixed formula converts those findings into a competency score from one to five. When the transcript lacks enough evidence, the service flags the competency instead of assigning a score. Amazon says missing evidence does not automatically count against a candidate. That creates a distinction between an answer showing a concern and an answer that simply leaves something unresolved.
In a preliminary evaluation, Amazon compared assessments using only a candidate’s first response with assessments using the full conversation. First-response scores were up to 1.14 points lower on the five-point scale. Full-conversation assessments also came closer to expert evaluations. Without follow-ups, the scoring process identified fewer strengths and more concerns; some initial answers lacked details needed to demonstrate the relevant behavior.
The company also compared its structured approach with asking a language model to give an overall rating. Across hundreds of expert-rated interview assessments, Connect Talent used all five points of the scale, while the overall-rating approach concentrated scores in the middle three. That comparison shows a difference in score distribution, alongside the separate test of agreement with experts.
Hiring teams can review, select and edit the competencies, scorecards and questions before using them. Afterward, they can inspect the evidence behind ratings—including where the interview did not gather enough information to produce one.
Loading discussion...
Join the conversation
Explain where helpful prompting ends and coaching begins.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.