Skip to content
Nicole Salem
Back to projects

Graduate assessment memo

Evaluating a Hiring System for Healthcare Implementation Consultants

  • Selection & Assessment

A review of a proposed hiring system, with recommendations to make interview scoring more consistent and decisions near cutoffs easier to defend.

The question

How much confidence should hiring managers place in interview scores when interviewers apply rating standards inconsistently or candidates score close to a cutoff?

My approach

The review, written for SilverLine Health Technologies’ leadership, weighed each tool in the proposed system (a cognitive ability test, a personality inventory, and a structured interview) on reliability, validity, and how measurement error could tip individual hiring decisions.

What the numbers showed

The cognitive test held up; the interview did not. Its interrater reliability of .54 means scores depend heavily on who conducts the interview, and because reliability caps validity, its predictive value stays limited.

0.25.50.751Cognitive: internal consistencyCognitive: internal consistency: 0.84.84Cognitive: test–retestCognitive: test–retest: 0.77.77Interview: interrater agreementInterview: interrater agreement: 0.54.54
Reliability of each selection tool, from the Validity & Criterion Defense Memo. Higher is more consistent.
0.25.50.751Cognitive ability testCognitive ability test: 0.39.39Structured interviewStructured interview: 0.24.24Personality inventoryNo internal validation data
Validity: correlation (r) between each tool and job performance.

Measurement error matters most near the cutoff. With a standard error of 5.1 points, Candidates X and Z overlap almost entirely, and X’s likely range dips below the cutoff of 65, so accepting or rejecting on a two-point gap is hard to defend.

5560657075808590Cutoff 65Candidate XCandidate X: 70 (likely range 64.9 to 75.1)70Candidate ZCandidate Z: 72 (likely range 66.9 to 77.1)72Candidate YCandidate Y: 78 (likely range 72.9 to 83.1)78
Cognitive test scores with a ±1 standard error band (5.1 points). Overlapping bands mean the difference may not be real.

Work sample

Recommendations from the assessment memo

  • Have interviewers score independently before discussing candidates.
  • Clarify behavioral anchors and train interviewers to apply them consistently.
  • Monitor rater agreement and account for measurement error when interpreting scores near cutoffs.
  • Validate the personality inventory and interview within SilverLine’s own workforce, and run adverse impact analyses.

From the executive recommendation in the Validity & Criterion Defense Memo, pages 6–7.

Download Validity & Criterion Defense Memo (PDF)

Related work: legal and ethical risk review

A companion memo reviewed a different hiring process, for Client Services Coordinators at Meridian Health Solutions, and ranked its legal and ethical risks.

High risk

  • No job analysis for the role
  • Assessments used without validation evidence
  • Interview questions about family responsibilities

Moderate risk

  • “Digital natives” recruiting language
  • Automatic screening on employment gaps
  • Unstructured interviews

Low risk

  • Standardized rejection emails
Risk prioritization from the Meridian hiring risk memo, weighing severity, likelihood of challenge, and how well each practice is documented.
Download Meridian hiring risk memo (PDF)

Recommendation or status

Approve the system on the condition of targeted revisions: strengthen the structured interview first, expand internal validation, build measurement error into score interpretation, and document the rationale behind every step.