Consensus Conference 4: Validation Study Design

Consensus Conference 4: Validation Study Design
(November 17-18, 2012)

In the fourth FRS consensus conference, called “Validation Study Design”, the design of the FRS validation study was discussed by clinicians, psychologists, researchers and psychometricians. The validation study was designed to meet the most rigorous evaluation that would satisfy criteria for high stakes testing and evaluation.

Areas of discussion included:

  1. Defining the research questions
  2. Developing hypotheses within construct validity
  3. Defining what constitutes an “expert” for benchmarking purposes
  4. Criteria for institutions participating in Phase 4 studies (see study phase description below)
  5. Determining what will be measured
    • Face validity
    • Content validity
    • Construct validity
    • Concurrent validity
    • Predictive validity
    • Inter-rater reliability
    • Test-retest reliability
    • Usability
    • Acceptability
  6. Study design phases
    • Phase 1: Pilot at Florida Hospital Nicholson Center (logistics and refinements to model)
    • Phase 2: Get face and content validity from the society leadership and boards
    • Phase 3: Get face, content, and construct validity at test sites
    • Phase 4a: Get concurrent validity with video correlations
    • Phase 4b: Get predictive validity through the full research study at 10 research sites (ACS/AEI Accredited Institutes)

Graphic representation of validation study design