Table 5 Analysis of the exclusion attributed to the variability score and whether this was warranted
From: Beyond reliability: assessing rater competence when using a behavioural marker system
Rater | Score | Interpretation | Previously excluded? | If not, exclusion based on variability score warranted? |
|---|---|---|---|---|
Experienced faculty EF1 | 1.13 | Low variability (halo effect) | No | For the overall poor simulation performance, the rater had awarded only ‘poor’ scores for every element, in contrast with all other raters. Exclusion warranted |
Experienced faculty EF8 | 1.29 | Low variability (halo effect) | Yes — poor agreement with the expert rater |  |
Near-peer NP10 | 1.41 | Low variability (halo effect) | No | This rater gave only ‘poor’ or ‘marginal’ scores to both the overall poor and overall mediocre simulation performances, despite most other raters finding examples of ‘good’ or ‘acceptable’ behaviours. Exclusion warranted |
Near-peer NP1 | 3.80 | High variability | No | This rater awarded six inappropriate ‘poor’ scores to the overall good performance, in contrast with all other raters. Exclusion warranted |
Near-peer NP6 | 3.10 | High variability | Yes — data incomplete and poor agreement with expert rater |  |
Near-peer NP8 | 3.16 | High variability | Yes — data incomplete |  |
Near-peer NP12 | 3.57 | High variability | Yes — data incomplete, poor agreement with expert rater and inability to rank |  |