Not long ago, I had the opportunity to participate in a medical research project. In broad terms, the researchers developed a virtual tool to evaluate the skills of doctors on a particular procedure without performing on a real patient, and they needed people at various stages of proficiency to test the training program. Since I was a total novice, it made me an ideal subject – I was expected to stumble and burn. In fact, I was so clueless that I had to ask the experimenter to repeat the instructions for the simulation. Then, through either sheer luck or innate talent (ha), I scored near the top of the chart.
Shortly after the study concluded, I was notified that after discussing with the co-researchers, the research team has decided to discard my data-point because “the instructions were given twice, which gave an unfair advantage over the other participants.” I wanted to reply, “But if a complete novice can score like this without knowing how to do the actual procedure, doesn’t that say something about the quality of the virtual evaluation?”
More interestingly, if I had scored much lower than the average novice – making the results look even better – would the research team have thrown out my data-point all the same? Continue reading