It’s not me, it’s you: miscomprehension in surveys

Hardy, B and Ford, L R (2014) It’s not me, it’s you: miscomprehension in surveys. Organizational Research Methods, 17 (2). pp. 138-162. ISSN 1094-4281

Abstract

The ubiquity of surveys in organizational research means that their quality is of paramount importance. Commonly this has been addressed through the use of sophisticated statistical approaches with scant attention paid to item comprehension. Linguistic theory suggests that while everyone may understand an item, they may comprehend it in different ways. We explore this in two studies in which we administered three published scales and asked respondents to indicate what they believed the items meant, and a third study that replicated the results with an additional scale. These demonstrate three forms of miscomprehension: instructional (where instructions are not followed), sentential (where the syntax of a sentence is enriched or depleted as it is interpreted), and lexical (where different meanings of words are deployed). These differences in comprehension are not appreciable using conventional statistical analyses yet can produce significantly different results and cause respondents to tap into different concepts. These results suggest that item interpretation is a significant source of error, which has been hitherto neglected in the organizational literature. We suggest remedies and directions for future research.