Questionnaires in the behavioral and organizational sciences tend to be lengthy: survey measures comprising hundreds of items are the norm rather than the exception. However, literature suggests that the longer a questionnaire takes, the higher the probability that participants lose interest and start responding carelessly. Consequently, in long surveys a large number of participants may engage in careless responding, posing a major threat to internal validity. We propose a novel method for identifying the onset of careless responding (or an absence thereof) for each participant. It is based on combined measurements of multiple dimensions in which carelessness may manifest, such as inconsistency and invariability. Since a structural break in either dimension is potentially indicative of carelessness, the proposed method searches for evidence for changepoints along the combined measurements. It is highly flexible, based on machine learning, and provides statistical guarantees on its performance. An empirical application on data from a seminal study on the incidence of careless responding reveals that the reported incidence has likely been substantially underestimated due to the presence of respondents that were careless for only parts of the questionnaire. In simulation experiments, we find that the proposed method achieves high reliability in correctly identifying carelessness onset, discriminates well between careless and attentive respondents, and captures a variety of careless response types, even when a large number of careless respondents are present. Furthermore, we provide freely available open source software to enhance accessibility and facilitate adoption by empirical researchers.
翻译:暂无翻译