Abstract:

For almost its entire existence the spotlight of quality has fallen on the online access panel operator. Online panel companies have both driven the quality agenda and the research on research necessary to truly understand the online panelist, their motivation, behaviour, needs and wants. This quest for quality is by no means over however, but the parameters of quality are becoming ever clearer and a consensus emerging as to what precisely constitutes a 'bad panelist' and 'bad data'. Research companies must work in partnership with panel companies to help weed out those panelists who undermine the research process. At the same time they must recognize that the survey instruments themselves can have a negative impact on survey quality. Unlike traditional research methods there is no interviewer in the online, self-completion, process to clarify an ambiguous question or to probe more deeply into the respondents' answers. In order to clearly demonstrate the impact of poor questionnaire design we undertook 'a tale of two questionnaires'. We put two differently designed questionnaires to matched samples. One questionnaire was chosen as typical of 'worst in breed' and then re-written with the respondent experience in mind using the combined 60 years of research experience of the authors. Both 'schoolboy howlers' and commonly used, but difficult questions were included in the questionnaire. Interviews were conducting during August 2007 using SSI US SurveySpot panel. 508 housewives and househusbands completed the bad survey and 502 the good survey. Both samples were weighted to the population in analysis.