As evermore survey research is undertaken via online panels, including probability-based ones, an issue facing researchers is the possibility that data quality could be changed by panel conditioning effects that may arise from repeatedly interviewing the same respondents over time. This may especially occur when the same survey questions are asked repeatedly of the same panellists over data collection waves.

This paper presents findings from a study investigating the impact of panel conditioning on the reported attitudes of panellists in Australia’s only probability-based online panel – Life in Australia™. The focus of this study is to look for changes in attitudinal reporting over time consistent with the theorised explanations provided by the Cognitive Stimulus Theory (CST) of Sturgis et al. (2009). CST hypothesises that repeated exposure to similar questions may lead to a non-negligible change in attitudes amongst some panellists, which will manifest as a crystallisation of expressed attitudes between the first wave of data collection and subsequent waves. This crystallisation of attitudes is hypothesised to lead, over time, to an increase in response validity and an increase in the inter-wave consistency of reported attitudes.

We find mixed results for the contention that panel conditioning leads to more valid responses being given over time. While we observe a statistically significant increase in the validity of responses for two of the five attitudinal constructs measured, two show no significant change and one shows a statistically significant decline in validity. We also find no clear evidence that panellists’ answers become more consistent as a result of repeatedly responding to the same questions over time. Across a battery of 33-items, panellists are almost as likely to demonstrate less consistency in their responses as more consistency. Even where we do see increases in response consistency this cannot be simply interpreted as support for the CST. We show that other factors such as the innate invariability of some of the attitudes being measured, as well as potentially harmful panel conditioning effects such as inter-wave freezing and satisficing, may also be contributing factors to the level of response invariance that we observe.