This study directly compares survey data on social attitudes collected from an opt-in sample of voter advice application (VAA) users and a randomly recruited, probability-based online panel of respondents. Whereas much research to date has focused on the demographic representativeness of VAA-generated data, less is known about the attitudinal and other representativeness of these data. The findings from these Australian samples contribute to the emerging international literature.

VAAs are proliferating as a source of ‘big data’ among public opinion and political science researchers, despite concerns about the representativeness of the opt-in samples. During July 2016, VAA developer Election Compass collected email address details for approximately 40 000 Australian users of its application in the weeks before the 2016 Australian federal election. In November 2016, this study surveyed the sample of VAA users on their attitudes towards a range of Australian social issues. In December 2016, the same questionnaire was administered to a probability-based sample, using an identical mode of administration and similar response maximisation techniques. The questionnaire contains a broad range of questions designed to identify dimensions of sociopolitical attitudes in Australian society. Comparing the composition of dimensions and relationships between variables within the data contributes to our understanding of incidental samples such as VAA users, and the extent to which we can and should make inferences from VAA-generated data.

Comparison of point estimates of the unweighted and weighted datasets, and of estimates of the internal relationships between variables within both datasets suggests that VAA user data should not be considered externally valid sources of social attitude data, even after adjustment. Further, researchers and reviewers should use greater caution when adjusting VAA user data on the basis of observable measures (with well-established population parameter estimates) to explain unobservable measures without robust population parameter estimates (such as social attitudes).