Many studies comparing the accuracy of survey estimates generated from probability-samples and non-probability samples have been undertaken over the last 15 years. This study (the Australian Comparative Study of Survey Methods – ACSSM) is one of only a few to build upon a previous study, thereby enabling not only point-in-time comparisons of the relative accuracy of estimates generated from probability and non-probability sample surveys, but also the relativity of these comparisons over time.

The ACSSM compares the results from eight parallel surveys of the residential Australian adult population. The survey methods used are (1) computer-assisted telephone interviewing (CATI) with persons contactable via randomly generated mobile phone numbers, (2) mixed-mode (computer-assisted web interviewing [CAWI] and CATI) interviews via a probability-based online panel, (3) video-assisted live interviewing (VALI) via a probability-based online panel, (4) using SMS push-to-web to obtain questionnaires from a random sample of mobile phone numbers, and (5–8) four samples provided by four non-probability online panels.

We find that non-probability online panel surveys are cheaper, quicker, and generally less accurate, but sometimes only slightly so, than the probability-based alternatives. Within the limitations of this comparative analysis, there is also evidence that the accuracy gap in favour of probability-based sample surveys over non-probability online panel surveys may have narrowed in recent years.