Online Panel Sample Quality

Panel companies do their best to ensure that panels have the best quality survey respondents they can.
Quality measures for many panel companies, Opini panels and the panels provided via Online Panel Access include:
  • Quarantine rules so that survey participants do not complete too many surveys in a given period
  • Surveys are checked to avoid language and logic issues
  • De-duplication controls to ensure surveys are not completed twice
  • Active quality measures to identify and remove poor quality respondents
  • Random and stratified sampling to survey all people and not just high responders
  • Identification measures to validate respondents
And yet, there are times when some individual survey participants have not completed a survey as well as they might have. Different researchers and online research companies have different approaches towards panel quality checks. The approach may vary by survey, and there is no one size fits all standard approach that is optimal or better than other.

Three Quick Market Research Panel Data Quality Checks

The following is an example of a quality test the team at Surveygoo apply to panel surveys on behalf of clients. There are variations on the tests, but this offers a typical approach. It is based on a Three Stage Quality Filter:
  • Country Location
  • Speed Test
  • Trap Question
Stacks Image 12223
Quality Filter 1: Country Location Test
Although panel companies take measures to limit the number of people who join a panel outside the country the panel is based, a minority of panel members may be resident outside the country. There is a judgement call to be made about whether some panelists have completed a survey while on holiday (this could be the case during holiday periods). But if there are too many respondents completing a survey outside the country there is a real danger of poor quality responses (e.g. some of the survey responses have not been completed by nationals who are relevant to the survey). Fortunately, we can identify the location of panel respondents.

In Surveygoo survey tool, we can run an export of Survey Responses Data as a CSV file. More on How to Export Survey Responses Data file.

The Survey Responses Data file includes for each respondent (S.No). the Country, City and Region,  which is taken automatically from the survey respondent's IP address.

Any cases appearing outside the intended target country can be flagged in the Survey Responses Data file. Using this information on the individual respondent Ids, the case data can be removed from the data set. See How to Article on Removing Case Data.

Typically, failure to meet the minimum requirments of the first quality filter, the Country Location test, is often immediate grounds to remove the respondent(s) from the data set.
Quality Filter Test 2: Speed Test
The second stage is the Speed Test. It is common for online survey companies to look out for so called Speeders - respondents who complete a survey too fast.

The Survey Responses Data file includes for each respondent (S.No) the Start Time and Finish Time of the survey. It also automatically calculates the Survey Duration.
There is no standard way to differentiate a speeder from a non speeder. At the margins it must be recognized that respondents have different response times to surveys - some are able to fill out a survey more quicky than others without compromising on the quality.

So how do we differentiate speeders? Here is an approach we take at Surveygoo. First remove outliers. For example if the survey duration was estimated at 5 minutes but a respondent registered 2 hours, it should be removed from the calculation. Next, calculate the Median time of the survey duration. Excel can do this quickly using the appropriate formula.

Then assign cases by comparing their duration against the Median score. We can flag cases as follows:
• Red (10% or less than the Median)
• Amber (11-40% of the Median)
• Green (<41-70% of the Median)
• Gold (<71-100% + of the Median)

Red cases should be flagged to be removed immediately. Gold cases are closest to the Median and should pass immediately; it is also likely that most Green cases should pass. It may also be that Amber cases will have a mix of cases we want to keep and remove. But as mentioned earlier, we should not make premature judgements about the respondents who are relatively faster. We can calibrate the faster respondents against at least one other quality check.
Quality Filter Test 3: Trap Question
One flexible quality check is the so called Trap Question. Unlike the two previous tests, where the measure can be taken from data points automatically collected in the survey data, a trap question is a manual intervention. It needs to be asked as a component of the questionnaire. It is also a non standard question. E.g. there are many different ways we can ask a trap question. The type we recommend that makes less of an impact on programming and survey duration is to simply include an additional code in a Matrix or Drag and Drop question. For example, a code along the lines of "Please Select Option Don't Know". If you don't have a matrix question you could include an open end question and provide an instruction such as "Please Type the word Attention in Box 3". Alternatively, if you have asked a legitimate open ended question, you can flag up respondents who have written rubbish (e.g. aodhgvcaobsdf instead of a proper answer). 
Again, it doesn't mean that someone who has speeded through some questions, failed a Trap question and entered rubbish in a single open ended question should be automatically removed. But a combination of failed tests against core quality filters should act as an alert and make the case for removing these respondents from the data set.
Compute a Red Flag Score for all the quality checks undertaken against each of the Respondent Ids. Take a copy of this output. It is recommended that the cases to be removed from the data set are agreed with Surveygoo or the panel company before proceeding to delete them.

One final thought. Online panels are not perfect but most market research panel providers genuinely want to offer the best quality panel sample they can. Afterall, their reputation is based on delivering quality. But we should all be aware of what panels can achieve, and their limitations. Some panel companies take more efforts than others to mitigate against poor quality panelists. Finally, online survey companies like Surveygoo can take reasonable steps to look out for suspect data. The key to success is working collaboratively with clients and panel companies to get the best result in the time and budget available.

Surveygoo Books Available on Amazon and iBooks Store

Stacks Image 33184
Visit Amazon to buy | More info on Book
Stacks Image 33189
Visit iBooks Store to download Free

Start Using Our Survey Tool Now!

See Price Plans