Wednesday, October 04, 2006
Sometimes Engagement is Just Not as Good as It should be!
Recently I attended a conference (well actually I was present via telephone at 4 am in the morning - but that is a whole different story). One of the key topics was how to deal with respondents to research who cheat, or who just complete the surveys to get their rewards! In short that means a lack of engagement with the process.
Look, this is not a new issue. Some people try to state that on-line is the worst offender here, but that just doesn't register to the truth. People have always cheated and they always will. That is one of the reasons that we do larger sample sizes and we quarantine respondents for a period of time so that the nett effect is negligible. But what can we do?
A 'cheat' doesn't wear a t-shirt that says cheat. You have to go and find them:
- remove people who answer the survey too quickly from the panel and from the data file
- remove people who try to register to the panel too many times. Look in China I have had the same person try and register over 140 times! When someone boasts about their panel size I don't look much beyong this as my first sobering assessment of their claim. You cannot write a macro to find them - you have to eye-scan them!
- remove people that provide inconsistencies in the survey
As well as these hard measures we also have soft measures:
- everyone terminates at the end of the screening section regardless of which condition they 'failed' and hence cannot take the survey. This prevents respondents learning over time how to pass the screening.
- we do not use qualify or screen in the invitation as it suggests that responding a certain way may get you more points!
I heard someone say recently to set up a database of known cheaters that we can all compare with. What altruism! Do we really think that we can swap personal details of panelists to enable such a mechanism? Who would control such a database and is it open to mis-use?
I guess I see so much written and spoken about this whole 'cheating' issue that I had to say something. Research has been validated, nothing is perfect, but it works and it is a multi-bllion dollar industry.
Complacency, no. Realism and pragmatism, yes.Thank you for reading.
Look, this is not a new issue. Some people try to state that on-line is the worst offender here, but that just doesn't register to the truth. People have always cheated and they always will. That is one of the reasons that we do larger sample sizes and we quarantine respondents for a period of time so that the nett effect is negligible. But what can we do?
A 'cheat' doesn't wear a t-shirt that says cheat. You have to go and find them:
- remove people who answer the survey too quickly from the panel and from the data file
- remove people who try to register to the panel too many times. Look in China I have had the same person try and register over 140 times! When someone boasts about their panel size I don't look much beyong this as my first sobering assessment of their claim. You cannot write a macro to find them - you have to eye-scan them!
- remove people that provide inconsistencies in the survey
As well as these hard measures we also have soft measures:
- everyone terminates at the end of the screening section regardless of which condition they 'failed' and hence cannot take the survey. This prevents respondents learning over time how to pass the screening.
- we do not use qualify or screen in the invitation as it suggests that responding a certain way may get you more points!
I heard someone say recently to set up a database of known cheaters that we can all compare with. What altruism! Do we really think that we can swap personal details of panelists to enable such a mechanism? Who would control such a database and is it open to mis-use?
I guess I see so much written and spoken about this whole 'cheating' issue that I had to say something. Research has been validated, nothing is perfect, but it works and it is a multi-bllion dollar industry.
Complacency, no. Realism and pragmatism, yes.Thank you for reading.
Comments:
<< Home
Hi Jon, my name is Enric Cid, marketing director at Netquest. We are an online panel provider based in Barcelona, Spain. I agree with you on this "new era of permission of online panels". Now respondents are in the driving seats, so we do have to deliver real value in order to engage them.
Post a Comment
# posted by 
: 7:19 pm
: 7:19 pm << Home
