A Lesson About Surveys (and More) from November 8

We begin by understanding what is at stake when we depend on surveys to give advice to clients and take action on their behalfamp160We risk the money we invest our companyamp8217s reputation our own credibility client relationships and perhaps even our jobsamp160So we start with the measures of risk associated with surveysamp160 Generally speaking they are called confidence intervals or conversely ranges of erroramp160 These measures assume the following about the samples they measure The samples are supposed to be perfectly random yet representative of the population questions are presumed to be perfectly phrased and airtight and respondents are truthfulamp160Nonetheless some samples we use are overstuffed overburdened overblown oversold overused undersampled and imperfectEvery single survey we use has at least one or more of these characteristics but we choose to ignore themamp160I understand whyamp160I understand that it costs too much to solve many of these problems and that our surveys in many cases represent the only game in townamp160 This is not an indictment of research companiesamp160It is however aamp160hard look at how we use their productamp160That said given what happened on Novemberamp1608 I think its time to take a closer look at our understanding of surveysGenerally speaking clients want surveys to reflect reality while agencies that understand the nuances of sampling are content with consistency and the media just want big numbersamp160Maybe we should look more closely at margins of erroramp160The factors that affect the reliability of samples are the quality of the sample truthfulness of respondents market size demo size and theamp160size of the category being consideredEven if all these considerations were perfect sampling error merely tells you that if you went out and resampled 100 times you would get a fairly consistent result within the margin of erroramp160 With that in mind that error levels measure consistency that approximates reality letamp8217s look at a few resultsamp160 We will use a two sigma level of assurance which is another way of saying if you went out and sampled another 100 times 95 times out of 100 you would get a consistent result in the range givenOn broadcast network TV a huge rating of 15 against a big demo like adults 2554 is between 138 162amp160 Thatamp8217s a swing of 12 or a variance of 8 on the original 15 rating amp16012150 8 amp160Not badHowever a 3 rating against a smaller demo like women 1824 varies from 23 37amp160 Thatamp8217s a 27 variance on the 3 rating or amp8211 07 30 27amp160On lower rated cable stations the variance can be more than 75 of the original ratingamp160 In small TV markets with low rated shows the variance can be as much as 99 of the original ratingamp160 Sure these numbers improve with higher rated programs dayparts stations and demos but remember the only thing reflected here is the statistical theory that the sample approximate reality in a perfect worldPut another way samples do not have the veracity of a complete censusamp160You can do this error range exercise with CPMs and CPPsamp160 In a smaller market like Louisville in late fringe for example a cost per point of 45 has an error margin of 45amp160Wowamp160 Remember we programmatically negotiate for penniesOkay I know what critics will sayamp160When you gross up an entire schedule the confidence level increases and ranges of error decreaseamp160 So too when discussing reach and frequencyamp160 When I discussed this with the late Erwin Ephron The Media Guru he liked to say it was amp8220akin to regressing to the meanamp8221amp160 Like when you love a restaurant the first time you go and then revisit only to find that the experience is not quite as goodamp160The experience gets watered downamp160So too it does with large schedules reducing extremesHowever to make matters worse start to think about the small numbers associated with microtargeting and digital initiativesamp160 Frustrating isnamp8217t itSo Trump is PresidentElect and we still use the same ratings systemsamp160 What should we do to avoid the trap of November 8amp160in the way we use samplesTalk openly about confidence levels and what they meanamp160 Get a firm understanding of the risks involved with the survey you are usingamp160 Double the error margin to 95 confidence and add just a bit for the vagaries of reality vs consistencyamp160Donamp8217t be afraid to be anecdotalamp160 Sometimes a story is worth a thousand numbersFind ways to relate survey results to real life consequences like salesamp160 Draw relationships to return on investmentSummarize findings in a readable interesting and digestible form that senior executives will have the patience to read on a busy dayTry to confine your surveys to categories and demos that are sizeable enough to offset major error levelsWhen you encounter smaller segments regroup them to form larger unitsPay careful attention to trendsamp160 They tell you a lotRemember that digital initiatives generally can be expressed in actions that have already taken place thus removing the predictive nature of surveysTry not to make business decisions based solely on results from a sampleamp160 Find other criteria as wellamp160 Try to triangulate your way to a conclusionFind an advisor who knows how to use surveys productively and can explain then to you in simple EnglishThere is no silver bullet on this subjectamp160 Research houses do a wonderful job of trying to reflect reality with the money made available to them by the people who us their service and within the boundaries of sampling scienceamp160 Now it is up to us not to be fooled by the whimsy of chanceClick the social buttons above or below to share this content with your friends and colleaguesThe opinions and points of view expressed in this article are exclusively the views of the author andor subjects and do not necessarily represent the views of MediaVillagecomMyersBizNet Inc management or associated bloggersamp160

APPLY FOR JOBS | SUBMIT YOUR RESUME | REGISTER FOR FREE MEDIAVILLAGE CONTENT

A Lesson About Surveys (and More) from November 8

Author

Publish date

Channel

Steve Fajen

APPLY FOR JOBS | SUBMIT YOUR RESUME | REGISTER FOR FREE MEDIAVILLAGE CONTENT

A Lesson About Surveys (and More) from November 8

Enjoying This Commentary? There's More to Love

Steve Fajen

Cookies