Best Practices for Panel + Big Data

Over the last year it has become clear to most players in audience measurement that the ANA and VAB have stuck by the WFA mandate that big data needs to be used but calibrated by means of a representative probability sample panel of peopleNow that the industry has lined up in a consensus agreement with this principle it appears timely to review the best practices for carrying out this general design This is important because not just any panel will do and the way the two types of data are combined is a pivotal factor in maintaining the inherent stability of big dataAny Measurement System Will Tend to Affect the Data It CollectsHeisenberg was the first February 1927 to bring this fact to light in the context of measuring subatomic phenomena Marketing and media researchers knew about the Hawthorne effect at about the same time 19241932 although the term itself was coined by Henry A Landberger in 1958 The Hawthorne effect states that if people know they are in some sort of study their performance improves This is similar to John Kenneth Galbraitheamp8217s definition of conscience as quotthe awful suspicion that someone is watchingquot He may have been quoting HL MenckenIf one were to try to base crossplatform audience measurement solely using big data the biases introduced into the estimates would result from a number of causes including missing pieces of the audience and artifacts created by algorithms used to make adjustments for those missing pieces This is because big data today are convenience samples which cover some households but not othersMedia Communication Implies PeopleMost of the money which makes media possible comes from advertising Advertisers use media advertising as one of the main ways of increasing the number of customers that buy their brands and by maintaining those people as customers by reminding them to buy the brandsTo base a measurement system solely upon device measures amp8211 which is what all audience big data are amp8211 automatically sets up a procedural bias that puts people somewhere in the background something to be inferred an afterthought something secondary which can easily be reliably estimated which is not the case in reality inferring people data accurately from device data has been proven to be supremely challenging if not impossiblePanel Implies OptInSome of the new big data audience analytics platforms call their sample a panel This is a misuse of the term One can have a big data sample but it will only be a panel if the people in the sample have consciously agreed to be measured Not opting out of device measurement does not constitute conscious agreement to be a panelist If itamp8217s device measurement and not people measurement it is not a panelThe Best Panels are Area Probability Samples of People in HouseholdsProbability samples are the scientific method for achieving representative samples of a given universe eg the US household population Without a probability sample you cannot apply statistical laws and know what your sampling tolerance range is Big data convenience samples cannot be validly treated as if total US population audiences and sampling errors can be calculated although this deters no one from pretending that you canArea probability samples are more inclusive and accurate than random dialing telephony based probability samplesThe Challenges of Executing Panels Properly Have Been Famously UnderEstimatedOwen Charlebois was the man who opened the door to global rollout of the PPM portable passive peoplemeter by first testing it in Canada and then adopting it for the Canadian JIC of which he was CEO Google in its plans to mount panels in all nations brought Owen into Google and he did a great job of setting up many panels around the world for the company Google admitted that it thought panels would be easy and learned the opposite the hard way before bringing in an expert Even with Owen aboard Google eventually backed off from panels as not being their core competency This is the same Google that is barreling ahead in AI and yet finds audience measurement not the childamp8217s play it seemedSize of the Panel Matters and How You CalibrateCalibration tends to cause big data to act like it has the sample size of the calibration panel In my consulting for Nielsen which is ongoing it took a year working with the leadership teams to devise the best way to mitigate against stability loss caused by calibration Nielsen has recently advanced this work by a quantum leap and the new results and the dates of currency productization will soon be announcedNevertheless even with innovative new ways of calibrating big data using probability sample people panels the calibration sample size and design will have impact and so you want the largest possible calibration panel that you can as an industry afford To illustrate note that the confidence range around device data at scale might be in the range of plus or minus 2 but if you calibrate that using a 5000 panel depending on the way you put the data together this range could expand to plus or minus 28Response Rate Also MattersIndividual differences that relate to brand choice crucial to advertisers cannot be accurately predicted based on demographics and media choices the correlations are too low The lower the response rate the more likely that the quotcorrectionquot by weighting is not really correcting all the results properlyThe MRC finds that the highest levels of response rate achievable by a panel nowadays are around 30 That means the Original Predesignated Homes in the panel represent about 30 of the panel The other 70 are systematic substitutions in a properly executed and MRCaccredited scientific pattern within an area probability sampleIf you are going to build a calibration panel budget for the number of callbacks and the value of monetary incentives needed to come as close to 30 as possible And use the most trained team possible because each panelist signon is about as hard as getting a voteCollect PeopleLevel Data in Your Calibration Panel and Cover All of the Use of the Media StudiedA calibration panel ought not be a subset of the big data sample and we say this mainly to ensure the sampling quality and collection of proprietary meter data rather than using set top box andor smart TV data in those homes The reason for this is that there are errors in identifying programs ads and even networks in smart TV data and set top box data are sometimes inaccurately coded for program source The calibration panel must have an MRCgrade metering system at the people level so that by use of the common device method Nielsen gave it that name the calibration panel can also bring the program crediting up to the level of watermark accuracy the highest accuracy metering componentThe more obvious reason for calibration panels is that certain types of users may not be represented at all in a big data sample The number one reason for using area probability calibration people panels is to keep all the fragmented segments of the media in the proper relation to one another even as they change from day to dayUnless the people metering in the calibration panel has widelyaccepted watermarks measures linear CTV streaming antenna all devices connected to TV sets computers mobile and out of home the full benefits of calibration will not be delivered Only a patchwork of calibration is better than none but the purpose of this article is to urge going all the way Otherwise why do it in the first placePosted at MediaVillage through the Thought Leadership selfpublishing platformClick the social buttons to share this story with colleagues and friendsThe opinions expressed here are the author39s views and do not necessarily represent the views of MediaVillageorgMyersBizNet

SUBSCRIBE FOR FREE MEDIAVILLAGE CONTENT | BECOME A THOUGHT LEADERSHIP CONTRIBUTOR

Best Practices for Panel + Big Data

Author

Publish date

Channel

Archived in

Bill Harvey

SUBSCRIBE FOR FREE MEDIAVILLAGE CONTENT | BECOME A THOUGHT LEADERSHIP CONTRIBUTOR

Best Practices for Panel + Big Data

Enjoying This Commentary? There's More to Love

Bill Harvey

TAGS:

Cookies