A Trick to Increase Sample Size

Why a Convenience Sample Might Be Better Than a Probability Sample

Targeting as it gets more sophisticated tends to tighten the definition of who is a target. In the old days the sex/age groups used were nearly half the population. Current targets average closer to 20% of pop.

Everyone knows the argument against targeting, having heard it many times. Some people might never buy but on the other hand they always might. The probability is not zero. If you don’t give them a chance by hitting them with some ad impressions, how can you be sure? Plus the aspirational argument: That Joe may never buy a Cadillac but as long as he aspires to one, he can influence others who have more money.

As programmatic comes in and makes it easier to do extra math during the process, don’t forget this trick, which both optimizes the target selection for maximum ROI and increases the sample size. Simply place a value weight on everyone in the population that is much higher among the ROI Driving Segment (a TiVo Research/TRA term for the segment that is most sales responsive to the creative/media mix the brand is using at the time), but not zero even among non-category purchasers. The weight is derivable by singlesource analytics and reflects the projected incremental dollars that advertising can extract from each household or person. Because this approach uses the entire sample, the effective sample size is 5X higher than in present practice where typically only a fifth of the sample is being used. This shrinks the tolerance range around estimates down to less than half the range it was before.

Probability samples have been the preferred way to do research in the U.S. since the 30s, when some political polls using the quota method, by their dramatic miscalls, exposed the importance of probability sampling. Set top box and cookie/device ID samples are a third type of sample known as convenience samples. You can confidently project a probability sample to the population it measures because you know the probability that each respondent had of being selected, something you can only estimate in the case of convenience and quota samples.

Or can you confidently project a probability sample? You could, if there were not the classic three types of errors that make even probability samples imperfect. The first is Nonresponse bias: The errors that creep in because not every household/person you select will give usable intab data. Next is Response bias: The way the human mind can make up plausible answers that may have no bearing on the psychological processes leading to behavior. Also, laziness in button pushing, corner cutting, simplification in diary entries, prestige bias, forgetting, interviewer bias, fatigue bias, sequence bias and so on. All of these fall under Response bias. Finally, Sampling error: Simply the greater range of the estimates you are able to make. For example, with a smaller sample, a 1 rating could actually be anything between 0.2 and 1.8, whereas with a larger sample, it could be anything from a 0.98 to a 1.02. The latter is the 95% confidence range around ratings based on a set top box sample or cookie sample of 1,000,000, for example.

So as we move into the uncertain future, our drummed-in doctrine favoring probability sampling must be balanced against the sample size advantage of convenience samples where the passivity of data collection removes 99% of Response error and the low opt-out rate removes 99% of Nonresponse error.

Since we have lost our truth standard, the telephone coincidental, due to answering machines and changed mores, we cannot simply do a three-way comparison between the truth standard, the probability sample and the passive massive convenience sample. If we could still do that we could see which of the latter two methods came closest to the truth standard. We could make simulated buys to see how different they were based on the three methods and pick the method that came closest to the truth standard. Alas, that game is no more. We have no truth standard.

But is that true? TiVo Research/TRA, CBS, and Simulmedia, as well as every direct marketer on the planet and many digital marketers, have made up their own truth standard: Whatever gets the highest ROI. Meaning, every brand can test which plan produces higher ROI, the plan devised using probability sample vs. the plan devised using the right big data. A/B testing would be the best way to do this although brands often test by changing tactics from one quarter to the next, which is not as good but usually shows differences and answers the question. In the case of TiVo Research/TRA the average sales ROI increase associated with using purchaser targeting as measured by massive passive convenience sampling, over sex/age targeting as measured by probability sampling, is +28%. Which is why I say that convenience sampling may be better than probability sampling -- not in an abstract way, but because the particular conditions surrounding today’s probability sampling and today’s convenience sampling lead to generating more money through advertising by using convenience sampling or at least a combination of both types of sampling.

The opinions and points of view expressed in this commentary are exclusively the views of the author and do not necessarily represent the views of MediaBizBloggers.com management or associated bloggers. MediaBizBloggers is an open thought leadership platform and readers may share their comments and opinions in response to all commentaries.