Statistics, Survivor, and Media Research - Jonathan Steuer-TiVo

Nearly every week Jeff Probst reminds me of the tenuous relationship between truth and statisticsquotLosers I39ll see you at Tribal Council tonight Remember the last person left standing wins immunity and a oneinseven shot at winning this gamequotThing is that39s not true Or rather it39s only partly true the part about winning the challenge resulting in freedom from elimination at that evening39s Tribal Council is accurate it39s the part about the oneinwhatever chance of winning that isn39tOf course if Survivor were a fair game of chance where quotfairquot has its technical meaning meaning that each outcome is equally probable that part would be true too But of course that39s not how the game works Survivor isn39t a math problem but rather a tangled knot of complex flawed ornery human beings Early in the season contestants vote out one of their own every week strategically trying to eliminate either the weakest links or the biggest potential threats The ultimate results are decided by a jury of contestantpeers amp8211 they vote to determine which one of the last few remaining competitors most deserves to win That39s not a game of chance at all let alone a quotfairquot onequotWell duhquot you say Survivor is reality TV not realityreality I understand there39s a difference But using sloppy language about statistics is a serious issue and our realworld sloppiness about data frequently leads to inaccuracies that are as inaccurate as Jeff39s comments about the probabilities of Survivors winning a million bucks amp8211 and as potentially costlyquotStatisticsquot you say quotYuck Shut up with your fancy statistics stories Jeff just means that one of those seven people will winquot But here39s the thing precision does matter Sometimes it matters a lot media decisions involving programs that cost tens or hundreds of millions of dollars can depend on fractions of rating points amp8211 so it39s important to remember where our data like our food come from and how they are prepared And yes data is a plural noun Didn39t I just tell you precision about language mattersI39m not going to embark on a geeky rant about misuse of statistical data in media researchamp8230 mostly because I39ll be sure to lose the few of you who are still reading if I do That39s a rant for another day But here are a few examples of some basic filters to apply when looking at statistical data in the context of media research1 Acknowledge limits We don39t know what we don39t measure Even the most rigorous measurement doesn39t tell you what happened before or after that measurement was taken Making assumptions about what happens outside the range of what you are measuring might be reasonableamp8230 or it might not Example A current TiVo research project is examining the relationship between use of quotoverthetopquot TV technologies Netflix HuluPlus etc and other TV viewing To distinguish between viewers and nonviewers we asked TiVo panelists in January 2012 whether they used these technologies in the past month and if so how much they used them As much as I39d like to use those answers to compare with TV viewing in 2010 and 2011 I can39t because the question explicitly covered December 2011 amp8211 January 2012 and it39s not reasonable to assume that their viewing via OTT technologies was the same in January 2012 as it was in 2011 That39s something we39d have to ask specifically about So we will2 Know what39s in the data sausage We sometimes forget about amp8212or consciously choose to ignore amp8211 the large number of assumptions that are built into the way data are collected It39s important to remember where data come from before placing too much weight on the results Example Detractors of TV settop box data are constantly complaining about the quotTV on box offquot problem amp8211 where the STB is on and therefore quottunedquot to a channel but nobody is watching Those who have spent any significant effort looking at STB data know that it39s not difficult to identify this effect and reduce it by applying durationcapping rules amp8211 that is ignoring quottuningquot events longer than a certain length STB data analysts typically disclose how they apply this process and the magnitude of the effect so users of the data can decide for themselves whether to use the data Conversely when is the last time you remember seeing data about compliance rates among Nielsen households amp8211 specifically independent audits of how reliably members of Nielsen homes press the appropriate buttons to show whether they are in or out of the room when the TV is on3 Mind your projections It39s a very rare occasion when it39s feasible to collect complete census data amp8211 that is to be able to measure every single member of a group amp8211 and so almost all realworld media research uses sampling techniques to select an appropriately representative sample or uses statistical calculations to weight a sample to project to a particular population In using any research data it39s important to remember where the data you39re looking at come from and make sure that the conclusions you39re drawing are reasonable based on the sample and methodology you39re using Example At TiVo we39re constantly being reminded how quotTiVo owners are weirdquot because these intellectual moneyed ubergeeks went out and purchased and installed DVRs on their own which must mean that our data are nonrepresentative of the US population as a whole Unfortunately for me anyway this may be to a degree a reasonable criticism More unfortunately since TiVo has been so careful about maintaining our customers39 privacy our StopWatch sample is truly anonymous and I have no idea what the demographic composition of the sample is Without that information it39s impossible to model our ratings data to enable us to project to the US population as a whole As TiVo39s partnerships with cable MSOs such as our recentlyannounced Comcast Xfinity partnership continue to expand and as other aspects of our methodology evolve we will eventually be able to project to a national sample For now though we39re focusing on what we do know which is to provide a very high level of detail about how DVR users who now represent about half of US households use television4 Most media games aren39t zerosum The media landscape isn39t quite like the physical world most decisions and activities are no longer the simple tradeoffs of one experience for another as they were back in the days when there were three national TV networks a handful of independent broadcasters in most markets and one or maybe two TV sets in each household Since we can watch many different kinds of media broadcast cable movies videos etc on many devices live TV DVR computer tablet mobile phone portable media player etc it39s not a reasonable assumption to assume that because someone started engaging in one media activity they must have stopped doing something else The media universe has become too complicated to think of TV measurement as a zerosum game but we haven39t yet figured out how to measure it that way Example Trying to measure the impact of overthetop technologies on TV viewing has been very challenging as the increase in popularity of alternative ways to watch quotTVquot has grown in parallel with the adoption of many different opportunities for other kinds of media fragmentation The debate about decreases in viewing of certain kids39 cable channels eg Nickelodeon and the apparent correlation of this viewing decline with the availability of similar programming on OTT services eg Netflix may signal trouble amp8211 but it may also signal an increase in total viewing time of all similar content as kids increasingly watch not just on TV but on tablets and computers and phones during times when mom or dad are using the TV to watch their shows Research on media multitasking time spent with media like Ipsos MediaCT39s Longitudinal Media eXperience LMX study has shown steady increases in the length of the quotmedia dayquot when simultaneous media usage experiences like using a computer while watching TV are taken into account Understanding complex problems like evolution of media usage is difficult and there is much room for additional research in this areaI could go onamp8230 and I may in a future post In the meantime those who enjoy statistical nitpicking millions of you I39m sure may revel in a great and frequently updated analysis of stats in the news The nonprofit Statistical Assessment Service STATS Website at httpwwwstatsorg STATS analyzes everything from whether having one drink can increase the probability of a car accident to whether a soda tax could really curb obesity to whether Kabul is actually safer for kids than New York City For even more fun with media and statistics you might want to take a look at some of the following sources I consulted while writing this postamp183 Smith Martha K Professor Emerita of Mathematics University of Texas amp8211 Austin quotCommon Misteaks Mistakes in Using Statistics Spotting and Avoiding Themquotamp160amp183 Cue Kerry MathsPig blog quotArchive for the 3910 BIG Media Maths Errors39 Categoryquotamp160Jonathan Steuer joined TiVo as Vice President Audience Research ampamp Measurement where he leads TiVo39s efforts to develop innovative new media research products Jonathan can be reached at jsteuertivocomRead all Jonathan39s MediaBizBloggers commentaries at InteracTiVotyCheck us out on Facebook at MediaBizBloggerscomFollow our Twitter updates MediaBizBloggerThe opinions and points of view expressed in this commentary are exclusively the views of the author and do not necessarily represent the views of MediaBizBloggerscom management or associated bloggers MediaBizBloggers is an open thought leadership platform and readers may share their comments and opinions in response to all commentaries

Become a Thought Leader.
Share your thought leadership article with industry professionals & have them archived in Thought Leadership

Statistics, Survivor, and Media Research - Jonathan Steuer-TiVo

Author

Publish date

Channel

Archived in

Jonathan Steuer

"About That with Carl Mayer:" Alaina Donnellon (Episode 10: Political Spending) - By Carl Mayer

More Videos:

Become a Thought Leader. Share your thought leadership article with industry professionals & have them archived in Thought Leadership

Statistics, Survivor, and Media Research - Jonathan Steuer-TiVo

Enjoying This Commentary? There's More to Love

Jonathan Steuer

"About That with Carl Mayer:" Alaina Donnellon (Episode 10: Political Spending) - By Carl Mayer

The payment was successful

Cookies

Become a Thought Leader.
Share your thought leadership article with industry professionals & have them archived in Thought Leadership