The sorry state of research in India

If you have done quantitative research of any sort in India, you would have noticed ‘show card’ written on the questionnaire and noticed the interviewer clumsily thrust a wad of papers at the respondent. Ever wondered why it is a ‘show card’ and not ‘show paper’? To answer this, let me go back about 25 years and talk about the first consumer research I witnessed.

The research was a sequential monadic one with paried comparison at end for a white detergent bar. It was most probably for Hindustan Lever Limited (as Hindustan Unilever Limited was known back then). Don’t remember who the research agency was but the thing I clearly remember was how professionally it was conducted. The entire research took a few weeks – about 2 months if I remember right. There was ample sample given to test – ~4 big bars of each type which would easily last a couple of weeks or more. The researchers asked if my mom could wash a few clothes with the samples given before them, just to be sure of her washing method. And, there were always 2 researchers administering the questionnaire, one asking the questions and the other overseeing and manning the show card. The ‘show card’ was a set of A4 cards with the responses which were neatly laminated and spiral bound. Both the interviewers were extremely diligent in asking the questions and noting down the answers that my mom gave. The gift for participating in that research was a casserole of some sort.

That was the last I saw any ‘show cards’. 12 years later, when I oversaw my first research, they were ‘show papers’ but still called ‘show cards’.

Now contrast that research to a similar one done these days… The entire research is extremely hurried – often a sequential monadic test is completed within 2 weeks. A very small sized sample is given to test – ~30ml shampoo or ~50gm of toilet soap – which is hardly enough for a few uses. Not many check the actual usage in such quantative researchers, even on a sub-set of the sample. And, only one researcher manages the entire interviewing process. The gift given at the end is pretty much the same, a casserole of some sort.

To give other examples where I was a respondent… A researcher knocked on my door one day to check what kind of DTH connection I used. I said, “Tata Sky” and she asked my name and promptly replied, “Thank you, Sir. If some one comes to check, please tell that I interviewed you. I don’t want to waste your time.” It’s anybody’s guess what data was filled in as my response to that lengthy questionnaire she carried. BTW, I disqualify for such type of research as I am in one of the professions that is in the research exclusion list and she didn’t even check that part.

The worst was what I experienced in the Government of India census survey 2011. Either my wife or visiting parents gave the responses for the main census (which was largely filled in using the society member data with the chairman) so it’s hard to guess the quality of data capture. And, between the main census and the caste census, I shifted houses and misplaced the slip they gave that identified us and our responses in the main survey.  I was home when the interviewers came and all they asked was for the slip (which I couldn’t give) and paste a stamp on the door stating the caste census was completed.

Even with this broad, anecdotal comparison, its easy to see how the quality of research has gone down substantially over the years. While I am not sure when and how this happened, I suspect it has a lot to do with companies trying to cut costs at one end and heavy cost competition among the research agencies at the other. This meant cheap, contracted field teams collecting data on ground and low quality researchers analysing and presenting it at head office. Overtime, this cost pressure has also lead to dearth of good talent in the field. Today, as newer types of research fields open up –  data analytics, data science etc., – the best talent (statisticians, researchers) go there.

I think this ‘quality of research’ problem is very severe today. Sample these…

  • Indian Readership Survey: Several newspapers rejected 2013 data questioning methodology of the new research firm that conducted the survey.
  • Television Audience Measurement: BARC got setup citing inadequacy of data capture by TAM. Then, a few years later both merged.
  • FMCG Retail Audit Measurement: FMCG firms and Nielsen that conducts this survey have a love-hate relationship with retail audit data being questioned multiple times in the last few years – 2009, 2015.

The above issues are just a few about large scale researches. I am sure every brand manager has questioned and/or rejected findings of a smaller quantitative research citing ‘bad data collection’ once in a while. I am specifically stating ‘data collection’ because not many brand managers and even research managers understand the statistics behind the researches – the sampling methodology, the rating scales, the analysis techniques. Hell, many don’t understand the research methodology/design in  its entirety. And, that’s where the bigger problem lies – people aren’t questioning the basics.

 

A deeper look into the Fin Serv emails that go to SPAM (5/5)

Following the previous post on brands and products from financial service firms, here is a look at some of the actual emails that I got during those 2 months.

The we don’t want you to know who we are emails:  A huge chunk of emails just landed without any brand name in them. Some have a bit of design going into them, most don’t.

The PowerPoint slide converted to email: What’s the point in sending this badly designed email?

The overzealous marketers: Every festive occasions is a spam occasion.

The template so old that we don’t care: This is an actual email I received on my birthday and no it’s not Gmail that didn’t load the images!
ICICI template without images

The we’ve got one spam email to make it count email: This has to be one of the longest spam email’s I got. Pretty much the entire website seems to have been emailed!
MaxMyWealth email

The bad databases: Nothing can put you off more than an email telling you to marry once again and this time with their loan.
StanC personal loan for wedding

The cut-paste code gone wrong: This one was a just about filling the space.
Aegon email with message repeated

Just as I blogged about this analysis elsewhere, I got this email from DSP BlackRock MF with an appalling mistake – they  incorrectly mentioned the potential returns of a lumpsum investment as that for a SIP thereby making an error of the magnitude of 100x. In a correction to the email, they failed to convert millions to crores correctly. It really tells you about the state of email marketing in India and those who man it, doesn’t it?

The late-night comedy trip on Trump

Of late I’ve been binge watching late-nights shows from the US, thanks to a good data plan and immediate posting on YouTube of these shows (take that Star World Premier).

Here are the channels of shows of Stephen Colbert (CBS), Trevor Noah (Comedy Central), Seth Meyers (NBC), John Oliver (HBO), James Corden (CBS), Samantha Bee (TBS), Chelsea Handler (Netflix), Jimmy Kimmel (ABC), Jimmy Fallon (NBC), Conan O’Brien (TBS), Bill Maher (HBO) and Larry Wilmore.

I discovered them accidentally when I was looking for presidential debates, except for John Oliver who I discovered thanks to his take on Net Neutrality. Most of these show hosts gave their comic and informed take on these debates and I sort of liked them.

In the last few months, there is a peculiar pattern to most of these shows; especially the first three linked above. Long monologues dissecting and lampooning Trump news, antics and same guests doing the rounds. And a few days ago, Trump went after them with this:

Repetitive – Yes. Unfunny – No. You do get a déjà vu watching these shows with same news sources being cited and similar jokes being told. But, I don’t mind as one or the other of them, has an extremely sharp take. Vox in fact praised how they covered Trump citing the reason behind this as their extremely low tolerance for bullshit.

This brings me to the question, if there are 10 odd shows on TV in the US, with all of them having a similar theme, look and format, why aren’t they any on Indian TV?


Addendum: Here are a couple of very well written articles about the late night journalism – Firstpost: Late-night comedy is new face of conscientious journalism, Guardian: How Fallon fell: why is the late-night host floundering in Trump’s America?