Jimmy Kimmel’s Lie Witness News Teaches Us about Interviewer Bias

question design

Have you seen Jimmy Kimmel’s Lie Witness News? The segment where a fake newscaster goes out on the streets of Hollywood to ask the general public tricky questions about current events?

If you haven’t seen it check out this clip from May 2014 when they asked people “considering Godzilla is based on the true story of the giant lizard attack on Tokyo that killed more than 100,000 people in 1954, do you think it’s wrong of Hollywood to glamorize this event for entertainment value?”

And then they throw in questions like “Do you support the bill before Congress that would allocate $600 million to equip the U.S. Navy with anti-reptile capabilities?” “Do you believe these creatures are caused by global warming?”

At 3:50 we see one of the tricked participants in Jimmy’s audience so Jimmy Kimmel takes the opportunity to ask her what was going through her mind during the interview. We learn that she later consulted Google to confirm or deny the reality of Godzilla.

I would venture to say that Jimmy’s team chose which interviews to show in this segment based entirely on entertainment value. Hopefully, the general American public isn’t quite this gullible. But, this does bring to mind a few good points about interviewing techniques and questionnaire design.

THE QUESTION DESIGN STANDPOINT

Leading Questions. By frontloading the question with “Based on the fact that… ” the respondent is primed to assume that what comes next is truly a fact. Plus, if the respondent believes the interviewer is from a real news station, they may be less likely to engage their critical thinking skills. Critical thinking muscles needed to be exercised in order to say, “Wait a minute, I don’t think that’s true.”

But, even if critical thinking muscles raised doubt in the respondent’s mind, they may have still fabricated an answer that they thought fit the situation, like the woman’s response to the anti-reptile bill. There is an innate pressure to avoid looking stupid in front of other people, especially when there’s a TV camera around. So rather than question the bill’s existence or even just say she is not aware of the anti-reptile bill, she made up an answer that fit her ideology. This may be a result of social desirability — an answer is given that will gain approval or fit in as a “normal” response.

Plus, people are generally agreeable. If you ask if they approve of something, they are likely to say yes. If you ask if they are opposed to something, the answer may still be yes. This is why Yes/No questions are not the best question type for opinion or attitudinal questions.

Interviewer bias is when a respondent feels any level of pressure to answer in a way that they think will make the interviewer happy. When people are asked questions by another person, they are likely to be influenced to some degree by the other person. Online and paper surveys avoid interviewer bias since they are self-administered. Without an interviewer’s influence, a respondent may be more likely to admit they don’t know something or to answer questions more objectively, without worry that they will feel embarrassed by giving a socially undesirable answer.

LEADING QUESTIONS

Examples of how not to phrase questions if you really want an objective answer.

1. “Mr. Politician has the endorsement of local businesses. Are you in favor of Mr. Politician?”

 

2. “We recently redesigned our website to make it more user-friendly. Do you think the website has improved?”

 

3. “Knowing that Mark Grace has been arrested twice for drunk driving, do you think he should be allowed to become a Diamondbacks coach?”

vs.

“Knowing that Mark Grace was among the players when the Diamondbacks won the World Series in 2001, do you think he should be allowed to become a Diamondbacks coach?”

REVISIONS TO INCREASE OBJECTIVITY

1. Do you have a favorable or unfavorable opinion of Mr. Politician?

  • Favorable
  • Unfavorable
  • Not sure
2. We recently redesigned our website. In comparison to the prior design, how would you rate the following elements of the current website?

Worse
About the Same
Better
Ease of Use
o
o
o
Organization
o
o
o
Visual Appeal
o
o
o
Content
o
o
o

3. Mark Grace is a former Major League Baseball first baseman. He played for 16 seasons with two teams: the Chicago Cubs and Arizona Diamondbacks. He helped the Diamondbacks win their one and only World Series Championship in 2001. After retiring from baseball in 2003 he was a TV commentator with the Diamondbacks and Fox Saturday Baseball.

Twice, Mark Grace has been arrested for drunk driving. He did not return to broadcasting in 2013 during which time he completed his sentencing. In 2014, he became the hitting instructor for the Diamondbacks Class A team. He bats and throws left.

In your opinion should Mark Grace be eligible to coach Major League Baseball? Please briefly explain why or why not.

Check out more of Jimmy Kimmel’s Lie Witness News. I especially like the one where they ask for reactions to Apple’s new iTime Smart Watch (actually a Casio watch with an Apple sticker on it).