The old saying everybody has heard says “don’t believe everything you read”. Current research shows while this is true, we also need a saying instructing us “don’t believe everything you see – especially on a chart”.
In their recent article Blinded with science: Trivial graphs and formulas increase ad persuasiveness and belief in product efficacy Tal and Wansink experiment with different ways of providing information about medical drugs to consumers. They have two groups of people: the ones seeing just text, and the ones seeing also a chart. The graph was trivial: it added no new information at all. The control group saw a text saying that
In their recent article Blinded with science: Trivial graphs and formulas increase ad persuasiveness and belief in product efficacy Tal and Wansink experiment with different ways of providing information about medical drugs to consumers. They have two groups of people: the ones seeing just text, and the ones seeing also a chart. The graph was trivial: it added no new information at all. The control group saw a text saying that
A large pharmaceutical company has recently developed a new drug to boost peoples’ immune function. It reports that trials it conducted demonstrated a drop of forty percent (from eighty seven to forty seven percent) in occurrence of the common cold. It intends to market the new drug as soon as next winter, following FDA approval.
The experimental group saw the same text, followed by a graph:
The result? Participants who saw the graph believed the drug to be more effective than the control group.
As it is, this result could be just due to increased information retention: graphs make it easier to remember things. Well, the authors thought the same, and replicated the study by checking for retention 30 minutes after the study. It turned out there was no difference in memory, but the same effect persisted. What could be the cause? Looking at the data, the authors found out there’s a significant interaction of belief in science and the chart effect – people with more belief in science ended up being more convinced because of the graph! Uh oh, this is worrying for us science buffs.
In their third experiment, they changed the setting by replacing the bar chart with a chemical formula. What happened was that once again, a scientifically-looking item increased the effectiveness rating. Unfortunately, the authors neglect to mention whether the interaction with belief in science was significant here 8probably not, since it wasn’t reported).
It’s understandable that a chart would make a claim seem more scientific. After all, one of the hallmarks of pseudoscience is hiding all the results behind vague words – a chart is at least clear. Unfortunately, by choosing the right measures and axes, one can design compelling yet false claims with graphs quite easily. A chart does not ensure sound science, or sound data. I guess we’ll just have to stick to trying to evaluate the actual claims carefully. Don’t believe me? Well, here’s a chart to convince you:
As it is, this result could be just due to increased information retention: graphs make it easier to remember things. Well, the authors thought the same, and replicated the study by checking for retention 30 minutes after the study. It turned out there was no difference in memory, but the same effect persisted. What could be the cause? Looking at the data, the authors found out there’s a significant interaction of belief in science and the chart effect – people with more belief in science ended up being more convinced because of the graph! Uh oh, this is worrying for us science buffs.
In their third experiment, they changed the setting by replacing the bar chart with a chemical formula. What happened was that once again, a scientifically-looking item increased the effectiveness rating. Unfortunately, the authors neglect to mention whether the interaction with belief in science was significant here 8probably not, since it wasn’t reported).
It’s understandable that a chart would make a claim seem more scientific. After all, one of the hallmarks of pseudoscience is hiding all the results behind vague words – a chart is at least clear. Unfortunately, by choosing the right measures and axes, one can design compelling yet false claims with graphs quite easily. A chart does not ensure sound science, or sound data. I guess we’ll just have to stick to trying to evaluate the actual claims carefully. Don’t believe me? Well, here’s a chart to convince you:
RSS Feed