Fallacies (2) – cherry picking

Cherry picking is a familiar fallacy in research.* A simple definition of cherry picking is “the intentional or unintentional** selection of evidence to prove one’s point or support one’s position, in research that is presumably objective.” Here’s a more detailed definition.

The activity of cherry picking is very common; so is the allegation of cherry picking. Here are some examples of the contexts in which cherry picking may occur or may be alleged to occur (I do not know or care whether these allegations are correct or incorrect).

Examples of actual or alleged cherry picking

First example

“[News reports that allege ‘Facebook raises cancer risk’ are] based on an appalling article by psychologist Aric Sigman which was published in the magazine Biologist. You can read it online as a pdf and it is a wonderful example of cherry-picking evidence and citing correlations as causes.” Full story

Second example

“[Tony Perkins of the Family Research Council] told TheDC that the SPLC [Southern Poverty Law Center] cherry-picked the scientific evidence it chose to cite against the Family Research Council and other similar groups in its related report, titled ‘10 Anti-Gay Myths Debunked,’ and ignored contrary evidence.” Full story

Third example

“[The United Kingdom’s national weather service] was last night facing accusations it cherry-picked climate change figures in a bid to increase evidence of global warming. UK climatologists ‘probably tampered with Russian-climate data’ to produce a report submitted to world leaders at this week’s Copenhagen summit, it is claimed.” Full story

Fourth example

“[Two researchers who studied the risk of cancer among smokeless tobacco users] make it clear that they produced their summary estimates by either combining risks for exclusive users (i.e. 1.8 and 0.85), or risks for users who might have smoked (0.9 and 1.67). This is both logical and scientifically valid. But which estimates did [another researcher] use? The highest, of course. For the Lancet study he used the estimate of 1.8 in exclusive users, but for the International Journal of Cancer study he used the estimate of 1.67 in smokeless users who might have smoked. His goal was to maximize the health risks from smokeless tobacco use…. The scientific term for this epidemiologic method is cherry picking.” Full story

Fifth example

“Many [securities] trading systems do backfitting, avoiding the recessions or downturns. It is easy to find an indicator that gave a definitive signal when looking at past data. The problem is that such systems, lacking rigorous development of hypotheses, failing to use out-of-sample data, and willing to accept an insufficient number of cases, usually produce post-diction rather than robust predictive models.” Full story

The Takeaway: Remember that clear writing presupposes clear thinking. Try to avoid fallacies such as cherry picking.

See disclaimer.

*In debating, selling, and litigating, the use of cherry picking is not usually considered a fallacy; generally, one is assumed to be subjectively promoting one’s own position. But in research, one is assumed to be striving for objectivity.

**Unintentional (often unconscious) bias is very common; we all prefer familiar evidence to strange evidence and comforting evidence to disturbing evidence. But we must overcome this preference.