Many perspectives, 1 simple etiquette

How Reliable Are Fact Check Sources?

Author: Andrew Gripp
Created: 11 July, 2014
Updated: 21 November, 2022
5 min read

In 2011, New York Times columnist Paul Krugman lamented that the proliferation of half-true statements and fallacious accusations during the 2012 presidential campaign had inaugurated an era of "post-truth politics."

This era is particularly enabled by the features of our contemporary media-saturated political environment: the distrust of the mainstream media, the rise of alternative and slanted news outlets, and a 24-hour news cycle that emits data and soundbites at a velocity and volume that can overwhelm even the most omnivorous and scrupulous consumer.

But the very technology that enables the proliferation of misinformation should, theoretically, be able to combat it. Enter the fact-checking phenomenon.

The first major political fact-checker to arrive on the scene was FactCheck.org, which was launched in 2003 by the University of Pennsylvania's Annenberg Public Policy Center. FactCheck.org’s dozen-member staff regularly updates its blog evaluating the claims of politicians, pundits, and advertisements by checking them against authoritative reports, studies, and other documented facts.

In 2007, the Tampa Bay Times started its own fact-checking project: 

PolitiFact. Its staff harvests the statements of politicians, lobbyists, journalists, and pundits and assigns them a rating on the "Truth-O-Meter," ranging from "True," then through several degrees of veracity -- all the way to "False" and "Pants on Fire."

Finally, The Washington Post made its Fact Checker permanent in 2011 after a temporary existence during the 2008 presidential election. Its operator, Glenn Kessler, doles out 0 to 4 Pinocchios to statements based on the degree of a claim's factuality, completeness, and intended effect.

While these fact-checking outfits nobly strive to maintain a floor of objectivity in their work, it is important to remember that old Enlightenment figure Giambattista Vico's verum factum principle: the truth is made -- made by people with their own biases, limitations, and subjective standards.

So, how reliable are fact check sources? Let's start with an illustrative and comparative example.

PolitiFact's Angie Holan awarded President Obama's claim, "If you like your health care plan, you can keep it" as 2013's Lie of the Year. However, in 2008, Angie Holan rated that same claim as "True." As Forbes' Avik Roy points out in his chronicling of PolitiFact's evolution on the veracity of this claim, Holan initially only cited sources that confirmed her conclusion and neglected more critical -- and clairvoyant -- voices.

Though one may forgive this error somewhat given that Congress did not pass the Affordable Care Act (ACA) until March 2010, even thereafter, PolitiFact and The Washington Post's Fact Checker were slow -- if not reluctant -- to reach their later, more accurate conclusions.

For instance, The Washington Post's Kessler makes no mention of Obama's untruth in a lengthy examination of facts and myths about the ACA in 2011, and not until October 2013 does he assign it 4 Pinocchios.

As for PolitiFact: even in late 2012, after an exhaustive summary of analyses from various government reports and assessments from think tanks, Louis Jacobson -- in an article edited by Angie Holan -- arrives at the following conclusion:

…Obama suggests that keeping the insurance you like is guaranteed.In reality, Americans are not simply able to keep their insurance through thick and thin. [...] We rate Obama’s claim Half True.

A year later, PolitiFact "evolved" and gave this exact same claim a "Pants on Fire" rating even though it was already long understood that some plans -- even those that had been "grandfathered" in at the time of the ACA's passage -- were going to be cancelled because they did not comply with rules and regulations that the law imposes on insurers.

FactCheck.org, on the other hand (writing weeks before the bill was even passed), was correct in determining that while most people could keep their plans and doctors, the president "can’t make that promise to everyone."

FactCheck.org's reliability is due in part to its somewhat inconvenient yet also wise refusal to assign simple ratings along a continuous spectrum: it does not summarily declare statements "True" or "False," but instead provides detailed analysis explaining their incorrectness or incompleteness and lets the reader reach his or her own conclusion.

Critics from all sides have pointed out the bias and editorial nature of some of PolitiFact's and Kessler’s purportedly objective verdicts.

For instance, PolitiFact chose the claim "Republicans voted to end Medicare" as 2011's Lie of the Year and had given the assertion that Republicans wanted to "end Medicare as we know it" a "Pants on Fire" rating. Liberals pointed to features of Paul Ryan's budget plan that would have drastically altered Medicare as evidence that there was some truth to the claim and thus it deserved a more sympathetic rating.

Conservatives have pointed to the abuse of the "Pants on Fire" label as a way to embarrass Republicans, a rating which began as a joke and is only to be reserved for claims that are both inaccurate and ridiculous, such as Joe Biden's statement that President George W. Bush was "brain-dead."

According to one time-bound study, PolitiFact rated 119 Republican and conservative claims as "Pants on Fire" compared to 13 from Democrats and liberals.

The Washington Post's Kessler likewise blends objectivity and opinion, in part due to his 

admission that, "We are interested only in verifiable facts, though on occasion we may examine the roots of political rhetoric."

For instance, in 2012, he researched the accusation made in a pro-Romney ad that Obama had not visited Israel since becoming president. Though entirely true, Kessler gave the claim 2 Pinocchios because he disagreed with the insinuation that President Obama had been uniquely unfriendly to Israel compared to his predecessors.

Thus, the most appealing feature of some fact checking websites -- the simplified determination of a statement's truthfulness -- may also be their greatest weakness. Such ratings, whether on a "Truth-O-Meter" or the "Pinocchio scale," needlessly inject subjectivity into what are otherwise objective analyses.

Furthermore, Kessler's one-man fact-checking operation is prone to missing contradictory information. For instance, when Herman Cain claimed that terrorists had crossed the Mexican border during a Republican debate, Kessler mistakenly called that a canard, having overlooked ample and reliable evidence that terrorists have indeed come to the U.S. through Mexico.

Relatedly, fact checkers are not immune to confirmation bias; that is, citing sources that support their favored conclusion. Holan's positive evaluation of the president's insurance promise in 2008 is one example. Also, critics can point to other examples of fact checkers relying on establishment or self-proclaimed expert opinion to defend the conventional wisdom or to falsify controversial statements.

In 1920, the great American writer Walter Lippmann asserted, “There can be no liberty for a community which lacks the means by which to detect lies," which brings us to the real importance of fact check sources.

Their strength lies in the creation of databases, many of them easily searchable, of statements and promises made by influential and powerful people. Such readily available resources are a prerequisite for journalistic and political accountability. By compiling so many statements and bringing outside and authoritative sources to bear on them, fact check sources -- despite their imperfections -- can aid us in preventing the threatening entrenchment of a "post-truth" political environment.

Photo Credit: Ivelin Radkov / shutterstock.com