Studies Show Professional Fact-Checking Is Subjective

Ask 10 professional fact-checkers to rate something as true or false, and get one, unanimous answer. That’s what we assume will happen based on our understanding of the word “fact” and our understanding of the responsibilities of a person who “checks” them. That assumption is incorrect, according to a recently published study that found a “low level of agreement of professionals over what is misinformation.” Fact-checkers sometimes enjoy a reputation as paragons of objectivity — but, based on the conduct of many fact-checkers, the opposite conclusion isn’t all that surprising.

In a paper published in Nature on December 20, 2023, a six-person research team (Aslett, et. al.) found that “Online searches to evaluate misinformation can increase its perceived veracity,” as broadcasted in their headline. But a more significant finding went unheralded: that fact-checkers often disagree about what misinformation is. As part of the research model, four to six professional fact-checkers evaluated 265 news articles to rate them as “true” or “false/misleading.” According to the supplementary information they posted, the professional fact-checkers only agreed unanimously on how to rate less than half (44.62%) of the articles — a far larger discrepancy than the online search effect they were actually studying.

Keep in mind that fact-checkers have a rather simple task: they can rate articles as true or false, or possibly choose from a limited set of options in between. As anyone who has ever taken a test with true/false or multiple-choice questions knows, there is a non-negligible chance of selecting the correct answer by pure accident, so there is a chance of a limited number of fact-checkers selecting the same option from a limited number of ratings by pure accident.

Researchers can apparently control for random chance in agreement between raters by calculating a Fleiss Kappa score. I’ll admit I don’t fully understand how this statistic is calculated, but complete agreement would yield a score of 1, while a complete lack of agreement would yield a score of 0, or a negative number. In this case, the researchers found a Fleiss Kappa score of 0.42, which is again less than one-half.

Aslett, et. al. were not the first research team to find such low agreement among fact-checkers; in fact, they noted the level they found was “slightly higher than other studies.” They referenced a paper published September 1, 2021 in Science Advances, in which a four-person research team (Allen, et. al.) found that “small, politically balanced crowds of laypeople” could produce results just as good as professional fact-checkers. In that study, three fact-checkers agreed unanimously on how to rate only 49.3% of 207 articles (with a Fleiss Kappa score of 0.38).

Allen, et. al.’s paper, in turn referenced a July 19, 2018 paper by researcher Chloe Lim, published in Research & Politics. Lim compared fact-checks of 77 identical or nearly identical claims, reviewed by both Politifact and the Washington Post Fact Checker. She found the two sites agreed on 49 (63%) of the claims on a five-point scale, but she calculated a Cohen’s Kappa score of 0.47 (Cohen’s Kappa is like Fleiss Kappa, but only for exactly two raters). “Fact-checkers,” she noted, “disagree more often than one might suppose.”

The finding that fact-checkers don’t necessarily agree all that much should act like an unexpected ice shower on those who would use fact-checkers to either control “misinformation” or advance a political narrative. This tactic is especially employed against independent media, such as Family Research Council’s “Washington Watch” and The Washington Stand, which cover the stories the mainstream media refuses to cover.

One recent example is TWS’s senior reporter and editor Ben Johnson’s extensive coverage of the World Health Organization’s planned pandemic treaty, which he then discussed on “Washington Watch.” A reel of that interview, which FRC posted to Instagram, was flagged as containing “Partly False Information,” after it was “reviewed by independent fact-checkers.”

To be more specific, FactCheck.org rated the interview as “partly false” because Johnson claimed that the “WHO pandemic agreement threatens national sovereignty.”

FRC disputed the rating based on the following information: “The WHO Pandemic Agreement places a number of restrictions and demands on U.S. sovereignty:

• “Under the WHO Pandemic Agreement, nations would retain their sovereignty only ‘in accordance with the Charter of the United Nations and the general principles of international law’ (Article 3:2).
• “The agreement will create a global medical force at WHO’s disposal. Member nations must create and fund ‘a skilled and trained multidisciplinary global public health emergency workforce that is deployable’ to nations at their request to ‘prevent the escalation of a small-scale spread to global proportions’ (Article 7:3).
• “It gives The Hague jurisdiction over members’ disputes. If WHO is not able to solve disagreements between members, nations may agree to the ‘submission of the dispute to the International Court of Justice.’ They may also settle things through arbitration by the Conference of the Parties (Article 34:2).
• “Real decisions are made by nameless, unaccountable bureaucrats from around the globe. The agreement creates a ‘Conference of the Parties,’ headed by a secretary, within one year of the treaty’s ratification. It will meet annually, or at any member’s request. ‘Only delegates representing Parties will participate in any of the decision-making of the Conference of the Parties’ (Articles 21 and 24).
• “WHO takes a double tithe of U.S. vaccines, medicines, and equipment. ‘In the event of a pandemic,’ the United States must give WHO ‘a minimum of 20%’ of all ‘pandemic-related products,’ such as vaccines or personal protective equipment, for global redistribution: ‘10% as a donation and 10% at affordable prices’ (Article 12:4b(ii)(a)).”

Of course, evenhanded justice is nearly impossible when the prosecuting attorney is also the judge and jury. “Thanks for your email disputing our rating of your post,” FactCheck.org replied insincerely. “We’ve reviewed the examples you gave and believe our rating is correct.”

The email went on to explain, “The agreement would not affect national sovereignty — meaning it does not affect countries’ sovereign rights to set policies within their own national borders. The examples you give are related to international obligations and do not mean the WHO would interfere with national sovereignty for any country.” Utterly ignored in this illogical reply are the multiple ways in which FRC pointed out that the treaty’s international obligations impinge on a country’s sovereign rights by attaching strings to their pandemic equipment stockpiles and public health emergency staff.

In the article FactCheck.org referenced, they reason that the WHO Pandemic Accord will not affect national sovereignty is because the WHO says it won’t — which sounds like the claim they should be fact-checking. Johnson’s research pores through the proposed text of the agreement; FactCheck.org does not.

FactCheck.org quoted only a single, biased expert, Lawrence Gostin, a Georgetown law professor who helped draft the treaty. “The US constitution is the highest law of the land. No international treaty can override the provisions of our constitution,” insisted Gostin. It doesn’t take a law degree to know that the Constitution’s Supremacy Clause (Article VI, Clause 2) makes international treaties equal to the Constitution as “the supreme law of the land,” by which judges in every state are bound.

Obviously, FactCheck.org already “believed” in their rating, despite the slim evidence, and no recitation of the facts was going to change their opinion. (That’s what it was, a judgment based upon opinion, not fact.) But there is no one else to appeal to. Social media platforms outsourced the business of fact-checking in the first place because they don’t want to wade into the inescapable morass of contradictory opinions, borderline rulings, and fact-less findings.

In many ways, this recent incident with a fact-checker is characteristic of a trend of biased fact-checking, seemingly designed to discredit disfavored opinions, which FRC has been experiencing for years.

“Our social media posts have had fact-check labels applied from time to time, especially since the COVID-19 pandemic. However, I would characterize the fact-checks as more of a difference of opinion, rather than a factual correction,” Keri Boeve, director of Social Media at Family Research Council, told The Washington Stand. “When we have taken the time to file a dispute or appeal the fact-check on a post, it has never generated a change, and the responses (if we get one) claim the slimmest and most debatable reasons.”

The virtue of a fact-check is it discredits outright falsehoods, allowing public debate to more quickly proceed toward the truth. This virtue becomes a vice when fact-checks are weaponized against debatable propositions — opinions or interpretations of the facts. They are particularly odious to the ideals of free society and open debate when they are targeted against independent voices and minority viewpoints, with the goal of protecting the prevailing groupthink from having to do the hard work of either defending itself or persuading others.

Two plus two equals four, the calculator tells you every time you put in that equation. The word “fact” is spelled F-A-C-T, every time you look it up in the dictionary. These are facts, and checking them produces the same result every time. This turns out to be very different from the business of “fact-checking.”

AUTHOR

Joshua Arnold

Joshua Arnold is a senior writer at The Washington Stand.

EDITORS NOTE: This Washington Stand column is republished with permission. All rights reserved. ©2024 Family Research Council.

The Washington Stand is Family Research Council’s outlet for news and commentary from a biblical worldview. The Washington Stand is based in Washington, D.C. and is published by FRC, whose mission is to advance faith, family, and freedom in public policy and the culture from a biblical worldview. We invite you to stand with us by partnering with FRC.

---