How To Spot Faulty Statistics

Five questions to ask yourself before taking that number at face value

By DomesticShelters.org
Dec 18, 2017

If you’ve ever read any news story—including those on this site—or listened to an impassioned speech or heard a politician talk, ever, then you’ve heard or read a statistic. One in four, 20 percent, more than half, 6 million … the numbers being thrown around are meant to grab your attention, quickly put into perspective the issue and make you exclaim, “Really? Wow, I had no idea it was that much.”

But, if you’re paying attention, you probably also noticed that statistics, quite often, contradict one another. They also often conveniently fit the rhetoric of whatever story or speech they’re accompanying.

It’s like the old joke: 74 percent of statistics are made up on the spot.

Questioning statistics, said Mona Chalabi in her February 2017 TED Talk on stats, “doesn’t make you some kind of crazy conspiracy theorist, it makes you skeptical. And when it comes to numbers, especially now, you should be skeptical.”

Before you take any statistic at face value, you might want to ask yourself the following questions:

1. Does the statistic come from a reliable source?

Consider whether or not the statistic you’ve just read or heard is backed up by the words, “According to,” as in, “According to the Department of Justice,” or “According to the Centers for Disease Control and Prevention.” If so, check to see if there’s a link to the original study. Does the stat match up to the research provided?

If the stat stands on its own with no attribution, a quick Google search should tell you if that statistic comes from an actual research study or validated survey. If you can’t find one, you may want to file it under the category of “Questionable At Best.”

2. Is the sample size of the study or survey too limited?

It’s important to look at the sample size of survey, poll or study when determining if the statistic is valid. If the sample size is too small (they polled 100 people) or too limited (they only talked to people in one state), and then they applied those findings to claim the majority of the U.S. population believes something, that should raise a red flag.

“The larger the sample, the greater the reliability,” says Barry Goldstein, advocate, former attorney and author of The Quincy Solution: Stop Domestic Violence and Save $500 Billion.

3. Is there a bias or agenda at play?

This might come as a shock, but a lot of statistics are created or incorrectly regurgitated to serve someone’s biased agenda. Crazy thought, right?

“Certainly, you want to see if the people [who created the stat] had an ax to grind,” says Goldstein. Are they trying to prove a specific point, or are they objectively looking at the outcome of a survey? Do they have a financial motive? Is the company doing the research trying to sell you something?

4. Is the study or survey outdated?

Find out when the study was conducted that the stat came from. If it’s a decade old, for instance, ask yourself if there is anything that’s happened since then that would change the outcome of that stat, such as a change in the laws or additional scientific or medical research that has since had an impact. Think about the fact that our viewpoints 20 years ago on any number of issues are likely not the same as they are today.

5. Can you spot logical fallacies?

Fallacies are a type of illegitimate reasoning people use to make a point. One of the most common is known as “post hoc ergo propter hoc,” or if “A” occurred after “B,” then “A” must have caused “B.” For instance, saying that abusive partners are more violent after drinking means that drinking must cause abuse is a fallacy. There is no evidence that abuse is caused by drinking (though alcohol can definitely perpetuate and/or escalate abusive behaviors already present).

A Disparity in Domestic Violence Stats

On this site, you’ll see many numbers used to try and paint an accurate picture of domestic violence. And elsewhere, you may read numbers that contradict those we’ve compiled. It can be maddening to try and find an agreement in just how many people are victims of domestic violence every year and how the demographics of those victims break down. Why is there such a disparity?

“One of the things about domestic violence is it’s the most underreported crime, so you have to be careful about studies that are based on reports,” says Goldstein. For instance, would a survey of 1,000 homes in the U.S. really reveal a true percentage of how many families are dealing with domestic violence?

“If they call and the woman answers, and her abusive partner is nearby, she can’t talk about it,” says Goldstein. “So she’ll say nothing is happening, and they’re going to get the wrong information.” Concurrently, if the man answers, says Goldstein, “If he wants to lie, he’ll just go ahead and do it.”

Looking at police reports will only reveal the number of cases in which police were called—countless other incidents of domestic abuse happen in private, behind closed doors, with no involvement by law enforcement. Surveying court records or convictions for domestic abuse will yield similar skewed results—even if a survivor calls police, she may be coerced or threatened by her abuser to drop the charges.

Look for studies that go right to the source, says Goldstein, such as polls of survivors in domestic violence shelters or findings from emergency rooms. Here, you’re more likely to get an accurate reading as they’re safe places for survivors to speak out.

Where Can You Find Reliable Statistics?

You want the numbers, but you don’t want to be led astray. Goldstein says look for government research—the CDC, Department of Justice, National Institutes of Health, among many others. “In general, that’s a good source. The studies are likely to be held to high standards, be peer-reviewed and done by people who have the expertise needed for the subject.”

He says academic research—studies done by colleges and universities—also tends to be reliable, but when it comes to domestic violence, specifically, he says, “Sometimes, the researchers don’t have domestic violence education needed.” For domestic violence statistics, he recommends looking at well-established domestic violence organizations with years of experience.

Here are a couple of places to start for reliable statistics:

Domestic Violence