Journalists Should Do More Digging When It Comes to Quoting Studies

“A new study…”

There are many problems in the world of journalism worth attacking—fake news, a lack of minority representation, how to adequately fund robust digital reporting. But a festering problem that exists in newsrooms across the country is our continued willingness to promote and share less-than-scientific surveys for the sake of social media success.

Back in May, many editors were pitched what likely appeared to be a fun study that purported to show that half of the adults in America have used the pool as a substitute for taking a shower.

Funny. Gross. Relatable. Unlikely to offend.

Not surprisingly, a number of high-profile outlets took the bait. CBS News, the New York Daily News, the Detroit Free Press, the Times-Picayune—they all ran stories with some variation of the headline, “Half of Americans use swimming pool in lieu of shower.” A quick Google search is littered with video reports that ended up on scores of local news shows across the country, likely seen by millions of people.

But odds are good the target of most of these stories was social media—specifically Facebook—which has the might and algorithmic know-how to reward stories like this with massive amounts of traffic. The New York Daily News’ version of the story was shared more than 2,000 times on Facebook, according to CrowdTangle. CBS News garnered nearly 800 shares for its version of the story.

Unfortunately, editors and writers looking for a quick hit didn’t take the time to read the fine print of the study. Jessica Huseman, a ProPublica politics reporter and adjunct professor at the Columbia Journalism School, did some quick digging and discovered the unscientific survey of just 3,100 people was conducted online by Sachs Media Group, a public relations firm that counts the chlorine industry among its clients.

Included in the survey is an offer for free chlorine test strips from the website of the Water Quality & Health Council, which Huseman revealed is sponsored by the Chlorine Chemistry Division of the American Chemistry Council. Even WebMD wrote up the survey as a story, which it later took down after Huseman’s 2 a.m. tweetstorm.

“After reviewing the source, we have decided to take down the article on our site. Thank you for flagging,” WebMD wrote in response to Huseman’s findings. CBS News added an editor’s note to the top of their story, noting that an earlier version “failed to report that the PR firm that conducted the survey was working on behalf of the chlorine industry.”

“I did more reporting on this survey on my phone in 10 min at 2 a.m. than any of these reporters who got paid to write that,” Huseman wrote on Twitter, adding that if the results of a poll “seem weird” or ask questions no one really cares about, then chances are good “it’s a crap poll or industry sponsored.”

Unfortunately, the attention Huseman was able to garner on Twitter likely reached just a small fraction of the people who read, shared or liked the pool story without even giving it a second thought.

“Questions can be phrased in ways to tilt results the way a group wants,” NPR host Scott Simon said about the chlorine incident on a recent episode of Weekend Edition. “But professional pollsters who work with news organizations have an interest in offering surveys that illuminate the range of diverging views people can hold in a huge and varied country, regardless of results.”

Ironically, the same week this survey was released and quickly disseminated by a number of national news outlets, the Centers for Disease Control and Prevention released an overlooked report that revealed pool chemical injuries led to an estimated 13,508 emergency department visits in the U.S. between 2015 and 2017. According to the CDC, most of those injuries occurred at a private residence, and two thirds happened between Memorial Day weekend and Labor Day.

The bathing in the pool survey got a lot of attention, but wasn’t alone in its ability to dupe reporters. In May, many of the same national outlets also ran with stories based on an unsourced survey purporting to rank the sexiest accents in the U.S.

Nevermind that the original post appeared uncredited on a website called Big 7, which describes itself as a Dubai-based travel website. Big 7 didn’t source the methods or questions for its survey (other than to say it asked its purported 1.5 million social media followers) and appears to work on behalf of travel resorts and cities willing to pay for sponsorships.

The list appears designed to grab the attention of the nation’s top media markets with a ready-made local angle that doesn’t have any association with divisive topics like politics or Donald Trump. Not surprisingly, the random post garnered news coverage from outlets across the country, as varied as Marketwatch to the Charlotte Observer and nearly everything in between.

(Seriously, when you finish reading this column, Google “sexiest accent” and just check out the hundreds of outlets who wrote it up for their own unique audience.)

Is this the Pentagon Papers? Hardly. Watergate? Not really. But it reveals the problems that occur when downsized newsrooms are forced to write with a social media audience in mind and traffic quotas hanging over their heads. Standards suddenly slip away, and we quickly aggregate studies done with inappropriately small sample sizes or unprofessional polling because it’s a slow day and we need to meet our post quota.

“There is no pressure on the media to focus on reporting the ‘best’ studies, since people have a hard time separating the good studies from less-good ones,” Emily Oster, a professor of economics at Brown University, wrote in her 2019 book “Cribsheet: A Data-Driven Guide to Better, More Relaxed Parenting, from Birth to Preschool”

“Media reports can get away with saying ‘A new study shows…’ without saying ‘A new study, with very likely biased results, shows…’ ” Oster wrote.   “And other than the few of us who get our dander up on Twitter, people are mostly none the wiser.”

So how can staff writers better guard against unintentionally promoting a bogus or biased survey? Huseman offered three simple suggestions to writers tempted by the next too-good-to-be-true pitch:

• Look at the actual questions.

• Find out who funded the poll.

• Do not just rewrite a press release about a survey.

Seems simple enough, right? If not, expect to hear some complaints on Twitter…and little else.

Rob Tornoe is a cartoonist and columnist for Editor and Publisher, where he writes about trends in digital media. He is also a digital editor and writer for the Philadelphia Inquirer. Reach him at robtornoe@gmail.com.