Guest post by S. Stanley Young and Warren Kindzierski
Back in April WUWT reposted several articles from Dr. Judith Curry’s Climate Etc. website on How we fool ourselves. One of the articles – Part II: Scientific consensus building – pointed out that… ‘researcher degrees of freedom’… allows for researchers to extract statistical significance or other meaningful information out of almost any data set. This past week WUWT reposted one of our articles from Climate Etc. about how epidemiologists try to fool us with flawed statistical practices.
The elephant in the room that drives these types of discussions is the irreproducibility crisis that afflicts modern science. This is no trivial matter. We believe this problem has – over time – infected a broad range of government regulation and policy that has enormously harmful implications to the American public.
These are the questions people should be asking… How many federal regulations reflect irreproducible, flawed, and unsound research? How many grant dollars have funded irreproducible research? In short, how many government regulations based on irreproducible claims harm the common good?
Irreproducible science
Academic researchers, grantors, and journals (and peer reviewers) all play a part in contributing to the irreproducibility crisis. We previously revealed that academic researcher incentives reward exciting research with new positive (significant association) claims—but not reproducible research. This generally encourages scientists in academia to wittingly or negligently use a variety of questionable research practices to produce claims that cannot be replicated.
Well-published university researchers earn tenure, promotion, lateral moves to more prestigious universities, salary increases, grants, professional reputation, and public esteem—above all, from publishing exciting research. The National Academy of Sciences, Engineering and Medicine (NASEM) recently reported on a survey of academic researchers. NASEM indicated that researchers themselves believe they are responsible for addressing issues of reproducibility, but that a supportive institutional infrastructure (e.g., training, mentoring, funding and publishing) is needed.
That is all we need, more tax dollars used to fund academics so that they can learn how to do the job they are supposed to know how to do. What does that tell you about the current academic mentoring process? Do not look to university administrations to want to help solve the irreproducibility problem. American university administrations want to host grant-winning research, from which they profit by receiving overhead costs—frequently a majority of overall research grant costs.
The irreproducibility problem extends outside of academia. Our experience is that there will be little incentive to change current academic researcher practices unless there is also a commitment to change how grantors and journal editors (and reviewers) conduct their end of the research publication process.
Grantors (funding agencies and research sponsors) tend to emphasis novelty of research. These organizations want to fund the same sort of exciting research. Plus, government funders possess the added incentive that exciting research with positive results also supports the expansion of their organizational mission. The downside is that this encourages many researchers to further fiddle with their research until they come up with a publishable, novel claim (publish or perish) that still will not replicate.
Journals too seek novelty in research in part because of the competition for impact factors. An impact factor is a measure of the citation frequency, which offers a quantitative metric for evaluating journals and assessing their relative influence in the scientific community. The same incentives that drive academics affect journal editors, who receive acclaim for their journal, and personal reputational awards, by publishing exciting research—even if the research has not been vetted thoroughly. Editors are often rewarded for actions that increase the impact factor of their journal. Research which reports novel findings are more often highly cited and thus contribute to the stature of a journal and greater rewards for journal editors.
Academic researchers, university administrations, grantors, journal editors—each has an incentive to seek out exciting research that draws money, status, and prestige, but few or no incentives to double check their work. Above all, they have little incentive to reproduce the research, to check that an exciting claim holds up—because if it does not, they will lose money, status and prestige.
Compromised US government regulation
As we had previously discussed in our Climate Etc. post, how irreproducible science afflicts government regulation can be revealed by looking at how research is used by the Environmental Protection Agency to regulate outdoor air quality. Over the past 40 years the EPA has continuously considered and imposed increasingly restrictive regulation of air quality.
This is not necessarily bad. The EPA must develop air quality criteria as part of their mandate, informed by expert opinion, and describe their effects singly and in combination on the health and welfare of American citizens. Also, the EPA must set National Ambient Air Quality Standards (NAAQS), as a yardstick by which states and localities can measure their own air quality, and as a legal requirement for enforcement of NAAQS.
However, the debate about whether the EPA should make a particular regulatory decision based on scientific evidence raises questions central to the irreproducibility crisis—data accuracy, research protocols, statistical analyses, publication bias, sponsorship bias, etc. Complicating this are bureaucrats in government who fund research—these bureaucrats depend on regulation to support their existence. Why would they want to acknowledge an irreproducibility crisis if it were to affect their careers?
There is a burden to society that lurks behind the intentions of academic researchers and government bureaucrats who use their research to develop regulation. They collectively skew the process towards justifying increased regulation. The costs of insufficiently substantiated environmental regulation cannot be dismissed.
An example is estimated costs requiring ships to use “cleaner fuel” with less sulfur to reduce sulfur dioxide emissions. The EPA argues that the move to low-sulfur ship fuel could prevent 14,000 deaths each year by 2020 based on epidemiologic research that sulfur dioxide kills. Let’s ignore the large uncertainties around their estimate (which must be taken with a chunk of salt). Further, the EPA estimates a value of statistical life at $7.4 million (in 2006 dollars). Let’s also ignore large uncertainties around this estimate.
Using EPA’s numbers and roughly factoring in 2006−2020 inflation at 28%, health-related benefits are inferred to be more than US$130 billion per year in 2020 (14,000 x $7.4 million x 1.28) for this requirement. This may sound convincing to an environmentalist, but is it real? A growing body of research fails to support the EPA position. In particular, there is research providing evidence that sulfur dioxide in outdoor air is not associated with death, heart attacks, asthma or lung cancer.
Extensive regulatory schemes always come with excess costs. As to the true excess costs requiring ships to use fuel with less sulfur, who know? However, these are not zero. Fuel costs represent as much as 50-60% of total ship operating costs, depending on the type of ship and service. On the other hand, there is a distinct possibility that inferred health-related benefits (i.e., statistical lives saved) from this requirement are meaningless.
The EPA issues an extraordinary number of regulations, which affect every area of the economy and constrict everyday freedoms. Extensive regulatory schemes can amount to a competitive advantage for large companies over small ones. Large companies have greater capacity to comply with an extensive regulatory framework. Furthermore, regulatory costs are ultimately borne by American consumers. These costs can also have negative health implications, such as those that follow from increased unemployment—e.g., depression, substance abuse, domestic violence.
Development of air quality regulation by the EPA can be shown to reflect irreproducible, flawed, and unsound research. How many other federal regulations suffer this fate? That is the trillion-dollar question.
S. Stanley Young is the CEO of CGStat in Raleigh, North Carolina and is the Director of the National Association of Scholars’ Shifting Sands Project. Warren Kindzierski is an Adjunct Professor in the School of Public Health at the University of Alberta in Edmonton, Alberta.
via Watts Up With That?
May 20, 2021