From Watts Up With That?

Guest post by Warren Kindzierski, Stan Young and John Dunn

Summary

We published a study which evaluated gas stove/NO₂−childhood asthma epidemiology research. We observed that this research topic has many falsehoods and hidden biases. Most readers of scientific publications are hardly aware of researchers’ methodological biases we describe in our study unless they happen to be knowledgeable in method limitations of the research topic. But largely they are not. Evidence presented in our study disputes the legitimacy of a gas stove/NO₂−childhood asthma causal claim. The public can consider the gas stove−childhood asthma causal claim unproven.

Background

Discussion has occurred recently on wuwt about the gas stove ban because they emit nitrogen dioxide (NO₂). See New York Gas Stove Ban, Gas Stoves: The Beloved Blue Flame, Attacks on Gas Stoves Aren’t Really about Health.

We published a study which evaluated gas stove/NO₂−childhood asthma epidemiology research. We posted an article, Part I, on the American Council of Science and Health website based on our study. In the article we discussed some falsehoods and hidden biases in gas stove−childhood asthma research. These include:

1. Research on NO₂ has not established that this combustion gas, composed of three non-carbonaceous atoms, is a carbonaceous allergen. As such, a biologically plausible explanation of a causal link with childhood asthma is tentative at best.

• Questionable statistical methods (excessive hypotheses testing) appear routine in NO₂−childhood asthma epidemiology studies. These studies are compromised with unknown false-positive results. This may be, in fact, a characteristic of the NO₂−childhood asthma epidemiology literature.
• A p-value plot for a gas stove/NO₂−childhood asthma association that we constructed from a set of test statistics in the meta-analysis support randomness – this is not consistent with proof that gas stoves are associated with or cause asthma or respiratory harm to children.

An honest scientist or well-read person with common sense might ask the question… Why is there even discussion of a gas stove/NO₂−childhood asthma causal claim if NO₂ is not a biologically plausible explanation of a causal link with childhood asthma?

The answer is simple… Academic researchers continue to carry on as if it is fact that NO₂ from gas stoves causes childhood asthma. In this article we point out two additional hidden biases of academics in gas stove−childhood asthma research… selective reporting and p-hacking.

Selective reporting

Academic researchers have flexibility to use different methods during a study. They also have flexibility to only report those methods that yield positive results and disregard those that yield unfavorable results. This reporting preference involves a selective bias to highlight statistically significant findings and to avoid nonsignificant (null) findings in research. What is reported can be challenging for policy makers to use because the significant findings may turn out to be nothing more than false positive (chance) results.

Our study gives three examples of selective reporting in published literature specific to the gas stove/NO₂−childhood asthma causal claim. One notable example that we discuss is of a well-published, senior academic researcher from Utrecht University, Utrecht, The Netherlands (Bert Brunekreef) with more than 850 publications and 89,000 citations to date.

Brunekreef was at the same time involved in two epidemiology research studies back in 2013 – the ISAAC study and a gas stove meta-analysis.

ISAAC study – Brunekreef, along other researchers across the world, investigated the association between asthma and use of various cooking fuels, including gas stoves, as part of the International Study of Asthma and Allergies in Childhood (ISAAC). ISAAC researchers collected data on 512,707 primary and secondary school children from 108 cities in 47 countries between 1999 and 2004.

The ISAAC researchers examined and presented results for two statistical analysis models for analyzing their data for gas stove−childhood asthma associations. Also, they examined two asthma outcomes (‘current symptoms of severe asthma’ and ‘had asthma ever’) for two age groups (6–7-year-olds and 13–14 year-olds) in these statistical models.

The ISAAC study was published 31 May 2013. All their model results were non-significant. ISAAC researchers, including Brunekreef, stated in their Abstract: “we detected no evidence of an association between the use of gas as a cooking fuel and either asthma symptoms or asthma diagnosis”.

Gas stove meta-analysis – At the very same time of the ISAAC study, Brunekreef along two different researchers, conducted a meta-analysis to quantify the association of indoor NO₂ and gas stove cooking with childhood asthma and wheeze.

For a gas cooking−current asthma association endpoint, they combined odds ratios (ORs) and confidence intervals (CIs) from 11 epidemiology studies. They declared their result significant (OR = 1.42; 95% confidence interval, CI = 1.23–1.64).

The meta-analysis was published 20 August 2013, almost three months after the ISAAC study. The meta-analysis researchers, including Brunekreef, stated in their Abstract: “This meta-analysis provides quantitative evidence that, in children, gas cooking increases the risk of asthma and indoor NO₂ increases the risk of current wheeze”.

This gas stove meta-analysis made no mention of the large, international ISAAC study and its null findings. Why?

p-hacking

p-hacking is a form of multiple hypothesis testing involving the search for significance during statistical analysis of a data set. A layman’s description of p-hacking would be statistical cheating to get the answer you want. A decent discussion of different p-hacking strategies that academic researchers use is provided in a recent open publication.

p-hacking is widespread in scientific literature. An advanced Google Scholar search of “p-hacking OR p-hack OR p-hacked” for the period 2013−2023 returns 12,600 results excluding citations. Statistical experts know about p-hacking, but they appear to be largely silent.

Of interest to the gas stove/NO₂−childhood asthma causal claim is a p-hacking strategy involving re-analysis on subsets of a data set in a multiple testing environment. When a data set is subdivided into subgroups and if a particular subgroup is re-analyzed, it can easily find, via diligent searching, chance statistically significant results. If you roll a pair of dice enough times, snake eyes will show up.

As explained in our study, the very same 2013 gas stove meta-analysis discussed above showed evidence of p-hacking:

• The meta-analysis researchers initially examined eight different cause−effect associations (endpoints) in their meta-analysis related to gas stove/NO₂ and asthma and wheeze endpoints using the data set they put together.
• The researchers then conducted additional statistical re-analysis on a subset of their data using the same methods. This re-analysis was only on six of the eight endpoints.
• They then undertook further statistical re-analysis on a different subset of their data using the same methods, this time focusing on just three of the eight endpoints.

Others have noted that in epidemiology investigations “there is an increased risk of false claims of effect modification when several subgroup analyses are explored” … and … “there is a need to exercise restraint, viewing subgroup findings as exploratory and hypothesis generating rather than definitive”.

Discussion

Collectively, evidence presented in our study disputes the legitimacy of a gas stove/NO₂−childhood asthma causal claim. The public can consider the gas stove−childhood asthma causal claim unproven based on the following:

• Research has not established NO₂ as a biologically plausible explanation of a causal link with childhood asthma.
• Questionable statistical methods (excessive hypotheses testing) were common in epidemiology studies combined a 2013 gas stove meta-analysis that we evaluated.
• A p-value plot for a gas stove−childhood asthma association in the 2013 gas stove meta-analysis supported randomness. A random finding is not consistent with proof that gas stoves cause asthma or and respiratory harm to children.
• p-hacking – specifically, re-analysis of data subsets (a questionable research practice) – is evident in the 2013 gas stove meta-analysis.
• Selective reporting (another questionable research practice) is evident in published studies and reviews of gas stove–childhood respiratory health studies.

The majority of published epidemiology research and reviews examined in our study were co-authored by academics. The role of academics in gas stove−childhood asthma research requires further commentary.

Most of the falsehoods and hidden biases in published academic research can be traced back to their methodological biases. Other have referred to these as researcher flexibility or researcher degrees of freedom. In essence, too many academics do what they want during research and then report what they want in a paper submitted for publication. All that is necessary is that they carefully write up a ‘tight scientific story’ to support their research finding to get it published.

Most readers of scientific publications are hardly aware of methodological biases we describe in our study unless they happen to be knowledgeable in method limitations of the research topic (i.e., gas stoves and childhood asthma). But largely they are not. Also concerning is the circumstances of policy makers and media recommending/supporting gas stove bans. They can hardly be expected to have any less knowledge deficit about method limitations and biases in gas stove−childhood asthma research.

As for academics, Ian Oxnevad of the National Association of Scholars describes perfectly the effect that they have on society… “People need to know that academia is not an innocuous place” … “[Academia] is the epicenter for just about everything bad that you’re seeing going on right now.”

Warren Kindzierski is a retired college professor in St Albert, Alberta. Stanley Young is CEO of CGStat in Raleigh, North Carolina and is Director of the National Association of Scholars’ Shifting Sands Project. John Dunn is a lawyer and emergency physician (retired) in Brownwood, Texas.