Reviewing Their Peers
Photos courtesy of CoEHAR
CoEHAR’s REPLICA Project evaluates nicotine-related science to ensure that policymakers and the public receive correct and reliable information.
By Stefanie Rossel
Nicotine products, stakeholders agree, should be regulated based on scientific evidence. Unfortunately, an increasing volume of research underpinning policy appears to be based on questionable science, where researchers are unable to replicate the results. This “reproducibility crisis” occurs across various disciplines, including psychology, medicine and natural sciences. The phrase was coined in the early 2010s, and efforts to remedy the problem have led to the creation of a new scientific discipline, meta science.
To tackle the reproducibility crisis in the nicotine sector, the Center of Excellence for the Acceleration of Harm Reduction (CoEHAR) in November 2019 launched the REPLICA Project, an initiative to independently validate the results of e-cigarettes and heated-tobacco products (HTP) studies.
“Annually, billions of dollars are poured into biomedical research, yet more than half of these studies cannot be replicated due to flawed experimental designs, inadequate methodologies and faulty statistical analyses,” explains Riccardo Polosa, the founder of the CoEHAR, who also is a full professor of internal medicine in the Department of Clinical and Experimental Medicine of the University of Catania in Italy. “Bad science holds back medical progress,” he says. “The challenge of replicability is widespread, spanning numerous scientific domains, including the field of THR.”
According to Polosa, the problem is amplified due to the contentious nature of tobacco and nicotine. “Often, the focus seems more centered on perpetuating controversy rather than prioritizing public health concerns,” he says. “The unfortunate reality is that there appears to be minimal interest in resolving the replicability crisis within the field of THR science, as it may conflict with the mainstream anti-THR narrative.”
Unique Approach
The REPLICA Project replicates published in vitro toxicity studies with the help of seven research laboratories spread over various countries, including the U.S., Indonesia and Serbia. According to Polosa, this approach is unique. “Its success hinges on complete harmonization between these research labs, necessitating ongoing international training, adherence to standard operating procedures and uniformity in lab equipment/material,” he says.
“We uphold the highest tobacco research standards, employing ISO-regulated setups for smoking/vaping laboratories and approved standardized smoking/vaping machines in our replication studies. While you cannot directly compare the REPLICA’s approach with the individual studies we replicate, many of the studies we aim to reproduce lack adherence to the best tobacco research standards. They often overlook smoking/vaping laboratory standards regulated by ISO guidelines and resort to unregulated homemade smoke/vape machines. Our methodology aligns with the open science principle—in contrast to the works we assess, which frequently present methodological ambiguities and sometimes obscure interpretations of data.”
According to Polosa, the advantage of REPLICA’s cooperation model lies in the consensus reached when results from one laboratory align with those from others. This, he says, ensures a more reliable basis for new knowledge. Replication, Polosa adds, bolsters confidence in scientists’ findings.
“Furthermore, this networking model of multiple research labs united in pursuing a common objective fosters unique opportunities for knowledge sharing and building capacity, thereby ensuring elevated standards in THR research,” says Polosa. “In line with [the] CoEHAR vision and mission, nascent labs in low-[income] and middle-income countries can connect with the REPLICA consortium, providing them with the chance to thrive. Specifically, young researchers from Serbia and Indonesia have been offered CoEHAR mobility training programs at the University of Catania under the guidance of REPLICA researchers.”
Promoting Understanding
In the first phase of the REPLICA Project, which ran until 2022, researchers replicated seven studies. The project initially focused on independently verifying science published by the tobacco industry. As it gained expertise, the project also began evaluating papers published by academia in scientific journals.
“Each decision to replicate a study revolves around identifying topics vital for better understanding of the THR paradigm and for improved protection of human health,” explains Polosa. “In addition, feasibility considerations for all partners within the study’s framework become paramount. Although consideration about studies that appear to be flawed are secondary, we occasionally opt for studies that are partially well-executed, recalibrating inadequately set parameters. This meticulous approach aims to inform the scientific community, regulatory bodies and lay public with credible and comparable data that mirrors the normal condition of use of combustion-free nicotine products.”
In several studies, the REPLICA Project researchers were able to replicate outcomes. “Specifically, our investigations revealed consistent evidence indicating the reduced capacity of e-cigarettes and HTPs to induce cytotoxicity, inflammation, endothelial damage, genotoxicity and mutagenesis in human lung epithelial and endothelial cells,” says Polosa.
Other studies revealed irregularities, however. “We’ve pinpointed noteworthy methodological flaws in at least three recently replicated studies,” says Polosa. “For instance, one study encountered issues due to an inadequately supplemented culture medium, resulting into abnormal cytotoxicity in cells. In another instance, researchers used incorrect fluorochromes, hindering the clear distinction of adhesion events of inflammatory cells to the vascular bed. Lastly, flavors were vaporized at excessively high temperatures using a furnace instead of the intended vaping machine under standard conditions of use. These identified flaws are either in the process of being addressed or have been thoroughly examined and will be disclosed in forthcoming peer-reviewed publications.”
According to Polosa, the reported shortcomings relate not only to scientists’ mistakes but also the editors and reviewers who approved the publication of low-quality research.
Providing Accurate Information
Polosa says that CoEHAR’s researchers actively seek out the original authors to discuss methodological shortcomings and to request additional information with the goal of establishing productive collaboration and enhance the replication protocol. “Unfortunately, this is not as straightforward as it should be,” he says. “Previously, we encountered situations where authors offered only partial responses to our concerns, leading to increased confusion regarding their ‘irregular’ approaches in their scientific work.
“Regrettably, this lack of comprehensive information resulted in the abandonment of replication attempts in most cases. Often, eminent scholars in this field, despite their expertise, unintentionally make errors in their study approaches. However, some may hesitate to acknowledge or fail to grasp these errors. Through focused scientific engagement, our aim is to educate researchers, especially those new to the THR paradigm, facilitating a deeper comprehension of these intricacies.”
While the CoEHAR team hasn’t encountered many studies that warrant retraction, there have been instances that would justify such a course of action, like the most recent one investigated, which has not been published yet. “We’re considering this course of action pending discussions with the original author,” says Polosa. “It’s important to note that while some studies may not be inherently flawed, they might be inadequately aligned with the subject matter. For example, certain studies might provide valuable insights in other areas but fall short in their relevance to evaluating the effects of vaping. When the study setting is drastically different from the context of vaping, its conclusions can’t be responsibly applied to this specific practice. It’s akin to studying the impact of a bicycle accident at 350 km/h and then advocating to ban bicycles due to their perceived risk, a distortion of reality that fails to accurately address the intended subject.”
The project’s foremost objective, Polosa insists, is to provide accurate information to the public rather than preserving the reputation of the scientists behind the original studies. “While retractions aren’t taken lightly and involve careful consideration, our obligation remains toward ensuring the public and policymakers receive correct and reliable information.”
Phase Two
CoEHAR is funded by the Foundation for a Smoke-Free World (FSFW), which was financed exclusively by Philip Morris International until Oct. 20, 2023, when the organization announced it would no longer accept tobacco funding in an effort to address concerns about is independence. Polosa emphasizes that credibility of his team’s work is not an issue if REPLICA studies debunk original science. “Although we’re relatively new, our credibility is steadily building,” he says. “The strength of CoEHAR’s REPLICA project rests in its robust, high-quality research, consistently published in reputable scientific journals. Despite funding ties with FSFW, our project has swiftly garnered a positive reputation worldwide. This funding association doesn’t diminish our research’s significance. REPLICA’s growing scientific output unequivocally proves this point.”
Increased funding should be directed toward replication studies of THR products, notes Polosa. “Stakeholders need to commit additional resources to this vital research area,” he says. “We are grateful to the FSFW that has recognized the importance of replicability in science and has generously provided cutting-edge equipment and other resources, supporting our mission to reinvigorate replication science in THR.”
Engaging with media outlets to communicate findings from replication studies is an essential aspect of the project. “However, a significant challenge arises when the original study’s results were sensationalized, as there might be little motivation to rectify misinformation if the correction lacks the same attention-grabbing quality,” says Polosa.
He cites the example of the 2019 EVALI crisis in the United States, when an initially mysterious outbreak of lung injuries was at first erroneously attributed to nicotine vaping but later turned out to have been caused by adulterated (and illegal) THC products. Once the original, sensationalized narrative had spread, it became difficult to correct.
“Similarly, in the realm of in vitro toxicological studies investigated by the REPLICA researchers, algorithms often prioritize popularity over accuracy, complicating efforts to disseminate corrected information effectively,” say Polosa. “Despite these hurdles, our commitment remains steadfast in sharing our findings extensively with the public, academia and relevant stakeholders.
“While our replication efforts are starting to gain traction in shaping public opinion and influencing regulatory bodies, the primary goal of our studies is to assess ongoing and future research and prevent recurring errors within THR science. Ultimately, our aim through this correction process is to dispel substantial misconceptions about the health risks associated with combustion-free nicotine products. As we progress, we anticipate a reduction in sensationalized and inaccurate reporting by the media, leading to a more informed and balanced understanding among the public and policymakers.”
In 2023, the REPLICA Project entered its second phase, which among other things aims to align the project’s studies with academic research while elevating the complexity level. “Our focus is on replicating and expanding studies conducted in collaboration with partners,” says Polosa.
“Specifically, we’ll conduct thorough screenings for cytotoxicity and mutagenicity, examining increasing concentrations of most common vaping flavors. These results aim to ascertain the safety of these flavors and their concentrations for inhalation. Within our REPLICA 2.0 project, we aim to shed light on the potential impact of nicotine on cancer development. Employing a comprehensive approach, we’ll utilize various assays to screen for carcinogenicity. Regulatory bodies like the U.S. Environmental Protection Agency and the International Agency for Research on Cancer heavily rely on these models’ data to classify substances and make informed regulatory decisions.”
