Researchers have created a star-based metric that rates the quality of the evidence for a link between a given behavior, such as eating red meat, and a particular health outcome.Credit: Educational Images/Universal Images Group/Getty
Does eating red meat reduce life expectancy? Some researchers certainly think so. Works such as the Study of the Global Burden of Diseases, Injuries and Risk Factors1 has led the World Health Organization and the US Department of Agriculture to recommend that people limit their consumption of unprocessed red meat, to protect themselves from diseases such as type 2 diabetes and several types of cancer.
Other researchers are less sure. Targets for red meat consumption, set by public health officials and expert panels, vary widely, with some advising that people eat no more than 14 grams per day and others setting no recommended limit. This sends a mixed message, which in itself is not good for public health.
It’s not just about red meat: The evidence base surrounding much broader health and nutritional advice is equally contested. Now, a new approach could help health policymakers better assess the quality of studies assessing potential health risks. A team at the Institute for Health Metrics and Evaluation (IHME) at the University of Washington in Seattle has created a star-based metric that rates the quality of the evidence for a link between a given behavior, such as eating red meat or smoking. and a particular health outcometwo. A five-star rating means the link is clearly established; a star means that there is no association between the two factors or that the evidence is too weak to draw a firm conclusion.
The burden of test studies
What the researchers call “burden of proof” analysis doesn’t, by itself, clarify puzzling questions like the risks of red meat or the benefits of vegetables. But as a judgment on the quality of the available research, it can help point research funders to areas where better evidence is needed to draw firmer conclusions.
How is the star rating constructed? What are its parameters? Can the methodology itself be considered rigorous research? The IHME team did several things to try to quantify the effects of various biases in the studies they evaluated. An epidemiological study, for example, might be biased differently than a study testing the outcomes of health interventions. The researchers also removed what can be a common source of research bias, namely the assumption that health risks increase exponentially with the parameter being studied, for example, blood pressure or red meat consumption. without processing. And they tried to account for the bias that can arise when sample sizes are small.
Applying this framework to studies that evaluated a total of 180 questions produced results that are mostly not surprising. Studies evaluating an association between smoking and a variety of cancers, for example, get a five-star rating.3. Similarly, high systolic blood pressure, the force the heart exerts to pump blood, has a five-star association with narrowing of the blood vessels called ischemic heart disease.4.
Studies evaluating diet and its health outcomes get markedly lower star ratings. The IHME analysis, for example, finds only weak evidence of an association between unprocessed red meat consumption and outcomes such as colorectal cancer, type 2 diabetes, and ischemic heart disease.5. It finds no relationship in studies exploring whether eating unprocessed red meat leads to two types of stroke. There is stronger, but not overwhelming, evidence that eating vegetables reduces the risk of stroke and ischemic heart disease6.
Let’s go beyond the rhetoric: it’s time to change the way we judge research
In some cases, lower star ratings could be due to effect size: for example, any health risk from eating red meat is likely to be small relative to the huge toll smoking has on the body . Overall, the lower-rated findings show that studies in these areas need to improve if they are to yield convincing results.
It is difficult to separate the effect of a single dietary component from the complex array of exposures throughout a person’s lifetime. Larger studies, with a diverse group of participants and tight control over their daily diet, would be needed. Such studies will involve collaboration between research groups with different backgrounds and access to participants in different environmental settings, a move that funders should encourage. This is a company worth prioritizing. A small risk to an individual does not mean a small impact on public health: a low-risk behavior can have a large impact at the population level if it is very common.
The literature in the field of responsible research and innovation highlights how metrics in science must always be challenged for their robustness and rigor. There should be wide consultation and, to the extent possible, unintended consequences of the use of metrics should be anticipated, as initiatives such as the San Francisco Declaration on Research Evaluation and Leiden Manifesto to show. This job should come sooner rather than later.
We have evidence that underpowered clinical trials, which lack the necessary controls to make sense of the data, are not helping. If funders don’t focus their efforts on producing quality data, the public will remain confused, tired, distrustful, and deprived of the information they need to make informed decisions about their health and lifestyle.
