Meta-Analysis

Method Categorisation:
Quantitative - Qualitative
Deductive - Inductive
Individual - System - Global
Past - Present - Future

In short:In a Meta-Analysis, statistically comparable results from a range of scientific studies are integratively summarized to gain aggregated, quantitative insights into scientific knowledge on a specific issue.

Background

SCOPUS hits per year for Meta-Analysis until 2019. Search term: 'meta-analysis' in Title, Abstract, Keywords. Source: own.

Statistician Karl Pearson had a strong focus on openly integrating knowledge into overarching results, and in this spirit he created one of the first systematic Meta-Analyses by integrating results from several comparable studies on typhoid into one of the first Meta-Analyses in 1904 (1, 2). It was more than 35 years later that within psychology the next relevant Meta-Analysis was published: the book-length "Extrasensory Perception After Sixty Years", authored by Pratt et al. in 1940, summarizing experimental results from 1882 to 1939 (1). Despite some few studies being compiled over the next decades (most based on clinical research), more sophisticated statistical analysis tools emerging after WW2 ultimately sealed the breakthrough of the method. In the 1970s, Gene V. Glass - together with many other statisticians - developed what he called the 'analysis of the analyses' (2). By integrating statistical results into a meta-analytical frame, it became possible to combine results from the at the time already rising number of publications. Another key development was the availability of modern computers, which allowed for the conducting pf the more sophisticated analyses, along with more studies becoming available online with the rise of the Internet. Meta-analysis thus represents a method where the demand for knowledge of a different order became possible because the general data basis increased, and advanced analyses were made possible by developments in statistics and the wider availability of technology. Today, the method is widely established within all branches of normal science, and generates knowledge way beyond individual studies, yet new challenges emerge concerning which studies can be integrated, and how.

What the method does

Regarding methodological design criteria, it is important to note that meta-analyses are deductive, as they try to integrate and test results from designed studies. While these results are clearly, quantitative, it is important to notice that there is a trend of so called qualitative meta-analysis, which we will not include here. The spatial scale depends on the scale of the studies that are integrated here. While hence a meta-analysis can be global in its focus, more often than not do the studies that a meta-analysis is based on an individual scale, such as in medicine and psychology. It is a matter of debate whether meta-analyses are a snapshot of the current state of the art, or a look into the past. To this end, it is important to recognize that the development or changes over time can be implemented into the analysis, and thus be a relevant part of the hypothesis.

How it works

The Meta-Analysis is a well established method that revolves around the systematic identification of relevant studies, integration into a meta-analytical design, and interpretation based on established norms. This process is most established within medical research and psychological science, with extensive protocols being available, while in other scientific disciplines the procedures are more diverse. Meta-Analyses most commonly take place in a Systematic Literature Review, allowing for a quantitative analysis of the results of the gathered publications. The Meta-Analysis is thus not the same as a Systematic Literature Review, but the quantitative process of integrating data within such a process.

In a strict sense, a Meta-Analysis summarizes statistics from scientific studies, with estimates, sample size and levels of significance as an important basis for the meta-analytical framework. Meta-Analyses are thus able to correct for different sample sizes from the respective studies. Equally, other information can be included, such as different sub-groups or random factors. For instance, a meta-analytical scheme in medicine is able to differentiate between studies that included children, adults, or both. Through such random factors, a Meta-Analysis can hence include and integrate information that was not analyzed in the original studies, but that only emerges from the wider analysis of multiple studies. Typical models for such an analysis are Mixed Effect Models, and many advanced statistical software solutions (R, SPSS) are able to conduct a Meta-Analysis. Another important staple of Meta-Analyses are summarizing tables or figures that give an overview of the diversity of studies that were included, as well as both overall effects and mean effects for sub-groups.

Strengths & Challenges

Meta-Analyses are at least as strong (and weak) as the multiple studies that they are based on. Great care needs to be taken when identifying studies that are suitable for a meta-analytical approach, since all studies should be as comparable as possible in terms of their study design. In other words, unexplained variance between studies because of factors that are not taken into account should be avoided.

The analysis of data within a Meta-Analysis demands a clear utilization of both the available data and the usage of a proper statistical approach. Many scientific disciplines have established standards that are robust, but can also be rigid. In the long run, a greater exchange of knowledge on the available approaches would be helpful.

Within rigid deductive ranges of science, Meta-Analysis can serve as a framework to integrate quantitative results. However, it remains a challenge to take more qualitative aspects of research into account in these kinds of analyses. In addition, the integration of inductive studies is often a challenge, as Meta-Analyses build on comparability of studies above all else. However, many Meta-Analyses are not rigid to this end, and there is a tendency to not build on clear hypotheses, but instead apply an inductive statistical data crunching post hoc which is subsequently translated into a pseudo-deductive pretending. Such abductive approaches are clearly a challenge and question the quality of the research in some areas of science. Meta-Analyses need to be safeguarded against scientific misconduct, as these few bad examples endanger the reputation of this method, and science in general.

Normativity

Regarding the interpretation of the results of a Meta-Analysis, many studies are often somewhat prone to not properly indicate the extent and limitations of their results. While this is again a standard in some disciplines, other disciplines integrate studies that are not based on comparable designs, and thus create a Type I error by integrating studies that are not based on the same assumptions. It is less often that studies are excluded that should be included (Type II error), and this is also less relevant since it does not increase the error in the meta-analytical result. The main challenge of Meta-Analysis is hence that the overall procedure is not careful enough, and many Meta-Analyses literally compare apples with oranges. Since the method is considered to be relevant, allowing for the integration of knowledge, studies using Meta-Analyses are increasingly published. It remains to be seen if the knowledge that is thus created is always sound, and whether some branches of science did not overplay their hand. After all, the origins of this method lie in psychology and medicine, where study designs are often more comparable, although the reproducibility crisis psychology even calls this into question.

The prestige of a meta-analytical study is often well rooted in the long-standing career of an expert researcher, and this is probably an ideal case. If an expert in a branch of research oversees the integration of knowledge in a Meta-Analysis, then the balanced and nuanced evaluation of the results is safeguarded. However, sometimes Meta-Analyses are more opportunistic and lack expertise that is traded off for a highly cited paper. Ideally, experts on a topic and statisticians should team up to jointly create a blended piece of work that combines the long experience about a topic with the experience in the application of statistical analysis. Recognizing bias and relevant categories in the baseline studies and taking it into account in the analysis are both of importance, as this decides more than anything about the quality of a Meta-Analysis.

Outlook

Over the last decades, meta-analytical approaches protruded into diverse branches of research. Recognizing the context of individual studies and integrating them into a proper meta-analytical framework and analysis scheme is probably one of the key challenges of science in the 21st Century. The availability of more and more studies, and the development of analysis schemes in statistics call for larger and more diverse research groups that can cover all aspects needed when conducting a Meta-Analysis. Taking the context of individual studies into account is an already recognized challenge in Meta-Analysis, and with the availability of more and more studies as well as data, meta-analytical approaches are likely to grow both in terms of their extent and complexity of the analysis and interpretation in the future.

After an early hype in many quantitative branches of science, sobering revision should try to focus on rigor both in terms of comparability of studies as well as statistical analysis. Meta-Analysis will play a pivotal role in the integration of knowledge in the future, yet if the results are flawed and the wish for a meta-analytical framework is larger than the actual data that can be included, then the research landscape will suffer in the long run. Since Meta-Analyses inform future research and thus play a vital role in terms of agenda setting of future research, the highest ethical standards need to be upheld, as the results and responsibility of a Meta-Analysis can emerge way beyond the sum of the studies that are included. Just as medical research continuously updates the meta-analytical knowledge on relevant topics, this may become a standard procedure in other areas of science as well. Establishing standards in reporting schemes, safeguarding comparability in analysis, and creating the norms that are needed for nuanced interpretations will take a long time. This highlights the importance of a higher level of research way beyond the ever-finer atomization of scientific disciplines. If future research is interconnected, Meta-Analysis can become a guiding principle for knowledge integration.

Key Publications

to be added

References

(1) Wikipedia. Meta-analysis. https://en.wikipedia.org/wiki/Meta-analysis
(2) Booth, A. Sutton, A. Papaioannou, D. 2016. Systematic approaches to a successful literature review. Second Edition. SAGE Publications.

Videos

Grimme prize winner Mai Thi Nguyen-Kim explains Meta-Analysis in German

---

The author of this entry is Henrik von Wehrden.