Systematic Literature Review

Method categorization
Quantitative	Qualitative
Inductive	Deductive
Individual	System	Global
Past	Present	Future

In short: In a Systematic Literature Review, existing publications are systematially analysed to derive an overview of the specific slice of the literature, typically summarizing the state of the art regarding the specific topic.

Background

SCOPUS hits per year for Systematic Literature Review until 2019. Search term: 'systematic literature review' in Title, Abstract, Keywords. Source: own.

With the rise of empirical knowledge in the Enlightenment arose the possibility to synthesize knowledge from different studies into an overview work. The A Treatise on the Scurvy - A Critical and Chronological View of What has been Published on the Subject by James Lind is seen as the first systematic review (3, 5, 6), highlighting the importance of knowledge integration. Another important origin of research synthesis can be traced to the work of 17th Century astronomers who combined data sets from different studies to ammend their own observations (3). Systematic literature reviews gained a vital tool through the work of Karl Pearson, whose work on statistics allowed to compile the results from several datasets into an overview. His 1904 publication - in which he combined 11 studies on typhoid vaccines and highlighted irregularities in the results - can be considered the first Meta-Analysis (3, 5). Meta-Analyses were subsequently applied more commonly during the 20th Century, for example in agriculture (5, 6).

After the Second World War, US social scientists began to recognize the need to review the rising amount of research data and while considering how to reduce bias and enhance reproducibility of systematic reviews (6). This also led to the increasing recognition of qualitative elements. In the 1970s, statistician Gene Glass and colleagues proclaimed Meta-Analyses as a valid procedure for synthesising studies which helped to consolidate the approach (3). However, Systematic Literature Reviews were long viewed as second-class studies within Academia, since they did not yield primary data. This changed during the last decades, partly due to increasing interest in scientific evidence on diverse topics on the part of public policy makers, practitioners and the general public (6).

More recently, due to the emergence of digitalisation and improvements in information storage and retrieval, it became significantly easier to identify, gather and analyze the available research on a specific topic (3). Today, Systematic Literature Reviews are most commonly used in Medicine, in the Social Sciences, Business and Economics, but have found their way into several other disciplines (5).

What the method does

Systematic Literature Reviews exist in a broad variety of types. While the Literature Review may be seen as the overarching term, sub-types include the Meta-Synthesis, the Systematic Review, the Case Survey or the strict Meta-Analysis (7). Their differentiation may be done in terms of the data they summarize (quantitative and/or qualitative) as well as the way they do so (qualitatively or quantitatively) (7). Reviews may also be differentiated according to their focus, goal, perspective, coverage, organization and audience. In this entry, the Systematic Literature Review will be in focus. More information on different sub-types of the method can be found in (3). Further, the approach of Meta-Analyses as the statistical tool of summarizing a variety of studies on a specific topic into one is an important method in this regard. For more on Meta-Analyses, please read the respective Wiki entry.

Definition

A Systematic Literature Review is, in short, a reproducible process in which scientific publications that "contain information, ideas, data and evidence" (Hart 2018, p.13) on a specific topic are gathered; studies that fulfill a previously defined level of quality are selected; and their results and insights are summarized and evaluated. For the researcher, the results from this process provide insights into the current state of research and highlight relevant new directions for (his/her own) further research (1, 2, 3). The term 'systematic' refers to the fact that the process is structured to minimize bias (6) and maximimize reproducibility. Being 'systematic' means being reproducible and goes along with an a priori specified, dedicated research design and an explicit documentation of the steps taken during the research (3, see Normativity).

Source: Hart 2018, p.14

A Systematic Literature Review can be applied as the primary method of a scientific study, but is often also used as a first step in a larger research projector endeavor. In both cases, a review can help:

• recognize what has already been done already regarding a specific research field or topics
• find and resolve conflicts in (seemingly) contradictory studies, and
• identify evolving or even unexplored research topics, questions or new hypotheses for further research.

When used as a preparation to one's own study, a Systematic Literature Review additionally helps the researcher

• identify relevant literature and researchers to consult,
• design appropriate methodological approaches,
• understand important concepts, theories and topics and summarise this knowledge to the reader
• contextualize this research and show why it would answer an open question, often on integration level. (1, 3, 4)

Step by Step

The systematic steps that can be made in a Systematic Literature Review based on the example of an international student driven review. Source: Luederitz et al 2016.

A Systematic Literature Review follows a set of steps that is similar to any scientific research process (1, 3, 4):

1. Planning:

The research is designed by formulating the question and scope of the review (here, the different forms of the review - see above - are of relevance). This can be both in an inductive as well as a deductive way, depending on the focus of the review.

2. Data gathering:

Data (= literature) is searched and acquired. There may exist a wide range of documents of relevance to the research endeavour, which is why the researcher should attempt to become familiar with all related topical fields. The core of reviews are often articles in scientific journals due to their comparable structure and assured quality. However, also books and practitioner articles may be of help (1). The data collection can be done using library catalogues or online search engines and databases (1). Relevant literature can either be found by applying single search terms, but, since these searches can become very broad, more specific search options, combining search terms through AND, NOT or OR, may be of better help. In addition, useful documents should be identified asking colleagues and by scanning the bibliographies of already approved papers until saturation is reached, i.e. no more relevant documents come light (4).

3. Data selection:

Pre-defined quality and selection (= exclusion and inclusion) criteria are applied to the broad range of identified documents to decrease their number. One of four different approaches may be applied (3): exhaustive coverage (citing all relevant literature); exhaustive coverage with selective citation; representative coverage (discussion of works which typify particular groupings in the literature) or coverage of pivotal works. While the existing amount of research makes a truly 'exhaustive' collection rather 'exhausting' for many topics, any selection comes with the danger of bias. The relevance of the documents may be assessed based on reading the whole texts, just the abstracts, just the titles or some combination of these (4).

4. Data synthesis & analysis:

After the documents were selected, the researcher reads them and extracts information that helps to answer the research questions. In the process of synthesizing, mind mapping may be a useful tool which can help organize and understand the concepts and theories used in the documents. The documents may further be arranged in this mind map to support the structuring process of the review (1). The focus and thus the methodological process of the data extraction depend on the goal of the review. For example, a researcher might want to focus on theories or concepts, methodological approaches or different scales and actors that played a role in the specific paper that is being reviewed. Typically, a coding book or procedure should be developed (and recorded) in which the extraction process of information is defined, before all information of interest is sorted into the respective coding categories (4). While some variables can be extracted based on foundational work form previous studies, many parameters are often also extracted in an inductive way.

There are many diverse approaches to summarise and analysed the respective data, and to present it to the reader. Depending on the intended outcome of the review as well as the type of data gathered, these can be quantitative or qualitative.

Qualitative analyses are appropriate for the review of purely qualitative or mixed studies and can be often offer perspectives that go beyond quantitative analyses, for instance concerning deeply normative aspects. The qualitative review revolves around identifying essential themes of the documents and their relationships. Also, contrary findings and contradicting interpretations are of interest. "The goal here, unlike meta-analysis, is to increase the understanding of the phenomena being investigated, not to integrate outcomes and identify factors that covary with outcomes." (4, p.10).

Beside systematic literature reviews there is also another form of reviews, which is often referred to as narrative reviews. These are often conducted in a non-systematic sense, but instead consist of the purely deliberate selection of the literature, typically by an author deeply experienced in the literature, and consist a balanced overview of the available literature. While such reviews are often seen to be more subjective, these reviews can be containing a lot of experience, and were more abundant in the past. These days such narrative reviews is often frowned upon, which is a pity, because there is a difference between knowledge and experience, and such narrative reviews can often offer a lifetime of experience.

Often, a systematic literature review applies a Meta-Analysis in the analysis step. A Meta-Analysis is the statistical process of assessing quantitative, numerical data from separate studies that investigated a specific phenomenon. You might for instance calculate means or variations, and evaluate how specific factors influence specific processes. While Meta-Analyses are most commonly part of a Systematic Literature Review, they are not the same. For more information on this, please refer to the Meta-Analysis entry.

5. Writing & presentation of the review:

Finally, the results of the Systematic Literature Review are compiled into a structured paper. A sample structure for the review as a preface to an original study may look like this (Rowley & Slack 2004, p.38):

• 1. Basic Definitions of key terms
• 2. Why is the subject of interest?
• 3. What research has already been undertaken on the topic, and is there any research on aspects of the topic that this research might investigate?
• 4. A clear summary of the results and new research opportunities that emerge from the literature review. Quotations may be used to underline specific findings from the review.
• In addition, the literature gathered during the review should be listed. Many reviews consists a combination of information on the specific topic, the conceptual foundation, methodological approaches, and relevant scales that are associated to the available literature.

In summary, Systematic Literature Reviews are methods of data gathering - building on primary data from other empirical papers - and analysis. They are inductive because they conclude based on existing literature, but also deductive since they start with theoretical assumptions and a pre-defined thematic scope. They can be qualitative and quantitative, cover very local to global phenomena, and investigate past and present states of the literature, often offering a state of the art of the literature, and thereby suggestions and agenda for future research.

Strengths & Challenges

• A literature review is a helpful starting point for any research: it provides a structured overview of the current knowledge and open questions in the respective field and may lead to new research questions that had not been obvious to the researcher before. The strength here is the systematic nature of the literature review: by not only reading randomly through literature, but instead following a systematic and reproducible approach, the researcher generates a more objective overview of the literature.
• When used in preparation to an original study, "[r]eviewing the literature in a systematic way helps the author to be clear, to build confidence in their work and demonstrate the rigour of their methods." (3, p.9).
• The systematic approach enables the researcher to state conclusions about the strength of available evidence to a specific assumption, not only supporting his/her own subsequent work, but also providing insight into the state of science in a given field to policy makers and other public actors. This way, the Systematic Literature Review may for example also shine light on the effectiveness of programs and policies (3).

Potential mistakes and thus challenges in the review process include (4):

• not clearly relating the findings of the literature review to the researcher's own study
• not taking sufficient time to identify the best (primary!) sources
• not critically reflecting upon the studies' research designs and analysis
• not reporting the search procedures
• reporting isolated statistical results instead of meta-analytic or chi-square methods
• not considering contrary findings or alternative interpretations in synthesizing quantitative literature

Normativity

Connectedness

• The Systematic Literature Review is strongly connected to further methods. As explained before, it is often included as a groundwork in original studies to help justify the research topic, design and methodology (2). "Indeed, the concluding paragraphs of the literature review should lead seamlessly to research propositions and methodologies." (1, p.32)
• The analysis of the gathered studies, as mentioned before, can take place in a quantitative way based on a variety of quantitative approaches. Most notably, the Meta-Analysis is of relevance here.
• Systematic Literature Reviews represent an interesting and important form methodological approach. They allow research on research. If scientific results only cumulated and no one looked at the bigger picture, science would not work. Methods like the literature review are thus crucial for academic processes.

Quality criteria

• "Quality [of the literature review] means appropriate breadth and depth, rigour and consistency, clarity and brevity, and effective analysis and synthesis; in other words, the use of the ideas in the literature to justify the particular approach to the topic, the selection of methods, and demonstration that this research contributes something new." (Hart 2018, p.1f). The quality criteria of objectivity, validity and reliability apply to the Systematic Literature Review as follows:

• Objectivity & Validity: Literature reviews are to some level subjective because the synthesis of the screened literature (as well as the identification and selection of the literature in the first place) is, although to some extent reproducible, still a matter of the reviewer's normative decisions (3). Also, "(...) [a]ll reviews, irrespective of the topic, are written from a particular perspective or standpoint of the reviewer. This perspective often originates from the school of thought, vocation or ideological standpoint in which the reviewer is located. As a consequence, the particularity of the reviewer implies a particular reader. Reviewers usually write with a particular kind of reader in mind: a reader that they might want to influence." (Hart 2018, p.25). It can be said that the more systematically the method is applied, the more potential bias is reduced (3). Special attention may be paid to the 'publication bias', highlighting the fact that often, those studies are preferred for publication that offer new or 'interesting' results, which may influence the review results (3).
• Reliability or auditability is safeguarded by a detailed description of the system of the methodological process. This way, other reviewers following the same procedures under the same conditions should - in theory - find an identical set of articles and come to the same results and conclusions. Being systematic in these steps and reporting on the process improves clarity of what has been done and what has not been done, i.e. why certain documents have been included and why others have not (3). The process description may include:
  • the literature search terms, the date of search, the number of search results and exclusion criteria,
  • a list of the included studies, as well as a description of the synthesis process,
  • a sample description of the data extraction process as well as
  • a description of the applied quality standards (3).

Outlook

In the face of the ever-increasing amount of data gathered on diverse topics, as well as due to the changing role of science in society, there may be a increased interest and opportunity of the public in scientific results that may be summarized through Systematic Literature Reviews. With more and more data becoming available, we do not only ask to integrate existing knowledge to allow for a measured and responsible planning of future research. In addition, we would also suggest that such reviews offer a necessary critical perspective, which may not only generate a future research agenda, but in addition can highlight flaws and biases in past research. As it is already the case in medicine, systematic literature reviews should be continuously or at least regularly be updated to offer the latest finding in an integrated way. This would ideally generate a structured but critical research agenda, and a better integration of knowledge that can be increasingly communicated to the public

Key publications

• Hart, C. 2018. Doing a Literature Review. SAGE Publications.

An extensive overview on all relevant steps of the literature review process targeted at young scholars at the Master's and Doctorate level.

• Booth, A. Sutton, A. Papaioannou, D. 2016. Systematic approaches to a successful literature review. Second Edition. SAGE Publications.

A step-by-step illustration of the literature review process and what makes it 'systematic'.

References

(1) Rowley, J. Slack, F. 2004. Conducting a Literature Review. Management Research News 27(6). 31-39.

(2) Hart, C. 2018. Doing a Literature Review. SAGE Publications.

(3) Booth, A. Sutton, A. Papaioannou, D. 2016. Systematic approaches to a successful literature review. Second Edition. SAGE Publications.

(4) Randolph, J. 2009. A Guide to Writing the Dissertation Literature Review. Practical Assessment, Research, and Evaluation 14(14).

(5) Wikipedia. Systematic Review. Available at [1](https://en.wikipedia.org/wiki/Systematic_review#Research_fields) (last accessed 09.07.2020)

(6) Chalmers, I. Hedges, L.V. Cooper, H. 2002. A Brief History of Research Synthesis. Evaluation & The Health Professions 25(1). 12-37.

(7) Newig, J. Fritsch, O. 2009. THE CASE SURVEY METHOD AND APPLICATIONS IN POLITICAL SCIENCE. APSA 2009 Conference Paper.

Further Information

---

The author of this entry is Christopher Franz.