Systematic Literature Review
Quantitative - Qualitative
Deductive - Inductive
Individual - System - Global
Past - Present - Future
In short: In a Systematic Literature Review, existing publications are systematically analysed to derive an overview of a specific slice of the literature, typically summarizing the state of the art regarding a specific topic.
Contents
Background
With the rise of empirical knowledge in the Enlightenment arose the possibility to synthesize knowledge from different studies into an overview work. A Treatise on the Scurvy - A Critical and Chronological View of What has been Published on the Subject by James Lind is seen as the first systematic review (3, 5, 6), highlighting the importance of knowledge integration. Another important origin of research synthesis can be traced to the work of 17th Century astronomers who combined data sets from different studies to ammend their own observations (3). Systematic literature reviews gained a vital tool through the work of Karl Pearson, whose work on statistics allowed to compile the results from several datasets into an overview. His 1904 publication - in which he combined 11 studies on typhoid vaccines and highlighted irregularities in the results - can be considered the first Meta-Analysis (3, 5). Meta-Analyses were subsequently applied more commonly during the 20th Century, for example in agriculture (5, 6).
After the Second World War, US social scientists began to recognize the need to review the rising amount of research data while considering how to reduce bias and enhance reproducibility of systematic reviews (6). This also led to the increasing recognition of qualitative elements. In the 1970s, statistician Gene Glass and colleagues proclaimed Meta-Analyses as a valid procedure for synthesising studies which helped to consolidate the approach (3). However, Systematic Literature Reviews were long viewed as second-class studies within Academia, since they did not yield primary data. This changed during the last decades, partly due to increasing interest in scientific evidence on diverse topics on the part of public policy makers, practitioners and the general public (6).
More recently, due to the emergence of digitalisation and improvements in information storage and retrieval, it became significantly easier to identify, gather and analyze the available research on a specific topic (3). Today, Systematic Literature Reviews are most commonly used in Medicine, in the Social Sciences, Business and Economics, but have found their way into several other disciplines (5).
What the method does
Systematic Literature Reviews exist in a broad variety of types. While the Literature Review may be seen as the overarching term, sub-types include the Meta-Synthesis, the Systematic Review, the Case Survey or the strict Meta-Analysis (7). Their differentiation may be done in terms of the data they summarize (quantitative and/or qualitative) as well as the way this data is analyzed (qualitatively or quantitatively) (7). Reviews may also be differentiated according to their focus, goal, perspective, coverage, organization and audience. In this entry, we will focus on the Systematic Literature Review. More information on different sub-types of the method can be found in (3). Further, the approach of Meta-Analyses as the statistical tool of summarizing a variety of studies on a specific topic into one is an important method in this regard. For more on Meta-Analyses, please read the respective Wiki entry.
Definition
A Systematic Literature Review is, in short, a reproducible process in which scientific publications that "contain information, ideas, data and evidence" (Hart 2018, p.13) on a specific topic are gathered; studies that fulfill a previously defined level of quality are selected; and their results and insights are summarized and evaluated. For the researcher, the results from this process provide insights into the current state of research and highlight relevant new directions for (their) further research (1, 2, 3). The term 'systematic' refers to the fact that the process is structured to minimize bias (6) and maximimize reproducibility. Being 'systematic' means being reproducible and goes along with an a priori specified, dedicated research design and an explicit documentation of the steps taken during the research (3, see Normativity).
A Systematic Literature Review can be applied as the primary method of a scientific study, but is often also used as a first step in a larger research projector endeavor. In both cases, a review can help:
- recognize what has already been done already regarding a specific research field or topics
- find and resolve conflicts in (seemingly) contradictory studies, and
- identify evolving or even unexplored research topics, questions or new hypotheses for further research.
When used as a preparation to one's own study, a Systematic Literature Review additionally helps the researcher
- identify relevant literature and researchers to consult,
- design appropriate methodological approaches,
- understand important concepts, theories and topics and summarise this knowledge to the reader
- contextualize this research and show why it would answer an open question, often on integration level. (1, 3, 4)
Step by Step
A Systematic Literature Review follows a set of steps that is similar to any scientific research process (1, 3, 4):
1) Planning: The research is designed by formulating the question and scope of the review (here, the different forms of the review - see above - are of relevance). This planning can be both inductive as well as deductive, depending on the focus of the review. For example, the researcher may be interested in how the literature defines "Sustainable Agriculture" (see example below), and thus choose to search for literature that apply and preferably define this concept.
2) Data gathering: Data (= literature) is searched and acquired. There may exist a wide range of documents of relevance to the research endeavour, which is why the researcher should attempt to become familiar with all related topical fields. The core of reviews are often articles in scientific journals due to their comparable structure and assured quality. However, also books and practitioner articles may be of help (1). The data collection can be done using library catalogues or online search engines and databases, such as Google Scholar, SCOPUS, or Researchgate (1). Relevant literature can either be found by applying single search terms, but combining more specific search terms through AND, NOT or OR. This helps specify the results, whereas searching for single words can lead to very broad results. For example, "sustainable" AND "agriculture" may be of better help than just either of the terms. In addition, useful documents should be identified asking colleagues and by scanning the bibliographies of already selected papers until saturation is reached, i.e. no more relevant documents come light (4). When all data is gathered from the relevant different sources, duplicates are removed.
3) Data selection:
The researcher now has a wide selection of literature available that fulfills the search terms. Now, they apply pre-defined quality and selection (= exclusion and inclusion) criteria to decrease the number of documents. One of four different approaches may be applied (3):
- exhaustive coverage (citing all relevant literature)
- exhaustive coverage with selective citation
- representative coverage (discussion of works which typify particular groupings in the literature) or
- coverage of pivotal works.
While the existing amount of research makes a truly 'exhaustive' collection rather 'exhausting' for many topics, any selection of articles comes with the danger of a selection bias. The relevance of the documents may be assessed based on reading the whole texts, just the abstracts, just the titles or some combination of these elements, which should be documented (4). The process description may include:
- the literature search terms, the date of search, the number of search results and exclusion criteria,
- a list of the included studies, as well as a description of the synthesis process,
- a sample description of the data extraction process as well as
- a description of the applied quality standards (3).
Finally, the researcher attempts to obtain all full-texts for the selected publications. These may be available on the aforementioned platforms, sometimes via paid subscriptions, or by contacting the researchers directly, which are often happy to provide their publications. Some more publications may be excluded at this point, when it becomes obvious that a previously eligible-thought publications does not really fit the inclusion criteria after all.
4) Data synthesis & analysis: After the documents were selected, the researcher reads them and extracts information that helps answer the research questions. In the process of synthesizing, a Mindmap or Concept Maps may be useful tools which can help organize and understand the concepts and theories used in the documents. The documents may further be arranged in such a map to support the structuring process of the review (1). The focus and thus the methodological process of the data extraction depend on the goal of the review. For example, a researcher might want to focus on theories or concepts, definitions, methodological approaches or different scales and actors that played a role in the specific paper that is being reviewed. Typically, a coding book or procedure should be developed (and documented) in which the extraction process of information is defined, before all information of interest is sorted into the respective coding categories (4). While some variables can be extracted based on foundational work form previous studies, many parameters are often also extracted in an inductive way.
There are many diverse approaches to summarise and analysed the respective data, and to present it to the reader. Depending on the intended outcome of the review as well as the type of data gathered, these can be quantitative or qualitative:
Qualitative analyses are appropriate for the review of purely qualitative or mixed studies and can be often offer perspectives that go beyond quantitative analyses, for instance concerning deeply normative aspects. The qualitative review revolves around identifying essential themes of the documents and their relationships. Also, contrary findings and contradicting interpretations are of interest. "The goal here, unlike meta-analysis, is to increase the understanding of the phenomena being investigated, not to integrate outcomes and identify factors that covary with outcomes." (4, p.10).
Beside systematic literature reviews there is also another form of reviews, which is often referred to as 'narrative reviews', 'expert reviews', or simply as 'literature reviews'. These are often conducted in a non-systematic sense, but instead consist of the purely deliberate selection of the literature, typically by an author deeply experienced in the literature, and consist a balanced overview of the available literature. While such reviews are often seen to be more subjective, they can contain a lot of experience, and were more abundant in the past. These days such narrative reviews are often frowned upon, which is a pity, because there is a difference between knowledge and experience, and such narrative reviews can often offer a lifetime of experience.
Often, a systematic literature review also additionally applies a Meta-Analysis in the analysis step. A Meta-Analysis is the statistical process of assessing quantitative, numerical data from separate studies that investigated a specific phenomenon. You might for instance calculate means or variations, and evaluate how specific factors influence specific processes. Combining both types of review has the benefit of first focusing on conceptual and qualitative insights from the body of literature, and then going deeper into quantitative measures. For more information on this, please refer to the Meta-Analysis entry.
5. Writing & presentation of the review: Finally, the results of the Systematic Literature Review are compiled into a structured paper. A sample structure for the review as a preface to an original study may look like this (Rowley & Slack 2004, p.38):
- Basic Definitions of key terms
- Why is the subject of interest?
- What research has already been undertaken on the topic, and is there any research on aspects of the topic that this research might investigate?
- A clear summary of the results and new research opportunities that emerge from the literature review. Quotations may be used to underline specific findings from the review.
- In addition, the literature gathered during the review should be listed. Many reviews consists a combination of information on the specific topic, the conceptual foundation, methodological approaches, and relevant scales that are associated to the available literature.
In summary, Systematic Literature Reviews are methods of data gathering - building on primary data from other empirical papers - and analysis. They are inductive because they conclude based on existing literature, but also deductive since they typically start with theoretical assumptions and a pre-defined thematic scope. They can be qualitative and quantitative, cover very local to global phenomena, and investigate past and present states of the literature, often offering a state of the art of the literature, and thereby suggestions and agenda for future research.
Strengths & Challenges
- A literature review is a helpful starting point for any research: it provides a structured overview of the current knowledge and open questions in the respective field and may lead to new research questions that had not been obvious to the researcher before. The strength here is the systematic nature of the literature review: by not only reading randomly through literature, but instead following a systematic and reproducible approach, the researcher generates a more objective overview of the literature. Of course, the selected literature may still be biased and heavily depends on the research and selection criteria, but if this process is properly reasoned and documented, the bias may be reduced compared to a non-systematic review.
- When used in preparation to an original study, "[r]eviewing the literature in a systematic way helps the author to be clear, to build confidence in their work and demonstrate the rigour of their methods." (3, p.9). This is why Systematic Reviews are often done in a first step of a larger study, or as a first paper of a PhD student.
- The systematic approach enables the researcher to state conclusions about the strength of available evidence to a specific assumption. This does not only support their own subsequent work, but also provide insight into the state of science in a given field to policy makers and other public actors. This way, the Systematic Literature Review may for example also shine light on the effectiveness of programs and policies (3).
Potential mistakes and thus challenges in the review process include (4):
- not clearly relating the findings of the literature review to the researcher's own study
- not taking sufficient time to identify the best (primary!) sources
- not critically reflecting upon the studies' research designs and analysis
- not reporting the search procedures
- reporting isolated statistical results instead of meta-analytic or chi-square methods
- not considering contrary findings or alternative interpretations when synthesizing quantitative literature
Normativity
Connectedness
- The Systematic Literature Review is strongly connected to further methods. As explained before, this form of secondary research is often included as a groundwork in original primary studies to help justify the research topic, design and methodology (2). "Indeed, the concluding paragraphs of the literature review should lead seamlessly to research propositions and methodologies." (1, p.32)
- The analysis of the gathered studies, as mentioned before, can take place in a quantitative way based on a variety of quantitative approaches. Most notably, the Meta-Analysis is of relevance here.
- Systematic Literature Reviews represent an interesting and important form of methodological approach: they allow research on research. If scientific results only cumulated and no one looked at the bigger picture, science would not work. Methods like the literature review are thus crucial for academic processes.
- Again, we would like to highlight that there is a difference between Systematic Literature Reviews and Literature Reviews in general. Often, you will do a literature review for a university report, or your thesis. For this, you gain an understanding of the available theoretical and empirical literature on a topic by browsing through literature, possibly through snowball sampling, until you feel like you have read enough. This may be necessary, enables you, to structure your own data collection, or to write an essay about the topic. This is absolutely valid, but is different from a systematic review with the presented systematic approach and documentation, which may enable much deeper insights into available research in the field.
Quality criteria
"Quality [of the literature review] means appropriate breadth and depth, rigour and consistency, clarity and brevity, and effective analysis and synthesis; in other words, the use of the ideas in the literature to justify the particular approach to the topic, the selection of methods, and demonstration that this research contributes something new." (Hart 2018, p.1f). The quality criteria of objectivity, validity and reliability apply to the Systematic Literature Review as follows:
- Objectivity & Validity: Literature reviews are to some level subjective because the synthesis of the screened literature (as well as the identification and selection of the literature in the first place) is, although to some extent reproducible, still a matter of the reviewer's normative decisions (3). The approach to gathering as well as excluding data may change as the researcher gains more experience in the process. Also, "(...) [a]ll reviews, irrespective of the topic, are written from a particular perspective or standpoint of the reviewer. This perspective often originates from the school of thought, vocation or ideological standpoint in which the reviewer is located. As a consequence, the particularity of the reviewer implies a particular reader. Reviewers usually write with a particular kind of reader in mind: a reader that they might want to influence." (Hart 2018, p.25). It can be said that the more systematically the method is applied, the more potential bias is reduced (3). Special attention may be paid to the 'publication bias', highlighting the fact that often, those studies are preferred for publication that offer new or 'interesting' results, which may influence the review results (3). Also, researchers should not make the mistake of full-text-on-net-bias, but make sure to include all relevant literature, even though it may take some more time to access them.
- Reliability or auditability is safeguarded by a detailed description of the system of the methodological process. This way, other reviewers following the same procedures under the same conditions should - in theory - find an identical set of articles and come to the same results and conclusions. Being systematic in these steps and reporting on the process improves clarity regarding what has been done and what has not been done, i.e. why certain documents have been included and why others have not (3).
Outlook
In the face of the ever-increasing amount of data gathered on diverse topics, as well as due to the changing role of science in society, there may be a increased interest and opportunity of the public in scientific results that may be summarized through Systematic Literature Reviews. With more and more data becoming available, scientists need to integrate existing knowledge to allow for a measured and responsible planning of future research. In addition, such reviews offer a necessary critical perspective, which may not only generate a future research agenda, but in addition can highlight flaws and biases in past research. As it is already the case in medicine, systematic literature reviews should be continuously or at least regularly be updated to offer the latest finding in an integrated way. This would ideally generate a structured but critical research agenda, and a better integration of knowledge that can be increasingly communicated to the public.
An exemplary study
In their 2015 publication, Velten et al. investigate the definition and conceptualization of 'sustainable agriculture' in the available literature, with a focus on how social processes shaped the concept. They referred to both academic and practitioner-oriented literature.
Academic publications were searched on SCOPUS by use of the following search criteria:
- search terms were "sustainable agriculture" OR "agricultural sustainability" in the title, abstract or keywords
- English, German, French, Spanish and Portuguese publications
- up to the year 2012
- in the subject areas of social sciences and humanities.
Non-academic literature - 'grey literature' - that is directed more towards practictioners and decision-makers was searched on websites, reports and brochures, using the same search terms and language restrictions on Google. The search continued until saturation was reached and "no new usable publications were found" (p.7836). Further, the pages of international organizations were searched for relevant documents.
The initial set of publications was then limited to those that "gave at least a minimal definition or explanation of what was meant by sustainable agriculture" (p.7836), resulting in 129 academic and 26 non-academic documents. These documents were analyzed using an inductive qualitative content analysis, during which key elements ('categories') and overarching topics ('themes') were extracted in three general groups that play a role in the definition of the concept of 'sustainable agriculture':
... Goals
... Strategies
... and Fields of Action
Further, the researchers developed an analysis scheme to analyze the extracted categories and themes quantitatively:
In a third step, the authors conducted a Cluster Analysis on the academic literature based on the quantitative data in order to identify groups of literature that bring up specific aspects of the concept.
Based on their qualitative results, the researchers highlight that 'sustainable agriculture' can be separated into three distinct groups (Goals, Strategies, Fields of Action) with specific themes and categories, each. Based on their quantitative analysis, they show how the most frequent topics revolve around anthropocentric rather than ecocentric values when it comes to defining what is 'sustainable agriculture', and that the realization of sustainable agriculture mostly focuses on technological solutions on the farm level. However, they also found a strong alternative discourse that focuses on ecocentric and alternative approaches. They could further identify categories that changed in importance over time, and how the distribution of themes differed between academic and non-academic literature and between scientific disciplines. Lastly, their subsequent cluster analysis led to five groups of conceptual understandings of 'sustainable agriculture'.
This study exemplifies how a Systematic Literature Review with both qualitative and quantitative elements provides a rich conceptual overview of a topic which can be used to further structure work in this field, and raises questions for academia as well as practitioners. In their conclusion, the authors "(...) recommend embracing the complexity of sustainable agriculture with its varied and seemingly contradictory aspects" since they found "(...) the different conceptions of sustainable agriculture to be not as contradicting and mutually exclusive as they have often been portrayed". (p.7857). This shows how a systematic review can help grasp the diversity of aspects on a given topic, and draw the respective conclusions.
Key publications
Hart, C. 2018. Doing a Literature Review. SAGE Publications.
- An extensive overview on all relevant steps of the literature review process targeted at young scholars at the Master's and Doctorate level.
Booth, A. Sutton, A. Papaioannou, D. 2016. Systematic approaches to a successful literature review. Second Edition. SAGE Publications.
- A step-by-step illustration of the literature review process and what makes it 'systematic'.
References
(1) Rowley, J. Slack, F. 2004. Conducting a Literature Review. Management Research News 27(6). 31-39.
(2) Hart, C. 2018. Doing a Literature Review. SAGE Publications.
(3) Booth, A. Sutton, A. Papaioannou, D. 2016. Systematic approaches to a successful literature review. Second Edition. SAGE Publications.
(4) Randolph, J. 2009. A Guide to Writing the Dissertation Literature Review. Practical Assessment, Research, and Evaluation 14(14).
(5) Wikipedia. Systematic Review. Available at [1](https://en.wikipedia.org/wiki/Systematic_review#Research_fields) (last accessed 09.07.2020)
(6) Chalmers, I. Hedges, L.V. Cooper, H. 2002. A Brief History of Research Synthesis. Evaluation & The Health Professions 25(1). 12-37.
(7) Newig, J. Fritsch, O. 2009. THE CASE SURVEY METHOD AND APPLICATIONS IN POLITICAL SCIENCE. APSA 2009 Conference Paper.
(8) Velten, S. Leventon, J. Jager, N. Newig, J. 2015. What is Sustainable Agriculture? A Systematic Review. Sustainability 7. 7833-7865.
Further Information
- A short YouTube Video on Systematic Literature Reviews by 'Research Shorts' that summarizes the method in 3 minutes.
The author of this entry is Christopher Franz.
