Delphi

Method categorization
Quantitative	Qualitative
Inductive	Deductive
Individual	System	Global
Past	Present	Future

Background

The Delphi method originates from work at RAND Corporation, a US think-tank that advises the US military, in the late 1940s and 1950s (2, 3, 5). RAND developed "Project Delphi" as a mean of obtaining "(...) the most reliable consensus of opinion of a group of experts." (1, p.1). At the time, the alternative - extensive gathering and analysis of quantitative data as a basis for forecasting and deliberating on future issues - was not technologically feasible (4, 5). Instead, experts were invited and asked for their opinions - and Delphi was born (see (1)).

SCOPUS hits for the Delphi method until 2019. Search terms: 'Delphi Panel', 'Delphi Method', 'Delphi Methodology', 'Delphi Study', 'Delphi Survey' in Title, Abstract, Keywords. Source: own.

In 1964, a RAND report from Gordon & Helmer brought the method to attention for a wider audience outside the military defense field (4, 5). Subsequently, Delphi became a prominent method in technological forecasting; it was also adapted in management; in fields such as drug policy, education, urban planning; and applied in order to understand economic and social phenomena (2, 4, 5). An important field today is the healthcare sector (7). While during the first decade of its use the Delphi method was mostly about forecasting future scenarios, a second form was developed later that focused on concept & framework development (3).

Delphi was first applied in these non-scientific fields before it reached academia (4). Here, it can be a beneficial method to identify topics, questions, terminologies, constructs or theoretical perspectives for research endeavours (3).

What the method does

The Delphi method is "(...) a systematic and interactive research technique for obtaining the judgment of a panel of independent experts on a specific topics" (6, p.99). It is used "(...) to obtain, exchange, and develop informed opinion on a particular topic" (2, p.309) and shall provide "(...) a constructive forum in which consensus may occur" (2, p.309).

The Delphi procedure

Questionnaire results for the original RAND study, asking for an estimate of bomb requirements. The estimated numbers per participant converge over the course of the Delphi procedure. Source: Dalkey & Helmer 1963, p.15

A Delphi process typically undergoes four phases (see (4), (6)):

1. A group of experts / stakeholders on a specific issue is identified and invited as participants for the Delphi. These participants represent different backgrounds: academics, government and non-government officials as wel, as practitioners. They should have a diverse set of perspectives and profound knowledge on the discussed issues. They may be grouped based on their organizations, skills, disciplines or qualifications (3). Their number typically ranges from 10 up to 30, depending on the complexity of the issue (2, 3, 5, 6).

The researchers then develop a questionnaire. It is informed by previous research as well as input from external experts (not the participants) who are asked to contribute knowledge and potential questions on the pertinent issue (2, 5). The amount of consultation depends on the expertise of the researchers on the respective issue (2).

2. The questionnaire is used to ask for the participants' opinions and positions related to the underlying issue. The questions often take a 'ranking-type' (3): they ask about the likelihood of potential future situations, the desirability of certain goals, the importance of specific issues and the feasibility of potential policy options. The answer options are to be ranked by the participants (from least to most desirable, least to most feasible etc.) (2).

The questioning is most commonly conducted in form of a questionnaire but has more recently also been realized as individual, group, phone or digital interview sessions (2, 5). Digital questioning allows for real-time assessments of the answers and thus a quicker process. However, a step-by-step procedure provides more time for the researchers to analyze the responses (4).

3. After the first round, the participants' answers are analyzed both in terms of tendency and variability. The questionnaire is adapted to the new insights: questions that already indicated consensus on a specific aspect of the issue are abandoned while disagreements are included. Consensus may be defined based on a certain percentage of participants agreeing to one option, the median of the responses or a degree of standard deviation, among other definitions (2, 5, 6, 7). New questions may be added to the questionnaire and existing questions may be rephrased based on the first set of answers (4).

Next, the experts are again asked for their opinions on the newly adapted set of questions. This time, the summarized but - this is important - anonymous group results from the first round are communicated to them. This feedback is crucial in the Delphi method. It incentivizes the participants to revise their previous responses based on their new knowledge on the group's positions and thus facilitates consensus. The participants may also provide reasons for their positions (5, 6). Again, the results are analyzed. The process continues in several rounds (typically 2-5) until a satisfactory degree of consensus among all participants is reached (2-6).

4. Finally, the results of the process are summarized and evaluated for all participants (4).

Strengths & Challenges

The literature indicates a variety of benefits that the Delphi method offers.

• Delphi originally emerged due to a lack of data that would have been necessary for traditional forecasting methods. To this day, such a lack of empirical data or theoretical foundations to approach a problem remains a reason to choose Delphi. Delphi may also be an appropriate choice if the collective subjective judgment by the experts is beneficial to the problem-solving process (2, 4, 5, 6).
• Delphi can be used as a form of group counseling when other forms, such as face-to-face interactions between multiple stakeholders, are not feasible or even detrimental to the process due to counterproductive group dynamics (4, 5).
• The value of the Delphi method is that it reveals clearly those ideas that are the reason for disagreements between stakeholders, and those that are consensual (5).
• Delphi can be "(...) a highly motivating experience for participants" (2, p.309) due to the feedback on the group's opinions provided in subsequent questioning stages.
• The Delphi method with its feedback characteristic has advantages over direct confrontation of the experts, which "(...) all too often induces the hasty formulation of preconceived notions, an inclination to close one's mind to novel ideas, a tendency to defend a stand once taken, or, alternatively and sometimes alternately, a predisposition to be swayed by persuasively stated opinions of others." (3, p.2, after Dalkey & Helmer)
• Additionally, Delphi provides several advantages over traditional surveys:
  • Studies have shown that averages of group responses are superior to averages of individual responses. (3)
  • Non-response and drop-out of participants is low in Delphi processes. (3)
  • The availability of the experts involved allows for the researchers to (a) get their precognitions on the issue verified by the participating experts and to (b) gain further qualitative data after the Delphi process. (3)

However, several potential pitfalls and challenges may arise during the Delphi process:

• Delphi must not be used as a surrogate for every other type of communication - it is not feasible for every issue (4, 5, 6).
• A Delphi format that was useful in one study must not work as well in another context. Instead, the process must be adapted to the research design and underlying problem (4).
• The proper selection of participating experts constitutes a major challenge for Delphi processes (3, 4, 5, 6). In addition, the researchers should be aware that any expert is likely to forecast based on his specific sub-system perspective and might neglect other factors (4).
• The monitor (= researcher) must not impose his / her own preconceptions upon the respondents when developing the questionnaire but be open for contributions from the participants. The questions should be concise and understandable and should not incentivise the participant to "get the job over with" (4, p.568; 5).
• Diverse forms of bias might occur that need to be anticipated by the researcher. These biases on the part of the participants include discount of the future, over-optimism / over-pessimism, misinterpretations with regard to the complexity and uncertainty involved in forecasting the future as well as other forms of bias that may be imposed through the feedback process (4, 6).
• The responses must be adequately summarized, analyzed and presented to the participants (see the variety of measures for 'consensus' in What the method does)). "Agreement about a recommendation, future event, or potential decision does not disclose whether the individuals agreeing did so for the same underlying reasons. Failure to pursue these reasons can lead to dangerously false results." (4, p.568).
• Disagreements between participants should be explored instead of being ignored so that the final consensus is not artificial (4).
• The participants should be recompensated for their demanding task (4)

Normativity

Connectedness / nestedness

• While Delphi is a common forecasting method, backcasting methods (such as Visioning) may also be applied in order to evaluate potential future scenarios without tapping into some of the issues associated with forecasting (see more in the Visioning entry)
• Delphi can be a starting point for subsequent research processes.
• Delphi can be combined with qualitative or quantitative methods beforehand (to gain deeper insights into the problem to be discussed) and afterwards (to gather further data).

Everything normative related to this method

• The Delphi method is highly normative because it revolves around the subjective opinions of stakeholders.
• The selection of the participating experts is a normative endeavour and must be done carefully so as to ensure a variety of perspectives.
• Delphi is an instrument of transdisciplinary research that may be used both to find potential policy options as well as to further academic proceedings. Normativity is deeply rooted in this connection between academia and the 'real-world' (see the article on transdisciplinary research)

Outlook

Open questions

• The diverse fields in which the Delphi method was applied has diversified and thus potentially confounded its methodological homogeneity, raising the need for a more comparable application and reporting of the method (6, 7)

Key Publications

• Linstone, H. Turoff, M. 1975. The Delphi Method: Techniques and Applications. Addison-Wesley, Boston.

An extensive description of the characteristics, history, pitfalls and philosophy behind the Delphi method.

• Dalkey, N. Helmer, O. 1963. An experimental application of the Delphi method to the use of experts. Management Science 9(3). 458-467.

The original document illustrating the first usage of the Delphi method at RAND.

• Gordon, T.J. Helmer, O. 1964. Report on a long-range forecasting study. RAND document P-2982.

The report that popularized Delphi outside of the military defense field.

References

• (1) Dalkey, N. Helmer, O. 1963. An experimental application of the Delphi method to the use of experts. Management Science 9(3). 458-467.

• (2) Rayens, M.K. Hahn, E.J. 2000. Building Consensus Using the Policy Delphi Method. Policy, Politics, & Nursing Practice 1(4). 308-315.

• (3) Okoli, C. Pawlowski, S.D. 2004. The Delphi Method as a Research Tool: An Example, Design Considerations and Applications. Information & Management 42(1). 15-29.

• (4) Linstone, H. Turoff, M. 1975. The Delphi Method: Techniques and Applications. Addison-Wesley, Boston.

• (5) Gordon, T.J. 2009. The Delphi Method. Futures Research Methodology V 3.0.

• (6) Hallowell, M.R. Gambatese, J.A. 2010. Qualitative Research: Application of the Delphi method to CEM Research. Journal of Construction Engineering and Management 136(1). 99-107.

• (7) Boulkedid et al. 2011. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. PLoS ONE 6(6). 1-9.

Further Information

---

The author of this entry is Christopher Franz.