Case studies and Natural experiments

Note: This entry is a brief introduction to natural experiments. For more details on other types of experiments, please refer to the entries on Experiments, Experiments and Hypothesis Testing as well as Field experiments.

Natural experiments

Although this seems to be a little contradictory here, the impossibility of replication is a problem in the case of the Easter Island.

Out of a diverse rooting in discussions about complexity, system thinking and the need to understand specific contexts more deeply, the classic experimental setting did at some point become more and more challenged. What emerged out of the development of field experiments was an almost exact opposite trend considering the reduction of complexity. What do we learn from singular cases? How do we deal with cases that are of pronounced importance, yet cannot be replicated? And what can be inferred from the design of such case studies? A famous example from ethnographic studies is the Easter Island. Why did the people there channel much of their resources into building gigantic statues, thereby bringing their society to the brink of collapse? Or is it all different and is this an early testimony of the horrible ramifications of colonialism? While this is a surely intriguing question, there are no replicates of the Easter Islands. This is at a first glance a very specific and singular problem, yet it is often considered to be an important example on how unsustainable behaviour led to a collapse of a while civilisation. Such settings are referred to as Natural Experiments. From a certain perspective, our whole planet is a Natural Experiment, and it is also from a statistical perspective a problem that we do not have any replicates, besides other ramifications and unclarity that derives such single case studies, which are however often increasingly relevant on a smaller scale as well. With a rise in qualitative methods both in diversity and abundance, and an urge for understanding even complex systems and cases, there is clearly a demand for the integration of knowledge from Natural Experiments. From a statistical point of view, such cases are difficult and challenging due to a lack of being reproducible, yet the knowledge can still be relevant, plausible and valid. To this end, I proclaim the concept of the niche in order to illustrate and conceptualise how single cases can still contribute to the production and canon of knowledge.

For example the financial crisis from 2007, where many patterns where comparable to previous crisis, but other factors were different. Hence this crisis is comparable to many previous factors and patterns regarding some layers of information, but also novel and not transferable regarding other dynamics. We did however understand based on the single case of this financial crisis that certain constructs in our financial systems are corrupt if not broken. The contribution to develop the financial world further is hence undeniable, even so far that many people agree that the changes that were being made are certainly not enough.

Another prominent example of a single case or phenomena is the Covid pandemic that emerges further while I am writing these lines. While much was learned from previous pandemics, this pandemic is different, evolves different, and creates different ramifications. The impact of our societies and the opportunity to learn from this pandemic is however undeniable. While classical experiments evolve knowledge like pawns in a chess game, moving forward step by step, a crisis such as the Covid pandemic is more like the horse in a chess game, jumping over larger gaps, being less predictable, and certainly harder to master. The evolution of knowledge in an interconnected world often demands a rather singular approach as a starting point. This is especially important in normative sciences, where for instance in conservation biology many researchers approach solutions through singular case studies. Hence the solution orientated agenda of sustainability science emerged to take this into account, and further.

To this end, real world experiments are the latest development in the diversification of the arena of experiments. These types of experiments are currently widely explored in the literature, yet I do not recognise a coherent understanding of what real-world experiments are to date in the available literature. These experiments can however be seen as a continuation of the trend of natural experiments, where a solution orientated agenda tries to generate one or several interventions, the effects of which are tested often within singular cases, but the evaluation criteria are clear before the study was conducted. Most studies to date have defined this with vigour; nevertheless, the development of real-world experiments is only starting to emerge. For more info, please refer to the entry on Living Labs & Real World Laboratories.

A wonderful example for a bunch of real world experiments is the project Lüneburg 2030+. This map provides an overview of the different experiments.

Ramifications for statistics

1. Documentation is key.

Single case studies demand a clear documentation, and by clear I do not mean an exhausting documentation. Where many disciplines relying on single case studies often fail is the right balance between the details and the general information, or in other words, getting the Occam’s razor into the documentation. In sustainability science we did not learn to date which information is vital, and which one trivial. This is vastly different in medicine, where much of the canon of knowledge has been consolidated into a system that allows for a clear presentation of singular case studies. Take for instance the New England Journal of Medicine, one of the most prestigious scientific journals on this planet. Regularily, this journal publishes short case studies that provide either fundamentally new problems or even solutions that are recognised as important new knowledge in the respective field. Medicine is able to and actually demands such a presentation of knowledge, since much of the previous knowledge is compiled into a system that allows for a clear integration of new knowledge. In sustainability science such an established system is still widely lacking. People talk of complexity and the importance of recognising the individual case, yet I would argue that the world of medicine is equally complex. When people think that this is wrong, I think they are wrong. I consider the complexity of a singular individual and the world as such as being comparable, or at least equally complex. Sustainability science shall thus realize that making cases comparable is the key to coherent knowledge, and ultimately a more systematic approach to new knowledge. Statistics will certainly not be the only contribution to this end, but it is quite certain that it will contribute to a holistic understanding.

Chess games have certain openings. Most opening are known, and while some are famous, others are more obscure. Every chess player knows however the main 4-5 opening moves, since they determine the continuation of the game. Around the 8-12 moves however, something remarkable happens. We leave the range of known chess games, and enter a game that has probably never been played on this planet. It is a completely new game, and there are billions and billions of possibilities. This all then boils quite often down to fewer and fewer pieces, and if then two players are almost equally good, we might enter the realms of endgames. Endgames can be calculated, and here people that are good in thinking some moves ahead, have a higher chance of winning. However, in between the opening game and the endgame happens something that never happened before, a new game, which many consider to be complex. Here, a friend of mine taught me that this is indeed not complex, but instead builds on experience. He introduced me to the concept of board awareness. Ten thousand of games led in his head to a level of experience where when he would look at a middle game, his brain would almost instinctively interpret the board into hot zones and cold zones. Building on previous experience he would recognise the constellation and could derive actions and possible moves, and especially recognise which areas are prosperous for development through strategic moves, and which ones are endangered and ought to be defended. Experience builds reduction in complexity.

2. Look for mechanics you know from statistics

Statistics offer a certain perspective on the world. It makes a difference whether I count something or construct it into another data format. Many patterns follow a linear trend, yet some few may not. Variables can be grouped. When looking at a single case we can look beyond the complexities by looking at the case through the lens of knowledge we have from statistics. While many people argue that this creates a flawed picture, it might as well offer a more orderly view on the case. When working on a single case, we can look at cause and effect, we may look for predictive variables, and we may consider interactions between these. Many singular cases are rich and diverse, but also structured. We need to peel our way through the outer layer and understand the inner working of what we are looking at. Statistics is not only a method, but also a specific perspective. If we get versatile in statistics, it might help us to see something that we otherwise cannot see. As one of my first professors at University put it: “If you know more, you can understand more”. Consider statistics as your perspective on a single case. Practice this perspective, learn to see things through this lens.

3. Search for causality, yet be suspicious

What if single cases are not complex but simple? Many single case studies are very simple. And even the more complicated, some may have an inner working or a narrative. When looking at single case studies, we ought to remember that there can be a causal connection in the world, so there might be some in our case studies as well. Finding similarities between two things that have an effect on something else, may be worthwhile if you consider attracting a diverse set of costumers. Therefore I suggest that the supposably complexity of single cases should not repel us but instead energise and motivate us to find reason and causality within the case. Of course we may not know if there is such a thing as true causality at work, but based on previous cases and careful observation we may still infer new knowledge. Deciding that a case is complex does not help if we cannot say what is complex about it. To this end, complexity became a buzzword that is often used in a sense of something being not understandable. To me, it sometimes feels as if it is part of the zeitgeist to call something complex. Natural experiments should show us the value of single case studies, the value of understanding, and the value of creating new knowledge. I think it is our choice whether we want to seize this opportunity, or not.

4. Build on the available literature.

Although the "Wander über dem Nebelmeer" does not literally stand on the shoulders of giants, the painting of Caspar David Friedrich is a good visualisation of the metaphor.

No case is an island, even not cases on islands. While some cases can be novel in their characteristics - take wicked problems for instance -; every case may contain components that are known from previous cases. While wicked problems are new in the combination of building blocks, the building blocks themselves - i.e. parts of the problem - are maybe not new. The more you stand on the shoulders of giants, the further you can see. We need to realize that science is build on failure and iteration. Only by knowing about previous failures but also successes, and only by committing to the canon of knowledge, can we contribute and create new knowledge.

Natural experiments and statistics - a rough guide

Several layers of information are to be considered when linking statistics to natural experiments. Contextualising natural experiments can be structured into at least 4 steps.

1) How is the case connected and embedded in the global context? What are connections to other systems, where are dependencies, and where are gaps and missing links? A lot of information is available on a global scale, and it is worthwhile embedding a case into the global information available. This may allow us to specify the setting of the case, and to allow inference about how the results can be superimposed onto other systems. We hence acknowledge that this contextualisation is place based, and places have boundaries and borders. While it is way beyond this text to discuss this issue, the question how a place is bound can be a source of great confusion. A good rule of thumb from a statistical standpoint can be the idea that a place does only exists if its within variance, regarding central variables, is lower than the variance between the place and its neighbouring places.

2) what are imbalances and dispersions within the case, both spatially and temporally? In other words, what creates injustices in a place, both intragenerational but also intergenerational. Understanding the roots of injustices can be both as practical as money flows and resource dispersion, yet also as hard to tame as perceived injustices. Hence understanding the cultural and social context of the case's dynamics is often essential. Here statistics can offer ways for a clear documentation through observation, and system analysis and the like are good examples how the complexity of a case can be understood at least partly.

3) How are the livelihoods within the case? While many would connect this to the last point, it may be worthwhile to consider this on its own. Livelihoods are what defines groups of people and their possibility to thrive. While this is equally bound to culture and justice, it repeatedly proves that it is much more than the mere sum of the two. Livelihoods are often characterised by a deeply qualitative manifestation, which is where statistics need be to aware of its limitations. While statistics can compile descriptive information on a community and tackle some of the interactions within a community, it has to fail considering other points. This limitation is important to realise.

4) what is the intervention of the natural experiment within the case? Here, a whole world of interventions could be described. Instead, let us just settle on what was the active part of the researcher: systematically compare the state before the intervention with the state after the intervention.

Experiments in sustainability science

The world 3 model was part of the "Limits to Growth" report of the Club of Rome. The picture creates a sense of the multitude and complexity of variables that influence a single case study in the real world.

Experiments in sustainability science are characterised by an intervention and the production of empirical evidence. These two key criteria are essential for experiments in sustainability science, which can be in addition also differentiated into a problem focused and a solution orientated focus. Nevertheless, problem orientated studies with full control over all variables (except for the one(s) being investigated) are still clearly timely in sustainability science, as many experiments conducted in ecology prove. Less control over variables is equally important, and again ecology, agricultural science but also experiments in psychology come to mind. Problem orientated focus with no control over variables is at the heart of a joined problem framing of transdisciplinary research in sustainability science, and such an approach also characterised the first Club of Rome report. Solution orientated focus with total or some control over variables is insofar a remarkable step since it marks a shift from the still dominating paradigm of research where facts are only approximated. Since solutions may be compromises but can be achieved, a solution orientated research indicates a shift from more classical lines of thinking. The solution orientated approach with no control over variables marks the frontier of research, since it is clearly long term thinking. Prominent efforts are the IPBES, the sustainable development goal measures, and the current efforts of the rising number of transdisciplinary projects. While these are starting points, I consider these efforts to be a clear sign of hope, not only for out planet, but likewise for the development of science out of its dogmatic slumber.

Meta-analysis: Integrating cases towards holistic understanding

Out of the increasing number of scientific studies with a comparable design, an international school of thinking emerged: Meta-analysis! Within meta-analytical frameworks, great care is taken to combine several studies to investigate their overall outcome. Rooted deeply in medicine and psychology, meta-analysis used the statistical power of studies that show a similar design and setting. Since some studies are larger while others build on smaller samples, meta-analysis are able to take some of these differences into account.

It is however important to understand the thinking behind meta-analysis, as it helps to create supra-knowledge about certain questions that were enough in the focus so that a number of integrable studies is available. Graphical overviews show summaries of effects, and so called random factors allow for the correction of different sample sizes and other factors. Beyond the disciplines where meta analysis were initially established, this line of thinking found its way into ecology, conservation, agriculture and many other arenas of science that investigate building on deductive designs. To this end, challenges often arise out of field experiments where less variables are being controlled for or are being understood, and hence comparisons may not be valid or at least less plausible.

With a rising number of case studies, there is currently a trend to integrate more inductively designed studies as well into a meta-analytical scheme of analysis. Due to a more diverse study design and publication culture this is often a difficult and work intense endeavour. Still, it is vital to evolve science into an endeavour where more singular cases can be integrated to create meta-analytical knowledge. This is a task for the current generation, since the mode of science needs to be evolved. The knowledge and understanding of singular cases may be to precious to be lost and now integrated to build a deeper understanding. For more info, please refer to the Meta-Analysis entry.

External links

System Thinking: An introduction

Ethnographic Studies on the Easter Islands: An important example of a natural experiment

Natural Experiments: An introduction

Conducting and Evaluating Natural Experiments: A more detailed article

The financial crisis 2009: A very long & detailed paper

Real World Experiments in Ecology: A very interesting paper (VPN needed)

Sustainability Sciences: A short article about its origins and research agenda

Meta-Analysis: What is the difference to a systematic review?

---

The author of this entry is Henrik von Wehrden.