Difference between revisions of "Limitations of Statistics"

Latest revision as of 12:02, 12 June 2023

In short: This entry revolves around what Statistics cannot do, and where its limits are - if you are interested in what Statistics CAN do, have a look at the Statistics overview page. You might also be interested in the topic of Bias in statistics.

Statistics and the where it may fail us

Statistics provide a specific viewpoint with which you can look at the world, and it can help you understand parts of what you're seeing. If more people understood statistics and the insights we can gain through it, these people would become more empowered to make their decisions based on their own analysis and insight. However, we do not only need to recognise the benefits that can be gained through statistics, but we also need to be aware of the limitations of statistics. After all, it is a view through numbers and models, and can therefore only offer parts of the complete picture. In addition, we have to accept that statistics are also riddled by diverse biases, which we also have to take into our focus and conscious recognition as much as possible. More about Bias in statistics can be found here.

Beyond problems directly associated to statistics as such, there are also problems associated to the relative position of statistics within science. Statistics were feeling the flame of positivism, and a critical perspective on statistics is beyond a doubt of pivotal importance for the recognition of the role that statistics can play to produce knowledge, but also to understand where statistics may fail us. In other words, we need to be clear about what statistics can do for us as scientists, and what it cannot do. The other major problem that statistics face within science is the normativity of statistics. There are diverse norms and schools of thinking that got established over time, and within this entry we will take a look at some examples on how differences, co-evolutions and interchanges influence the role of statistics within science. To this end, a third point may also be a historical perspective, which can be by no means exhaustive, but should enable us to once more to a look at the role of statistics in science.

The question of the limitations of statistics in the current societal debates - and the possibilities and opportunities which statistics may offer for the great transformation - are in the focus of the last part of this entry. Why are many societal debates so disconnected from available results derived through statistics? And how can we improve the literacyy regarding statistics within society? Lastly, we want to examine with a critical eye how the limitations of statistics may contribute to some of the problems we face in society today, and how we may ultimately overcome these.

Limitations of statistics

Limitations of statistics can be divided into several parts.

The dependent variable

First, and many would argue most importantly, are the questions of plausibility and validity. Are the questions we are testing plausible, and are the predictors we use to access these models valid? We often test constructs: for instance, GDP is commonly used as a proxy for many other things. While this had some surely some merits in the past, it also draws an increasing amount of criticism. Another example would be the Intelligence Quotient, which may tell you something about people, but more often than not is misleading and judgmental. Have a look at the entry To Rule And To Measure to learn more, and also remember the role that data distributions can play.

The independent variable

Equally, predictors may vary in their validity and reliability. This may be related to the way the predictors are measured, how constructed specific predictors are, or if they can actually serve as proxies to answer the questions we want to answer. Money can be a predictor for happiness, but it has clearly its limitations, and the amount of money needed to have a decent life may differ between countries. Also remember that different data formats are highly relevant in order to understand the nature of your data.

Random factors

This brings us to the next factor that can affect our model validity. There are many things we might want to know when creating a statistical model, but there may also be things that we do not want to know. Statistically speaking, these are random factors or random variables. Regarding these, we explicitly exclude the variance that is created by these parts of the dataset, because we want to minimise their effects. An example would be a medical study that wants to investigate the effect of a certain drug on the recovery rate of some diseased patients. In such studies, the information whether a patient is a smoker or not is often included as a random factor, because smoking negatively affects many diseases and the respective recovery rates. We know that smoking makes many things worse (from a medical standpoint) and this is why such variables are excluded. More on random factors can be found here.

The model

The next part of a statistical model that needs to be examined when discussing the limitations of statistics is the statistical test or model itself. Models differ in terms of their mathematical foundations. This can lead to some models being more complicated than others, relying on different measures of model performance, and are also sometimes differently utilised within different scientific communities or disciplines. Data visualisations showcase how different branches of sciences have different ways to approach how to show data in figures, which can be diverse and often follow specific norms within different scientific disciplines.

This is difficult to explain in a nutshell, but let me try: basically, different historical developments in the diverse disciplines, as well as the specific relation of a discipline towards the respective theory and topics, enable differences in how statistics are being utilised by different disciplines. Yet, there are also examples of exchanges and even coevolutionary patterns. For instance, the ANOVA started within agricultural science, but equally unleashed its potential in psychology, medicine and ecology, as these all rely strongly on the testing of hypotheses. Econometrics and engineering build more strongly on correlative testing, which is again related to the specifics of these disciplines. Most simple tests are hardly being used these days, a phenomena that unites almost all disciplines, yet sometimes even very simple calculations can offer robust answers. However, the difference between scientific disciplines regarding their statistical use does not always make sense, and is not necessarily rooted in reasonable decisions.

Limitations of statistics within science

Equally grave (and not necessarily always meaningful) is the great divide between quantitative and qualitative branches within science. While we should acknowledge that there are clear limitations of knowledge driven from statistics, it is equally difficult to reject knowledge derived from quantitative methods altogether. It is well deserved that with the rise of positivism - which widely cemented its reign through quantitative methods and particularly statistics - a qualitative counter-culture established itself. However, this does by far not do justice to the highly creative and so desperately needed deeper unlocking of qualitative dimensions in research. Interviews, Grounded Theory, Ethnographic Observations and Content Analysis are examples of important methods being established that go way beyond previous methods, and some of these approaches have a long history of highly relevant applications in science. However, statistics is nevertheless embedded into a judgemental perspective, creating a dichotomy that has not always been of benefit for science. Some would argue that the positivist wars - and the subsequent science war and culture war - are on a long chain of interwoven conflicts that may have been solved by critical realism.

We have to acknowledge that the knowledge we create is subjective, which one of the main claims of critical realisms. Still, our way of using this knowledge to make sense of the world and derive heuristics and concepts that tell us how we ought to act may be objective. For instance, I might acknowledge that our knowledge about the validity of a vaccine to prevent me from getting a certain disease is a snapshot in time, and governed by conservative and robust norms in research. Hence I might decide to trust this knowledge in order to not only protect myself, but also to contribute to the protection of society. The decision to protect myself can be seen as hedonism, while the decision to protect my surrounding and ultimately society as a whole can be declared my contribution to the social contract. This showcases that my intentionality can be rooted in different principles, and the way we conduct experiments showcases how these areas of science are still under development. My actions may be informed by statistics, but statistics can only serve as a basis for my ontology, not be ontology in itself.

Limitations of statistics within society

This brings us to the complicated relationship between statistics and society. Through its contribution to agricultural science, medicine, economics and many other fields, statistics have served as one of the main methoetodological approaches to contribute to a growth-focused economy. It can be clearly argued that this led to manifold benefits, but also to problems, including destruction, degradation and inequalities. All the while, science became increasingly glorified in the middle of the 20th century. However, due to the many negative effects that statistics indirectly contributed to, a critical perspective on statistics increased within the societal debate. What in the past was perceived as factual knowledge was increasingly questioned, and even altogether rejected.

It is clear that the knowledge derived from statistics has value, but also limitations for societal discourse. Within an increasing rise of dichotomies as part of the culture wars, science was often called out, or simply taken as an emotional counterpoint. The debate on climate change showcases that both sides argue strongly from an emotional perspective (if we think in terms of the extreme points within the debate). At the same time, both sides also make selective arguments, having different focus points in terms of the goals they are trying to achieve. Climate change activists try to argue based on scientific evidence to protect the planet. Climate change deniers often take a limited or blurry understanding of bits and pieces of information to create a counterpoint that focuses, for instance, on the rights of some few people to continue their current strategies. Both sides seemingly argue about so-called 'facts', which showcases the deep entrenchment and differences that currently riddle societal debates. Statistics, to this end, is often devalued by one side or the other, and becomes a collateral due to criticism that emerges from false understandings or implausible claims.

This highlights that statistics in itself may not be normative if all people share the same understanding about it. Sadly, this is not the case, and hence the limitations of statistics within societal debates are not much more than a mirror of the lack of education and ignorance that many people have to endure. Statistics is thus a privilege that some people can benefit from, but to use this educational effort as a devaluation of people who do not share this privilege is a difficult point. It would be naive to say that if all people shared the same knowledge, we'd all come to the same conclusions. But would it not be wonderful if this was the case?

I believe that power to the people is still one of the best idea we ever had. Yet, with it comes the responsibility to provide an education that enables all people to use their power to take decisions. Ethics suggests that these decisions ought to be rooted in principles. These principles can be informed by information that can originate from - among many other sources - statistics. During the pandemic, for instance, the interest in numbers increased among the broader public. At the same time, it also became clear that different numbers and changes in these numbers represented different aspects of the pandemic at different times. Sometimes it was the amount of spread of the disease (r-value), sometimes the number of occupied hospital beds, at other times the mortality, and sometimes the increase in cases. This created confusion, and even the experts were not agreeing which number was relevant at different times, or which combination of values. This showcases that statistics are still emerging, especially when it comes to taming wicked problems, and also highlights the gap between science and society. It does not make any sense to blame people that were never educated in statistics for not understanding it. It becomes clear that science still has a long way to go to explain knowledge from statistics in a sense that it becomes a contribution to the great transformation, yet statistics should seize this opportunity and learn to contribute in a critical way.

External Links

Articles

A nice take on statistics and its limitations

Videos

Podcasts

The author of this entry is Henrik von Wehrden.

@@ Line 13: / Line 13: @@
 ==== The dependent variable ====
-First, and many would argue most importantly, are the questions of plausibility and validity. Are the questions we are testing plausible, and are the predictors we use to access these models valid? We often test constructs: for instance, GDP is commonly used as a proxy for many other things. While this had some surely some merits in the past, it also draws an increasing amount of criticism. Another example would be the Intelligence Quotient, which may tell you something about people, but more often than not is misleading and judgmental. Have a look at the entry [[To Rule And To Measure]] to learn more.
+First, and many would argue most importantly, are the questions of plausibility and validity. Are the questions we are testing plausible, and are the predictors we use to access these models valid? We often test constructs: for instance, GDP is commonly used as a proxy for many other things. While this had some surely some merits in the past, it also draws an increasing amount of criticism. Another example would be the Intelligence Quotient, which may tell you something about people, but more often than not is misleading and judgmental. Have a look at the entry [[To Rule And To Measure]] to learn more, and also remember the role that [[Data_distribution | data distributions]] can play.
 ==== The independent variable ====
-Equally, predictors may vary in their validity. This may be related to the way the predictors are measured, how constructed specific predictors are, or if they can actually serve as proxies to answer the questions we want to answer. Money can be a predictor for happiness, but it has clearly its limitations, and the amount of money needed to have a decent life may differ between countries.
+Equally, predictors may vary in their [[Experiments_and_Hypothesis_Testing#Validity | validity and reliability]]. This may be related to the way the predictors are measured, how constructed specific predictors are, or if they can actually serve as proxies to answer the questions we want to answer. Money can be a predictor for happiness, but it has clearly its limitations, and the amount of money needed to have a decent life may differ between countries. Also remember that different [[Data_formats | data formats]] are highly relevant in order to understand the nature of your data.
 ==== Random factors ====
@@ Line 22: / Line 22: @@
 ==== The model ====
-The next part of a statistical model that needs to be examined when discussing the limitations of statistics is the statistical test or model itself. Models differ in terms of their mathematical foundations. This can lead to some models being more complicated than others, relying on different measures of model performance, and are also sometimes differently utilised within different scientific communities or disciplines.
+The next part of a [[An_initial_path_towards_statistical_analysis | statistical model]] that needs to be examined when discussing the limitations of statistics is the statistical test or model itself. Models differ in terms of their mathematical foundations. This can lead to some models being more complicated than others, relying on different measures of model performance, and are also sometimes differently utilised within different scientific communities or disciplines. Data visualisations showcase how different branches of sciences have different ways to approach how to [[Introduction_to_statistical_figures | show data in figures]], which can be diverse and often follow specific norms within different scientific disciplines.
-This is difficult to explain in a nutshell, but let me try: basically, different historical developments in the diverse disciplines, as well as the specific relation of a discipline towards the respective theory and topics, enable differences in how statistics are being utilised by different disciplines. Yet, there are also examples of exchanges and even coevolutionary patterns. For instance, the [[ANOVA]] started within agricultural science, but equally unleashed its potential in psychology, medicine and ecology, as these all rely strongly on the [[Experiments and Hypothesis Testing|testing of hypotheses.]] Econometrics and engineering build more strongly on correlative testing, which is again related to the specifics of these disciplines. However, the difference between scientific disciplines regarding their statistical use does not always make sense, and is not necessarily rooted in reasonable decisions.
+This is difficult to explain in a nutshell, but let me try: basically, different historical developments in the diverse disciplines, as well as the specific relation of a discipline towards the respective theory and topics, enable differences in how statistics are being utilised by different disciplines. Yet, there are also examples of exchanges and even coevolutionary patterns. For instance, the [[ANOVA]] started within agricultural science, but equally unleashed its potential in psychology, medicine and ecology, as these all rely strongly on the [[Experiments and Hypothesis Testing|testing of hypotheses.]] Econometrics and engineering build more strongly on [[Correlations | correlative]] testing, which is again related to the specifics of these disciplines. Most simple tests are [[Simple_Statistical_Tests#Normativity | hardly being used these days]], a phenomena that unites almost all disciplines, yet sometimes even very [[Back_of_the_envelope_statistics | simple calculations]] can offer robust answers. However, the difference between scientific disciplines regarding their statistical use does not always make sense, and is not necessarily rooted in reasonable decisions.
 == Limitations of statistics within science ==
 '''Equally grave (and not necessarily always meaningful) is the [[Design Criteria of Methods|great divide]] between quantitative and qualitative branches within science.''' While we should acknowledge that there are clear limitations of knowledge driven from statistics, it is equally difficult to reject knowledge derived from quantitative methods altogether. It is well deserved that with the rise of positivism - which widely cemented its reign through quantitative methods and particularly statistics - a qualitative counter-culture established itself. However, this does by far not do justice to the highly creative and so desperately needed deeper unlocking of qualitative dimensions in research. [[Interviews]], [[Grounded Theory]], [[Ethnography|Ethnographic Observations]] and [[Content Analysis]] are examples of important methods being established that go way beyond previous methods, and some of these approaches have a [[History of Methods|long history]] of highly relevant applications in science. However, statistics is nevertheless embedded into a judgemental perspective, creating a dichotomy that has not always been of benefit for science. Some would argue that the positivist wars - and the subsequent science war and culture war - are on a long chain of interwoven conflicts that may have been solved by [[Bias and Critical Thinking|critical realism]].
-'''I believe that we have to acknowledge that the knowledge we create is subjective''', which one of the main claims of critical realisms. Still, our way of using this knowledge to make sense of the world and derive heuristics and concepts that tell us how we ought to act may be objective. For instance, I might acknowledge that our knowledge about the validity of a vaccine to prevent me from getting a certain disease is a snapshot in time, and governed by conservative and robust norms in research. Hence I might decide to trust this knowledge in order to not only protect myself, but also to contribute to the protection of society. The decision to protect myself can be seen as hedonism, while the decision to protect my surrounding and ultimately society as a whole can be declared my contribution to the social contract. This showcases that my intentionality can be rooted in different principles. My actions may be informed by statistics, but statistics can only serve as a basis for my ontology, not be ontology in itself.
+'''We have to acknowledge that the knowledge we create is subjective''', which one of the main claims of critical realisms. Still, our way of using this knowledge to make sense of the world and derive heuristics and concepts that tell us how we ought to act may be objective. For instance, I might acknowledge that our knowledge about the validity of a vaccine to prevent me from getting a certain disease is a snapshot in time, and governed by conservative and robust norms in research. Hence I might decide to trust this knowledge in order to not only protect myself, but also to contribute to the protection of society. The decision to protect myself can be seen as hedonism, while the decision to protect my surrounding and ultimately society as a whole can be declared my contribution to the social contract. This showcases that my intentionality can be rooted in different principles, and the way we conduct [[Case_studies_and_Natural_experiments | experiments]] showcases how these areas of science are still under development. My actions may be informed by statistics, but statistics can only serve as a basis for my ontology, not be ontology in itself.
 == Limitations of statistics within society ==
 '''This brings us to the complicated relationship between statistics and society.''' Through its contribution to agricultural science, medicine, economics and many other fields, statistics have served as one of the main methoetodological approaches to [[Scientific methods and societal paradigms|contribute to a growth-focused economy]]. It can be clearly argued that this led to manifold benefits, but also to problems, including destruction, degradation and inequalities. All the while, science became increasingly glorified in the middle of the 20th century. However, due to the many negative effects that statistics indirectly contributed to, a critical perspective on statistics increased within the societal debate. What in the past was perceived as factual knowledge was increasingly questioned, and even altogether rejected.
-'''It is clear that the knowledge derived from statistics has value, but also limitations ,for societal discourse.''' Within an increasing rise of dichotomies as part of the culture wars, science was often called out, or simply taken as an emotional counterpoint. The debate on climate change showcases that both sides argue strongly from an emotional perspective (if we think in terms of the extreme points within the debate). At the same time, both sides also make selective arguments, having different focus points in terms of the goals they are trying to achieve. Climate change activists try to argue based on scientific evidence to protect the planet. Climate change deniers often take a limited or blurry understanding of bits and pieces of information to create a counterpoint that focuses, for instance, on the rights of some few people to continue their current strategies. Both sides seemingly argue about so-called 'facts', which showcases the deep entrenchment and differences that currently riddle societal debates. Statistics, to this end, is often devalued by one side or the other, and becomes a collateral due to criticism that emerges from false understandings or implausible claims.
+'''It is clear that the knowledge derived from statistics has value, but also limitations for societal discourse.''' Within an increasing rise of dichotomies as part of the culture wars, science was often called out, or simply taken as an emotional counterpoint. The debate on climate change showcases that both sides argue strongly from an emotional perspective (if we think in terms of the extreme points within the debate). At the same time, both sides also make selective arguments, having different focus points in terms of the goals they are trying to achieve. Climate change activists try to argue based on scientific evidence to protect the planet. Climate change deniers often take a limited or blurry understanding of bits and pieces of information to create a counterpoint that focuses, for instance, on the rights of some few people to continue their current strategies. Both sides seemingly argue about so-called 'facts', which showcases the deep entrenchment and differences that currently riddle societal debates. Statistics, to this end, is often devalued by one side or the other, and becomes a collateral due to criticism that emerges from false understandings or implausible claims.
 '''This highlights that statistics in itself may not be normative if all people share the same understanding about it.''' Sadly, this is not the case, and hence the limitations of statistics within societal debates are not much more than a mirror of the lack of education and ignorance that many people have to endure. Statistics is thus a privilege that some people can benefit from, but to use this educational effort as a devaluation of people who do not share this privilege is a difficult point. It would be naive to say that if all people shared the same knowledge, we'd all come to the same conclusions. But would it not be wonderful if this was the case?
-I believe that ''power to the people'' is still one of the best idea we ever had. Yet, with it comes the responsibility to provide an education that enables all people to use their power to take decisions. Ethics suggests that these decisions ought to be rooted in principles. These principles can be informed by information that can originate from - among many other sources - statistics. During the pandemic, for instance, the interest in numbers increased among the broader public. At the same time, it also became clear that different numbers and changes in these numbers represented different aspects of the pandemic at different times. Sometimes it was the amount of spread of the disease (r-value), sometimes the number of occupied hospital beds, at other times the mortality, and sometimes the increase in cases. This created confusion, and even the experts were not agreeing which number was relevant at different times, or which combination of values. This showcases that statistics are still emerging, especially when it comes to taming wicked problems, and also highlights the gap between science and society. It does not make any sense to blame people that were never educated in statistics for not understanding it. It becomes clear that science still has a long way to go to explain knowledge from statistics in a sense that it becomes a contribution to the great transformation.
+I believe that ''power to the people'' is still one of the best idea we ever had. Yet, with it comes the responsibility to provide an education that enables all people to use their [[Glossary|power]] to take decisions. Ethics suggests that these decisions ought to be rooted in principles. These principles can be informed by information that can originate from - among many other sources - [[Why_statistics_matters | statistics]]. During the pandemic, for instance, the interest in numbers increased among the broader public. At the same time, it also became clear that different numbers and changes in these numbers represented different aspects of the pandemic at different times. Sometimes it was the amount of spread of the disease (r-value), sometimes the number of occupied hospital beds, at other times the mortality, and sometimes the increase in cases. This created confusion, and even the experts were not agreeing which number was relevant at different times, or which combination of values. This showcases that statistics are still emerging, especially when it comes to taming wicked problems, and also highlights the gap between science and society. It does not make any sense to blame people that were never educated in statistics for not understanding it. It becomes clear that science still has a long way to go to explain knowledge from statistics in a sense that it becomes a contribution to the great transformation, yet statistics should seize this opportunity and learn to contribute in a critical way.
 ==External Links==
 ===Articles===
+* [https://lifereconsidered.com/2018/04/25/six-major-limitations-of-statistics/ A nice take on statistics and its limitations]
 ===Videos===
+* [https://www.ted.com/talks/hans_rosling_the_best_stats_you_ve_ever_seen The classic Hans Rosling Ted talk]
+* [https://www.ted.com/talks/mark_liddell_how_statistics_can_be_misleading A Ted Talk about how statistics can be misleading]
+* [https://www.ted.com/talks/sebastian_wernicke_1_000_ted_talks_in_six_words The meat Ted talk]
+* [https://www.ted.com/talks/sebastian_wernicke_lies_damned_lies_and_statistics_about_tedtalks Ted Talk - Lies, dammned lies, and statistics]
+* [https://www.ted.com/talks/mona_chalabi_3_ways_to_spot_a_bad_statistic Ted Talk - Some suggestions to spot mistakes in stats]
 ===Podcasts===
-[[https://www.wnycstudios.org/podcasts/radiolab/episodes/91697-numbers | Radiolab 1]],[[https://www.wnycstudios.org/podcasts/radiolab/episodes/91684-stochasticity |1]], [[https://www.wnycstudios.org/podcasts/radiolab/articles/love-numbers | 3]] and [[https://www.thisamericanlife.org/88/numbers | 4]]
+* [https://www.wnycstudios.org/podcasts/radiolab/episodes/91697-numbers Radiolab Episode about numbers]
-The [[https://www.bbc.co.uk/programmes/p02nrss1/episodes/downloads | more or less]] podcast
+* [https://www.wnycstudios.org/podcasts/radiolab/episodes/91684-stochasticity Radiolab Episode about stochasticity]
+* [https://www.wnycstudios.org/podcasts/radiolab/articles/love-numbers another Radiolab Episode about numbers 3]
+* [https://www.thisamericanlife.org/88/numbers A this american life episode on numbers]
+* The [https://www.bbc.co.uk/programmes/p02nrss1/episodes/downloads more or less] podcast
 ----