# Exploring Different Correlation Coefficients and Plotting Correlations in Python

THIS ARTICLE IS STILL IN EDITING MODE

## Contents

## Introduction

There are three types of correlation coefficients, namely Pearson correlation, Spearman's rho and Kendall's tau. The main difference that should be considered, which correlation coefficient to use, is that the Pearson correlation is based on the assumption that the data is normally distributed, linearly related and equally distributed about the regression line (homoscedasticity). On the other hand, Spearman's rho and Kendall's tau do not require any type of distribution of the data. In the following we will explore how the different types of correlations vary, depending on the distribution of the data, missing values, outliers and noise in the data. This helps to understand which properties of the data effect the correlation coefficient in which way.

For a deeper background about correlations, please refer to *Correlations*.

## Exploring and Comparing the Different Correlation Coefficients

### Effect of differently distributed data on correlation coefficients

In the beginning the necessary libraries will be imported.

import pandas as pd import numpy as np import seaborn as sns import matplotlib.pyplot as plt from matplotlib import pyplot as plt import scipy.stats from scipy.stats import pearsonr, spearmanr, kendalltau from scipy.stats import linregress

First of all the data have to be simulated. Different distributions can be created as follows:

x = np.arange(0, 50) # Linear distribution l = np.array([n * 2 + 40 for n in x]) # Quadratic distribution q = np.array([n ** 2 for n in x]) # Reciprocal distribution r = np.array([5 / (n + 3) for n in x]) # Cubic distribution c = np.array([(n - 25) ** 3 for n in x])

Then the scatter plots for all distributions are made.

# Create a 2x2 grid of subplots fig, axes = plt.subplots(nrows=2, ncols=2) # Plot linear distribution axes[0, 0].scatter(x, l) axes[0, 0].set_title('Linear Distribution') # Plot quadratic distribution axes[0, 1].scatter(x, q) axes[0, 1].set_title('Quadratic Distribution') # Plot reciprocal distribution axes[1, 0].scatter(x, r) axes[1, 0].set_title('Reciprocal Distribution') # Plot cubic distribution axes[1, 1].scatter(x, c) axes[1, 1].set_title('Cubic Distribution') # Adjust layout to prevent overlapping plt.tight_layout() plt.show() # Figure 1

There are different ways to store the correlation coefficients for the further comparison. This case is using a dataframe.

# Create a DataFrame to store correlation coefficients correlations = pd.DataFrame(index=['Pearson', 'Spearman', 'Kendall'], columns=['linear', 'quadratic', 'reciprocal', 'cubic']) # Calculate correlations for x and each distribution for col in correlations.columns: pearson_corr, _ = pearsonr(x, eval(col[0])) spearman_corr, _ = spearmanr(x, eval(col[0])) kendall_corr, _ = kendalltau(x, eval(col[0])) correlations.at['Pearson', col] = pearson_corr correlations.at['Spearman', col] = spearman_corr correlations.at['Kendall', col] = kendall_corr # Display the DataFrame print("Correlation Coefficients:") print(correlations)

Correlation Coefficients:

linear | quadratic | reciprocal | cubic | |
---|---|---|---|---|

Pearson | 1.0 | 0.967051 | -0.751652 | 0.916029 |

Spearman | 1.0 | 1.0 | -1.0 | 1.0 |

Kendall | 1.0 | 1.0 | -1.0 | 1.0 |

When looking at the correlation coeffiecients for the different distributions, we can see that the Pearson correlation varies depending on the type of distribution, while the Spearman and Kendall coefficients reflect high either positive or negative correlation.

To interprete this it is important to notice, that all of the distributions are monotonously increasing or in case of the reciprocal distribution monotonously decreasing. As the Spearman and Kendall coefficients are based on ranks, the type of distribution does not change the absolute value of the correlation coefficient. Pearson on the other hand assumes a linear distribution and therefore the correlation coefficient decreases (in absolute values) as soon as it differentiates from a linear distribution.

### Effect of outliers on correlation coefficients

The quadratic distribution from previous example was taken to show the effect of outliers on correlation coefficients. Various number of outliers will be created manually by changing the values of some datapoints from the initial distribution. The results are reflected at the scatter plots (Figure 2).

*Figure 2: Scatter plots with different outlier effect*

Correlation coefficients:

q | o1 | o1f | o1x | o2 | o3 | |
---|---|---|---|---|---|---|

Pearson | 0.967051 | 0.788446 | 0.442784 | 0.755311 | 0.622115 | 0.498832 |

Spearman | 1.0 | 0.896182 | 0.896182 | 0.99904 | 0.730822 | 0.587255 |

Kendall | 1.0 | 0.924898 | 0.924898 | 0.993469 | 0.731727 | 0.540033 |

When exploring how outliers effect the correlation coefficients, we can see that the number of outliers and position(s) of the outlier(s) effect the absolute value of the coefficient. The Pearson Coefficient seems to be more sensitive to the distiance of the outlier. This can be seen when comparing o1 and o1f. The outlier being more far away from the distribution does not change the correlation coefficient according to Spearman or Kendall. The Pearson correlation however drops from 0.79 to 0.44. One can even see that this one outlier being very far away effects the Pearson coefficient stronger than many outliers being relatively close (compare o1f and o2). Generally adding more outliers (o2 to o3) similarly effects all correlation coefficients.

### Effect of noise on correlation coefficients

At the next step the noise will be added to the linear distribution. Here random samples from a normal (Gaussian) distribution with various values of standard deviation`(2, 20, 80)`

are implemented: `np.array([n * 2 + 40 for n in x]) + np.random.normal(0,2,x.shape)`

.

Figure 3 represents a different degree of noise.

*Figure 3: Scatter plots with different noise effect*

Correlation coefficients:

l | n1 | n2 | n3 | |
---|---|---|---|---|

Pearson | 1.0 | 0.997644 | 0.870757 | 0.465237 |

Spearman | 1.0 | 0.997791 | 0.87563 | 0.471789 |

Kendall | 1.0 | 0.97551 | 0.691429 | 0.32898 |

Here it is quite clear to see that adding noise effects the correlation coefficients by decreasing its absolute value. Kendall's tau seems to be the most sensitive coeffecient regarding noise.

### Effect of missing values on correaltion coefficients

Some values will be removed from the Cubic distribution, imitating missing values (scatter plots are illustrated in figure 4).

*Figure 4: Scatter plots with different missing values effect*

Correlation coefficients:

c | m1 | m2 | m3 | |
---|---|---|---|---|

Pearson | 0.891438 | 0.867976 | 0.821374 | 0.967922 |

Spearman | 0.90108 | 0.855109 | 0.793557 | 0.989474 |

Kendall | 0.776327 | 0.714588 | 0.643697 | 0.936842 |

By looking at the Correlation coefficients table, we can see that deleting of values does have an effect on the correlation, depending on which datapoints were removed. For example, the Pearson coefficient is the highest for the deletion m2, while for the deletion m3 the Spearman coefficient is the highest.

### Summary

The most important understanding of this exploration should be how the Pearson coefficient differs from the Spaerman and Kendall coefficients due to the assumption of a normal distribution. This assumption makes it more sensitive for varying underlying distributions and outliers. Whenever your underlying distribution is not a normal distribution, you should therefore prefer the Spearman or Kendall coefficient over the Pearson coefficient. The Spearman correlation coefficient is the more widely used one.

## Visualising Correlations

After exploring how different properties of the distribution of the data might effect the correlation coefficient, let us now have a look at different ways of visualising correlations. This will be done in Python. Please refer to this entry on correlation plots if you want to get an overview on visualisations using R.

### Line Chart

We have already seen that we can get an intuituion of the correlation by simply looking at the scatter plot of two variables. If you want to explicitly map the correlation within this scatter plot you can do this by adding a regression line. Addintionaly, you can put the value of the correlation coefficient in the title of the plot.

The following example will use the dataset, reflects the relationship between income and happiness of 111 countries (Kaggle Income and Happiness Correlation).

# Scatter plot for income and happyScore plt.scatter(happy_data['avg_income'], happy_data['happyScore']) # Add a regression line slope, intercept, r_value, p_value, std_err = linregress(happy_data['avg_income'], happy_data['happyScore']) # Create a regression line using the slope and intercept regression_line = slope * happy_data["avg_income"] + intercept # Plot the regression line plt.plot(happy_data['avg_income'], regression_line, color='red') # Add the correlation coefficient in the title plt.title('spearman correlation='+str(round(scipy.stats.spearmanr(happy_data['avg_income'], happy_data['happyScore'])[0],2))) # Add labels for x and y axis plt.xlabel('Average Income') plt.ylabel('Happy Score') plt.show() # Figure 5

*Figure 5: Scatter plot of HappyScore and Average Income with regression line*

### Scatterplot Matrix

If we want to get an overview of scatter plots for all numeric variables in our dataset, we can do so by simply using the command `sns.pairplot(name_of_dataset)`

.
If you want to edit the pairplot, you can find further information about the seaborn.pairplot here.

For example, you can select the variables that should be included, by using the command `vars`

.

# Plot the scatterplot matrix for selected variables sns.pairplot(happy_data, vars=["adjusted_satisfaction", "avg_income", "income_inequality", "happyScore", "GDP"]) # Figure 6

*Figure 6: Pairplot of happy_data*

### Correlation Matrix / Heatmap

Lastly, we will have a look at the heatmap in Seaborn. Here you can find further information about the searborn.heatmap. The heatmap is suitable for getting a first impression of the data and getting an overview of which variables might be related.

For instance, you can:

- specify the type of correlation:
`method="spearman"`

; - add values to the cells:
`annot=True`

; - change the colour map:
`cmap="coolwarm"`

; - add lines between the ceels with specific width:
`linewidths=0.5`

.

# Plot the heatmap according to the adjustments sns.heatmap(happy_data.corr(method = 'spearman', numeric_only=True), annot=True,cmap='coolwarm', linewidths=0.5) # Figure 7

*Figure 7: Correlation Matrix of happy_data*

## Conclusion

This entry helps to gain a deeper understanding of correlation coefficients, with respect to which properties might effect them and how to plot them. You might want to follow this link for further insides about the connection between causality and correlation.

The author of this entry is Nina Dürfeldt.