Spearman's Rank Correlation Coefficient

• The Logic of Ranks: By converting raw data into ranks (1st, 2nd, 3rd, etc.), the calculation focuses on the relative position of observations. This transformation effectively 'neutralizes' the distance between values, which is why a non-linear but consistently increasing curve yields a perfect Spearman correlation of $+1$.

• The Difference Variable ($d_i$): The core of the formula lies in $d_i$, which represents the difference between the rank of the $i$-th observation for the first variable and its rank for the second variable. If the ranks match perfectly across both variables, $d_i$ will be zero for all observations, indicating a perfect positive correlation.

• Sum of Squared Differences: Squaring the differences ($\\sum d_i^2$) ensures that all deviations are positive and penalizes larger discrepancies in ranking more heavily. This sum is then normalized against the total number of possible ranking permutations for a sample size $n$.

• Step 1: Ranking the Data: Assign a rank to each value in both datasets independently, typically from smallest to largest. If two or more values are identical (tied), assign each the average of the ranks they would have occupied (e.g., if two values tie for 2nd and 3rd, both receive a rank of $2.5$).

• Step 2: Calculate Rank Differences: For every pair of observations, subtract the rank of the second variable from the rank of the first variable to find $d_i$. It is a useful check to ensure that $\\sum d_i = 0$, as the sum of differences between two sets of the same ranks must always be zero.

• Step 3: Apply the Formula: Use the standard formula for cases without significant ties: $r_s = 1 - \\frac{6 \\sum d_i^2}{n(n^2 - 1)}$ where $n$ is the number of pairs of data. If there are many ties, the Pearson correlation formula should be applied directly to the ranks instead.

• Step 4: Interpretation: The resulting value $r_s$ will always fall between $-1$ and $+1$. A value of $+1$ indicates a perfect positive monotonic relationship, $-1$ indicates a perfect negative monotonic relationship, and $0$ suggests no monotonic association between the variables.

Spearman vs. Pearson Correlation

• When to choose Spearman: Use Spearman when your data is ordinal (like survey rankings), when the relationship appears non-linear but monotonic on a scatter plot, or when your data contains significant outliers that would distort a linear mean-based calculation.

• When to choose Pearson: Use Pearson when you are certain the relationship is linear and the data follows a bivariate normal distribution. Pearson is generally more powerful (statistically efficient) than Spearman when these parametric assumptions are met.

• The Zero-Sum Check: Always verify that $\\sum d_i = 0$ before squaring the differences. If your sum of differences is not zero, you have made an error in ranking or subtraction, and proceeding will lead to an incorrect final coefficient.

• Handling $n$: Be careful with the $n$ value in the denominator $n(n^2 - 1)$. Students often mistake $n$ for the total number of individual data points rather than the number of pairs; always count the number of rows in your data table.

• Sanity Check the Result: If your scatter plot shows a clear upward trend but your $r_s$ is negative, re-check your ranking order. Ensure you ranked both variables in the same direction (either both smallest-to-largest or both largest-to-smallest).

• Tied Ranks Caution: In exams, if ties occur, remember to use the average rank. Forgetting to do this and simply assigning the next integer rank is a common mistake that leads to inaccurate $d_i$ values.