NPS Score: Quality over Quantity

“This sample size is too small… we cannot gain any insights from it.” But is it really true that we cannot gain valuable insights from small samples?

To be more specific, is a 40-person sample too small to be meaningful for your NPS? The short answer is no. You can draw valuable insights by tracking your NPS, even with a limited number of respondents in any of your NPS surveys (i.e. channel, brand, eNPS, etc.). A 40-person sample may even be just as reliable as a 100-person sample, even if both contain high-quality data (i.e. conscious feedback given by real customers).

How is this possible? It all comes down to your sample’s variance. In other words, having a low-variance sample of 30 people is not the same as having a high-variance sample of the same size. The higher the variance of your sample, the less reliable your NPS will be.

Having a high-variance sample may indicate that you’re probably not providing a very consistent experience to your customers. For example, let's consider the following sample NPS rates from two different channels (e.g. retail stores vs. call center).

Channel	Sample NPS rates	Sample Standard Deviation	NPS Score
Call Center	10, 10, 10, 10, 10, 9, 9, 9, 9, 9, 9, 9, 9, 9, 2, 2, 2, 2, 2, 1, 1, 1, 1, 0, 0, 0, 0, 0	4.23	0
Direct Retail	9 , 9, 9, 9, 9, 9, 9, 9, 8 , 8, 8, 8, 8, 8, 8, 8, 8, 8, 7, 7, 6, 6, 6, 6, 6, 6, 6, 6	1.17	0

Both channel NPS scores are the same! However, it's clear that something's going on with your call center channel. Customers seem to have a love/hate relationship with it.

I don't know about you, but even if both NPS scores are 0 for both channels, I would feel much better about the direct retail channel. Why? It's all about consistency.

A lower variance in your sample indicates that you are providing a more consistent experience to your customers.
NPS scores can be misleading and less reliable, especially if the variance in your sample is high.

Rule of thumb

In general, the following rule of thumb can be followed:

For < 30 responses, NPS scores are unreliable. Get more responses and call back your customers.
For > 40 responses, you can track NPS if the sample standard deviation is below 1.2.
For 80 to 100 responses, you can track NPS if the sample standard deviation is below 2.2. Higher deviations may increase NPS error even with 100 responses.
For over 300 responses, NPS scores are very reliable. High variance samples are not ideal though, because they show inconsistent customer experience and loyalty.

If you want to know where all this comes from, keep reading.

Proof

1. The NPS Score is really a sample mean

The NPS formula is defined as follows

\text{NPS} = \frac{\text{\# promoters}}{\text{\# of respondents}} - \frac{\text{\# detractors}}{\text{\# of respondents}}

The NPS Score is an estimator of the true NPS score of your population, which could be your whole channel, brand, or program population. As an estimator, your NPS score has a Margin of Error (MoE), which can be calculated. To understand this, we firstly need to realize that the NPS can be thought of as the sample average of random variable $\lambda$ , which can get any of the following values according to the NPS rate $r_i$ given by a customer.

\lambda(r_i) = \begin{cases}-1 & \text{if } r_i \leq 6 \\0 & \text{if } 7 \leq r_i \leq 8 \\1 & \text{if } 9 \leq r_i \leq 10 \\\end{cases}

Which is equivalent to:

\lambda(\text{customer}) = \begin{cases}-1 & \text{if customer is detractor } \\0 & \text{if customer is passive } \\1 & \text{if customer is promoter } \\\end{cases}

According to that, the NPS formula can be defined as the sample mean (also known as the average) of $\lambda$ :

\text{NPS} = \frac{\sum_{i=1}^{n} \lambda(r_i)}{\text{\# of respondents}} = \text{sample mean of }\lambda = \bar{\lambda}

Defining the NPS as a sample mean allows us to easily determine the margin of error for our NPS Score.

2. Calculating the error of your NPS Score

The margin of error indicates how much the NPS score may deviate from the true population NPS score. For instance, if your NPS score is 60 and its margin of error is 5, you can state with 95% confidence that the actual NPS score of the population falls between 55 and 60. In other words, a small margin of error is desirable. Imagine if I told you that my NPS score is 20, but with a margin of error of 40. You would likely be quite disappointed, as this data tells you nothing. The true population NPS could be anywhere between -20 and 60!

From statistics the margin of error, assuming a 95% confidence interval, is defined as:

\text{MoE}=1.96\sqrt{\frac{s^2}{n}}

Where:

$n$ is the size of your sample

$s^2$ is the sample variance of $\lambda$ , which is defined as

\text{VAR}(\lambda) = s^2 = \frac{1}{{n-1}} \sum_{i=1}^{n} (r_i - \bar{\lambda})^2 = \frac{1}{{n-1}} \sum_{i=1}^{n} (r_i - \text{NPS})^2

From all the equations we’ve listed so far, we can infer a formula for the NPS margin of error:

\text{NPS Margin of Error}=1.96\sqrt{\frac{\text{\# Promoters }(1-\text{NPS})^2+\text{\# Passives }(0-\text{NPS})^2+\text{\# Detractors }(-1-\text{NPS})^2}{n(n-1)}}

3. NPS Error: Sample Size & Variance Dynamics

As previously mentioned, you can track your NPS score with a sample size of 40 respondents, provided that your sample standard deviation is below 1.2. To illustrate this, consider the graph below, which shows how the error of your NPS changes with different sample sizes and sample standard deviations:

As depicted in the graph, larger sample sizes consistently result in smaller margins of error, regardless of the diversity and dispersion of respondents' answers. For instance, in a survey of 300 individuals, the NPS score will have an error no greater than 10, even with a sample standard deviation as high as 4.

Interestingly, if you observe the gray horizontal line, you'll notice that the margin of error for a 40-person NPS survey can be comparable to that of a 100-person NPS survey. This occurs when the sample standard deviations for both surveys are 1.2 and 2.2, respectively (indicated by the points where the gray horizontal line intersects both the 40-person and 100-person survey lines).

In this iPython notebook, I conducted a simulation with 10,000 iterations to explore the relationship between standard deviation values and margin of error. Surprisingly, when the standard deviation is around 1.2 ± 0.1, the margin of error tends to be around 15. While 15 may initially seem high, it's worth noting that even with larger sample sizes like 300, it's possible to achieve a margin of error as high as 10.

One crucial point to highlight here is that as the sample size increases, the margin of error becomes less noisy and more predictable. In the graph, you can observe that a 10-person survey exhibits significant inconsistency and noise. Even with low sample variance, the margin of error demonstrates chaotic-like behavior. However, as the sample size approaches the 40 to 100 range, the margin of error becomes more predictable, making the NPS score more reliable.

Conclusion

In conclusion, tracking your NPS score can provide valuable insights into your customer experience and loyalty, even with a small sample size. However, you need to consider not only the number of responses, but also the variance of your sample. A low-variance sample indicates a more consistent and reliable NPS score, while a high-variance sample may signal a problem with your customer service or product quality.

By using the formula and graph presented in this article, you can easily calculate the margin of error for your NPS score and determine how confident you can be about your results.

Remember, it’s not just about quantity, but also quality.