Why Statistically Significant Isn’t Good Enough

In the not-so-distant past there was no easy or quick way to gather information from thousands of people. As a result, statistically significant survey populations emerged. Conceptually, this means talking to enough people to claim that your findings apply to the entire workforce, user group or constituency.

However, statistically significant sampling is misleading because it captures just a slice of data rather than revealing the full picture. It was used primarily because it was too expensive or difficult to reach everyone. With the advent of modern technology like machine learning and natural language processing technology, you can now gather data from everyone instead of a select few. Doing so can help supercharge your work and ensure it has real impact.

What could you do if you knew what was holding back every person in your organization from being successful?

Talking to an entire community provides greater value than just a sample in three ways. You can:

• Take informed action

• Gain community buy-in

• Identify outliers

Taking Informed Action relates to how easily you can use what you gathered from the target community to get to work and make progress. Statistically significant data gathering is anonymous — you don’t know which participants care about the problems identified. But in having conversations with everyone in your organization, you can uncover a number of valuable resources, like:

• Highly motivated and capable project leads

• Users to learn from and test with

• Experts and partners to leverage for knowledge and resources

Gaining community buy-in might be better referred to as bureaucracy hacking. This is the way to get everyone working with — not against — you. When you gather statistically significant data from only a sample of a community, everyone doesn’t feel heard. This means that regardless of what action you take, there will likely be a large group of people who feel that you are ignoring their needs. We have seen a massive decrease in this sentiment simply by asking everyone for input then telling them what we heard and have decided to do. We also gain buy-in by showing the community proof. For example, we share the total community reached and the number related to each issue identified.

Identifying Outliers. When you only survey a statistically significant sample of a community you miss the majority of outliers when you should instead be trying to include them To ensure you’re making an impact with the actions you choose to take, you want to gather input from people that don’t conform to the rest of the community. They are different for a reason, and that reason could reveal a huge blind spot. From outliers, we have gleaned critical insights about what is holding back an organization.

Now that technology allows us to gather information from everyone, why wouldn’t you?