Why Diversity Leads to Better Decision-Making

Hi, apologies for the lack of writing for the past few months! I wrapped up my sabbatical and am deep into interviews. (Shameless plug: I’d love to speak with you if you’re looking for a data leader.) A frequent interview question is about building teams and organizations, including clarifying the most critical elements.

I focus on five elements when shaping teams and organizational culture, each of which builds on the other.

This article focuses on diversity and how a diverse team leads to better outcomes. And it makes an unusual analogy, the mathematical relationship of variance and correlation, to explain why diversity leads to better decision-making.

Overview

The inspiration for this article came during a late night while brushing up on the random forest algorithm. It is one of my favorite statistical learning techniques! Random forest uses a neat trick to decorrelate the inputs, leading to better predictions and inference when averaging the outputs. And I realized the same happens with diverse organizations—decorrelated inputs lead to better decisions.

Diversity means having a range of people with various life experiences. Different life experiences include racial, gender, ethnic, socioeconomic, cultural, physical, mental, and more. I am fortunate to have witnessed the benefits of diversity first-hand. The two largest organizations I built, with 15+ individuals, were more than 50% female and 50% underrepresented minority, with a fantastic range of educational backgrounds and life experiences. I have also been part of non-diverse organizations, heavy with group-think and common backgrounds.

Thought Experiment: Team Diverse and Team Boring

Let’s do a thought experiment to show the value of diverse inputs in decision-making. First, create two teams: Team Diverse and Team Boring. Each team has five people, and the final decision is an average of the five people on the team. I assume that increased diversity leads to decreased correlation among team members because diverse teams have more life experiences to draw on. Then we’ll simulate 10,000 independent decisions.

We’re also going to explore two scenarios with each team. The first scenario is where both team members have the same decision-making ability. For the second scenario, I adjusted Team Diverse to have better decision-making ability than Team Boring. Logically, diverse teams have a larger pool of candidates to hire from, which increases their probability of having a better team.

A summary of the teams and scenarios:

• Team Diverse (scenario 1): Five people, each with a decision-making ability of 50 and a normally distributed covariance matrix with a mean of 0 and standard deviation of 1.
• Team Diverse (scenario 2): Five people with decision-making abilities of 40, 50, 50, 60, and 75. They use the same covariance matrix from scenario 1.
• Team Boring (scenario 1): Five people, each with a decision-making ability of 50 and a normally distributed covariance matrix with a mean of 0.5 and standard deviation of 1. The slightly higher mean increases the positive correlations among the team members.
• Team Boring (scenario 2): Five people with decision-making abilities of 30, 40, 50, 50, and 70. They use the same covariance matrix from scenario 1.

blog-notebooks/diversity_and_variance.ipynb at main · jrgilbertson/blog-notebooks

A repository of various notebooks and code for www.jasongilbertson.com. - blog-notebooks/diversity_and_variance.ipynb at main · jrgilbertson/blog-notebooks

GitHubjrgilbertson

A reminder of random variables, expected value, variance, and correlation and how they are related to this thought experiment:

• Random variable: A variable whose possible values are outcomes of a random experiment. Each person is a random variable with an underlying normal distribution and a mean, or expected value, equal to their decision-making ability.
• Expected value: The mean of a random variable. Increasing the expected value in this thought experiment means making better decisions.
• Variance: Measures the spread, or dispersion, of a random variable around the mean. Reducing variance in this thought experiment means one is more consistent with a decision.
• Correlation: A normalized measure of how two or more variables are related. In this example, Pearson correlation measures the strength and direction of a linear relationship.

Let’s look at the correlations among the team members.

Fantastic! There is a good mix of positive and negative correlations for Team Diverse, while there are mostly positive correlations with Team Boring.

After 10,000 decisions, diverse inputs lead to a 52% reduction in variance for this thought experiment. The only difference between the two teams in scenario 1 is the correlation among team members. The risk of decision-making is cut in half merely by reducing the correlation of inputs into a decision.

For scenario 2, where Team Diverse has slightly better decision-making ability plus less correlation, it’s not even close. The expected value is significantly larger and has less variance. Imagine the competitive advantage when a team can make better and less risky daily decisions!

Final Thoughts

• Diversity is a crucial element of the hierarchy of effective organizational culture.
• Diversity means having a range of people with various life experiences. Different life experiences include racial, gender, ethnic, socioeconomic, cultural, physical, mental, and more.
• After 10,000 decisions, diverse inputs led to a 52% reduction in variance for this thought experiment. The risk of decision-making is cut in half merely by reducing the correlation of inputs into a decision.
• After 10,000 decisions, a diverse team composed of slightly more capable employees plus less correlation blows away a non-diverse team in making decisions.

Thank you for humoring me with this thought experiment. If you are curious about genuine research, here are three recent articles to explore:

• Why Diverse Teams Outperform Homogeneous Teams from Psychology Today
• Why Diverse Teams Are Smarter from Harvard Business Review
• Diversity wins: How inclusion matters from McKinsey