Noise: A Flaw in Human Judgment

Intro

Noise: A Flaw in Human Judgment focuses on errors in human decision processes that impact our judgment. The book can be considered an extension of the famous Thinking Fast and Slow moving beyond individual and exploring impacts of those errors on organisations and societies.

The authors distinguish between bias - systematic deviations between people, and noise, which is a random scatter. To understand errors in judgment, we must understand both bias and noise. Often, noise is a more important problem for the organisations and is therefore the main focus of the book aiming at improving our understanding of noise and proposing some ways to mitigate it.

Key Ideas

Judgment is difficult, because the world is complicated and uncertain.

Objective ignorance is a matter of fact and it makes perfect prediction nearly impossible. Overconfidence bias makes people underestimate the impact of objective ignorance.

The level of noise (judgment disagreements) is much greater than we expect.

Reducing noise can significantly enhance decision-making quality, fairness, and efficiency.

Efforts at noise reduction are often objected and run into difficulties. An example of the comprised effort is "Sentencing Reform Act of 1984" - a reform initiated by a famous judge Marvin Frankel to reduce the level of noise in the US legal system.

Introduction to Noise

Definition: Unwanted variability in professional judgments.

Examples: Different doctors giving different diagnoses for the same symptoms.

Wanted variability is not a noise, it can be film critics; wine tasters having different opinions; teams competing to generate innovative solution for a problem. This is expected and wanted variability.

Distinction from Bias: While bias is a systematic error, noise is a random scatter.

Examples: Bias is like a miscalibrated scale, noise is like a scale that gives random readings.

Bias vs Noise

Source: Tim Isaksson, CC0, via Wikimedia Commons

Important characteristic of noise is that it can be recognised and measured while knowing nothing about the target or bias. To measure noise, we need to have multiple judgments of the same problem, we don't need to know a true value.

To understand error in judgment we need to understand both bias and noise.

• Mean squared error (MSE) is the best method to evaluate total error

• Percent concordant is a measure of predictive quality of the variables, more intuitive that to interpret than correlation. It is used to evaluate predictive judgments. I wrote about this in: #41 - Understanding Statistical Relationships in Social Sciences

Types of Noise

Level Noise: Differences in the average level of judgments by different judges.

Example: Different judges giving varying sentences for similar crimes.

Pattern Noise

Stable Pattern Noise: Variability in the pattern of judgments made by the same judge in different situations due to his unique features such as values or principles he follows.

Example: A judge being stricter with shoplifter or other specific type of crime.

Occasion Noise: Variability in judgments made by the same judge related to random events, such as mood, stress, fatigue (random error).

Example: A doctor making different diagnoses based on their mood or time of day.

Components of total error

Causes of Noise

Heuristics and biases explained in Thinking Fast and Slow are the main reasons why both bias and noise in judgment are created. Noise is created because people differ in their use of heuristics and psychological biases.

Cognitive Factors: Individual differences in perception, information processing, and reasoning.

Examples: Different levels of knowledge, experience, and intuition.

Environmental Factors: Contextual influences, such as mood, weather, and even the time of day.

Examples: A bad mood affecting judgment, decisions varying with weather conditions.

Social Factors: Influence of other people's opinions and behaviors.

Examples: Groupthink, peer pressure, social norms affecting individual decisions.

Impact of Noise

Economic Costs: Inefficiency and financial loss due to inconsistent decisions.

Examples: Financial discrepancies in insurance claims, inconsistent loan approvals.

Social Costs: Unfair and unequal treatment in areas such as justice, healthcare, and hiring.

Examples: Disparities in criminal sentencing, uneven healthcare treatment outcomes.

Detecting Noise

Noise Audit: A systematic examination of variability in judgments to identify and measure noise.

Steps: Collecting data, analyzing patterns, identifying inconsistencies.

Focus on the process allows to evaluate quality of non verifiable judgments.

Reducing Noise

Choose the right people

• Trained

• Intelligent

• Right cognitive style: actively open minded and willing to learn from new information

• Authors emphasise limitation of experts in predictive judgment, which echoes arguments from Thinking Fast and Slow.

Decision Hygiene: Practices aimed at reducing variability without necessarily improving accuracy. They can be used by judges or, in case of group decisions by decision observers.

Humans are more noisy than statistical models and mechanical procedures. Models and procedures eliminate pattern noise.

Guidelines and Checklists: Standardizing processes to ensure consistency.

Examples:

• Surgical checklists, standardized interview questions, information sequencing (to limit the formation of premature intuitions);

• Dialectical bootstrapping: assume your first estimate is wrong, why? what does it imply? make a second estimate.

Structuring complex judgment

• Decomposition - breaking down the decision into components.

• Independence - information on each assessment gathered independently.

• Delayed holistic judgment - do not exclude intuition but delay it until structured evaluation is completed.

Example: recruitment processes in Google, the Mediating Assessment Protocol developed by the authors.

Aggregation of Judgments: Combining multiple independent judgments to average out noise.

Examples: Consensus decision-making, averaging forecasts, Delphi method, Good Judgment Project. Check this post for more: #26 Superforecaster mind

Statistical Models: Using algorithms and data-driven approaches to minimize human variability.

Examples: Predictive analytics in hiring, algorithm-based medical diagnoses.

Using a common frame of reference grounded in the outside view

Favouring relative judgment and relative scales

Tools

In the appendices one can find a number of practical tools, aimed at managing noise level.

• Noise Audit

• A checklist for decision observer

• Correcting predictions procedure