The architecture of human error

Good design is often judged by its functionality or its user-friendliness. But what about when design has to be more than just functional or delightful?

The hallmark of good design varies depending on who you talk to. Ask an engineer and she will tell you that functionality, that the thing does what it is intended to do, is most important. Ask a UX designer, and they may tell you the customer experience is most important. These two opinions make up the lion’s share of beliefs when it comes to design considerations.

But is that all that makes for good design? Or to rephrase the question, are there other dimensions of design that should be taken into consideration, which may, in certain circumstances, be the most important aspects of design?

It is not just enough for a system to be designed such that it works, or that it is a delight for the end user. It must also work consistently and be designed in a way which reduces human error, or at the very least has safeguards in place to recover from errors.

For example, take a nuclear power plant. Is it enough to design it so that it works? Or should it also be designed in such a way to minimize human errors which may, you know, create a nuclear melt down.

Take a look at the picture above. Countless dials, each presumably conveying critical information about the health of a nuclear reactor. Is this well designed? From the “functionality” perspective — yes, it works. It does what it is intended to do. But from a perspective of human error it couldn’t be worse.

The design makes it almost impossible to detect changes, to identify critical components, and most importantly, to make decisions based on the information. The result will be an inevitable catastrophe. There will be finger pointing. “Human Error” will be the cause. But to credit human error as the cause is to miss the point entirely.

Human Error is a response to design.

Not all mistakes are created equal. The type of mistake that occurs depends fundamentally on how the individual is thinking about things when when things go wrong. This way of thinking, we can call the level of cognition. Understanding these different error types — and their underlying mechanisms — is crucial for preventing them and designing better systems.

Skill-Based Errors

Skill-based errors are referred to as slips or lapses. They occur when somebody does something, but it is not what they intended to do. Skill-based activities do not require much conscious control and only require occasional progress checks at particular points in the activity. The price paid for this efficiency is that strong habits usurp intended behavior when attention is diverted by distractions and unfamiliar activities hiding in familiar contexts.

These errors include familiar situations like missing your exit on the freeway, mis-typing a word on the computer, and the well-known Freudian slip. When mistakes occur at the skill-based level, they usually occur before a problem is detected — that is to say, you don’t detect them until the error has occurred and it’s already too late. The slip of the tongue and missed exit are always recognized after the error has occurred, and not before.

Specific types of skill-based errors include:

• Double Capture Slips: Failing to properly execute a task due to overlooking or repeating a step in a sequence
• Omissions Following Interruptions: Failing to complete a task or missing crucial steps due to interruptions in the workflow
• Reduced Intentionality: Actions performed with less focus or intention, leading to errors in execution
• Perceptual Confusions: Errors arising from misinterpretation of sensory information, leading to incorrect actions
• Interference Errors: Errors caused by conflicting stimuli or information, leading to incorrect responses

Rules-Based Errors: When Good Logic Goes Bad

Rules-based thinking is a slightly higher level of cognition. It relies on what Daniel Kahneman called in his book Thinking Fast and Slow, “system 2” thinkin. This type of thinking is non-automatic. It relies on a logical decision-making system. Consequently, errors that occur at the Rule-based level are best described as cognitive mistakes. Errors occurring at this level happen after an issue has been detected and active problem solving, judgement, and decision making has been initiated.

Rule-based decisions, such as putting $40 into a gas tank every time you fill up, regardless of the cost of fuel, will result in error when the rule fails to fit and mold into current events and a deviation from past experiences is not recognized.

Rules-based errors fall into two main categories:

Misapplication of Good Rules

• First Exceptions: Errors occurring when individuals deviate from established rules or procedures based on initial exceptions, leading to inconsistencies
• Countersigns and Nonsigns: Errors involving the misinterpretation or misapplication of signs or indicators within a rule or procedure
• Informational Overload: Errors caused by an excessive amount of information, leading to difficulty in processing and prioritizing relevant data
• Rule Strength: Errors stemming from the reliance on rules that are either too weak or too strong for the given situation
• General Rules: Errors resulting from the application of overly general rules or guidelines that lack specificity
• Redundancy: Errors arising from the inclusion of redundant or unnecessary rules or procedures
• Rigidity: Errors stemming from inflexible adherence to rules or procedures, without considering situational nuances

Application of Bad Rules

• Encoding Deficiencies: Errors occurring due to inadequacies in encoding information into memory, resulting in incorrect recall
• Wrong Rules: Errors arising from the selection or application of incorrect rules or guidelines
• Inelegant Rules: Errors resulting from the use of poorly designed or cumbersome rules or procedures
• Inadvisable Rules: Errors stemming from the application of rules or procedures that are ill-suited for the specific context

One fun example of rules based errors can be found during the Cuban Missile Crisis. In 1963 the Russians started building missile bases in Cuba. Despite the secret nature of their work the failed to take basic even remedial precautions. They failed to camouflage the missile sites. They built straight, easily detectable roads. They dressed in Soviet military uniforms. Most telling of all, they left missiles out in the open.

The reason for all of this, was because they were well practiced in building missile sites in the Soviet Union, where these precautions were not needed. Throughout their construction they relied on the Rules-based thinking which was determined in Moscow and drilled into their heads behind the Iron Curtain.

The Strong-But-Wrong Phenomenon

Both Skill-based and Rule-based errors usually have a “strong-but-wrong” feature to their behavior. They fail because of a misapplication of a rule, a detection of a false similarity, or the non-appreciation of a meaningful difference in the environment or activity. This “strong-but-wrong” characteristic is crucial because it means that the person committing the error often feels confident in their actions — right up until the moment the error becomes apparent.

Knowledge-Based Errors

Mistakes happening at the highest level of judgement and decision-making are called knowledge-based errors. These mistakes occur because somebody have exhausted all reflexive skills, rules, and relevant knowledge. Because of this exhaustion, knowledge-based cognition relies on inference, intuition and constant feedback. The result is a “hit-or-miss” quality to the decisions. Sometimes the decision is right. Sometimes the decision is wrong. At this level, there is very little to predict, shape or improve the outcome of the decision.

Knowledge-based errors include:

• Selectivity: Errors stemming from selective attention or focus on certain information while neglecting relevant data
• Workspace Limitations: Errors due to limitations in cognitive workspace, such as memory constraints or attentional capacity
• Out Of Sight Out Of Mind: Errors occurring when important information is not readily accessible or visible
• Confirmation Bias: Errors arising from seeking or interpreting information in a way that confirms pre-existing beliefs
• Overconfidence: Errors caused by excessive confidence in one’s knowledge or abilities
• Biased Reviewing: Errors in evaluation or review processes due to biased perceptions or interpretations
• Illusory Correlation: Errors in perceiving correlations between events or variables that do not actually exist
• Halo Effects: Errors in judgment or decision-making influenced by overall impressions or biases
• False Causality: Errors in attributing causality between events or actions based on incorrect assumptions
• Extra Complex Problems: Errors resulting from the complexity of a problem, including difficulties in processing feedback

Violations

There is another type of human error that occurs, which the SRK model does not fully explain, but which is worth mentioning. These are violations. Violations are purposeful though deviations from the known best practices. They are not inherently bad. Violations can be committed for many reasons. Excluding outright sabotage, the intent of these violative actions are usually good like improving productivity, performance, or taking the path of least resistance.

Violations can be broken down into two categories:

Routine: Routine violations are part of an individual’s regular, habitual behavior. Routine violations result from two factors. First, the individual takes the path of least resistance, expending less energy, time, strength, etc.) and commits a violation of known procedures or best practices. Second, the individual’s environment does not punish the violation nor reward the correct action.

Exceptional: Exceptional violations occur as one-off events and in a particular, non-repeatable set of circumstances.

Error Recovery

The frequency and number of errors is not the only important difference between the three levels of thinking (Skills, Rules, and Knowledge). Another important distinction is how well an error can be handled and recovered from. at the lowest level of thinking, recovery is usually rapid and efficient, because the individual will be aware of the expected outcome of his or her actions and will therefore get early feedback regarding any slips that have occurred which may have prevented this outcome being achieved.

In the case of mistakes which occur at either the rule-based or knowledge-based level, the mistaken intention tends to be very resistant to disconfirming evidence. This means that people tend to ignore feedback information that does not support their expectations of the situation.

The Implications for Design

Understanding these different error types reveals that prevention strategies must be tailored to the specific cognitive level where errors occur:

• For skill-based errors: Focus on reducing interruptions, improving feedback systems, and designing interfaces that make critical actions more distinctive
• For rules-based errors: Emphasize training quality, rule clarity, and creating systems that help people select the right rule for the situation
• For knowledge-based errors: Provide better decision support tools, encourage systematic problem-solving approaches, and create environments that surface disconfirming evidence
• For violations: Address the underlying incentive structures and remove barriers that make rule-following unnecessarily difficult

The architecture of human error is not random, it follows predictable patterns based on how our minds work. By understanding these patterns, we can design systems that work with human cognition rather than against it.

The architecture of human error was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

This post first appeared on Read More