Automation Bias

The cognitive tendency of human reviewers to over-rely on automated/algorithmic output, accepting it as correct without sufficient independent assessment. Explicitly named in:

WP29 Guidelines on Automated Individual Decision-Making (2018, endorsed by EDPB) — identifies routine application of algorithmic outcomes as failing the "meaningful intervention" test.
EU AI Act Art. 14(4)(b) — high-risk AI systems must be designed so that the natural person assigned to oversight remains aware of the possible tendency of automatically relying or over-relying on the output produced (explicitly using "automation bias").

Article 22 GDPR requires meaningful human intervention. Per Lazcoz & de Hert 2023:

A human reviewer who routinely signs off on algorithmic output is not providing meaningful intervention.
Automation bias cannot be detected on a per-decision basis — there is no way to know if a human agent is affected by automation bias evaluating a single decision.
Detection requires institutional/statistical review — DPIAs, override-rate analysis, post-market monitoring data.

This is the bridge from individual case review to systemic accountability — the core of Lazcoz/de Hert's argument that human intervention without human governance does not work.

Causes

Per Sartor & Lagioia (2020) cited in Lazcoz/de Hert:

Cost-and-incentive structures push human reviewers toward routine approval.
"Humans are likely not to substantially review automated decision, when the cost of engaging with the review — from an individual or an institutional perspective — exceeds the significance of the decision (according to the decision-maker's perspective)."
Time pressure, volume, output framing all amplify the bias.

Mitigations

Authority and competence for the reviewer to actually change the decision.
Institutional review — statistical analysis of override rates by reviewer, time, decision class.
DPIA as continuous evaluation tool — track and remedy where automation bias appears systemic.
Reflection machines (Cornelissen et al. 2022, cited by Lazcoz/de Hert) — meta-systems that ask the reviewer about their decision strategy and prompt critical evaluation.
AI Act Art. 14(4)(b) — system designed to surface the tendency.

Relevance to BPM / process automation

In agentic BPM systems with embedded human approval steps, automation bias is the default failure mode. The cost of override (re-routing, escalation, justification) usually exceeds the cost of approval — creating systemic drift toward acceptance of agent recommendations regardless of correctness. Process designers must instrument override-rate tracking into the perceive–reason–act loop.

concepts/human-oversight · concepts/gdpr-article-22 · concepts/dpia · concepts/eu-ai-act · concepts/automated-decision-making