Source: French to English Tester Published on: 2026-03-29
Source: The Conversation – France in French (2)– By Ikram Chraibi Kaadoud, XAI & Cognitive Sciences Researcher, Inria
This article is published in collaboration with Binary,the blog to understand the challenges of digital technology.
An AI system must always be supervised by a human, but that person must also be able to distinguish when they understand what the machine is proposing and when they can be influenced.
The contemporary governance frameworks of artificial intelligence (AI) rely on a rarely made explicit assumption: when a human operator receives the output of an AI system, they must be able to assess it in a meaningful way. The provisions of theEuropean AI Actrelated to high-risk systems require transparency, explainability, and human oversight.
Explicitly targeted are systems used in recruitment and evaluation of workers, access to social benefits, credit granting decisions, border control, administration of justice, and critical healthcare.
TheUnited States AI action plancalls for maintaining significant human control over AI decisions with important consequences. The principles of theOECD on AIplace the focus on the human being at the heart of their commitments.
These commitments are necessary but insufficient. They concern what AI systems must provide to human operators and leave entirely unanswered the question of what the latter must be capable of doing to act upon what they receive. This gap is not accidental. It is a structural blind spot in the current architecture of AI governance.
The implicit model of the human supervisor in most regulatory texts is that of a competent and attentive professional who, faced with precise and readable outputs, makes informed judgments. This is a plausible assumption in stable environments with low stakes and well-controlled conditions, but a fragile assumption in high-stakes contexts, subject to time pressure, and technically opaque—precisely the contexts in which AI systems are increasingly deployed.
For example, the nurse in the emergency room in charge of triage who receives a triage score produced by an AI system does not systematically have the explanations that generated it. The bank advisor who must decide within minutes to block an account based on an automated fraud alert potentially works with a proprietary model that he cannot query. The administrative officer who validates the allocation of social housing or an algorithmically prioritized benefit generally cannot explain why one file was ranked before another. The teacher who countersigns an automated exam grading does not have access to the criteria that produced the score. In each of these cases, human supervision is formally present—and substantially impossible.
Metacognitively aware operators
Metacognition – the ability to monitor and regulate one’s own cognitive processes – is the psychological substrate of effective supervision. A metacognitively aware operator knows when they understand something, when they are conjecturing, and when their judgment is shaped by factors they have not consciously registered. This ability cannot be assumed; it varies significantly among individuals, training, and situational pressures.
Research in human-automation interaction has documented a set of failure modes that specifically emerge when humans supervise automated or AI-powered systems. Automation bias – the tendency to overweight machine-generated recommendations over one’s own judgment – is one of the most robust findings in the field. In a frequently cited study, theResearchers Parasuraman and Rileyshowed in 1997 that humans systematically misuse (that is, make a bad use of or use inadequately or inappropriately) automation by applying it where it is unreliable, and neglect it where it would be beneficial – two types of errors that reflect a metacognitive calibration fault rather than an information provision fault. For example, in flight simulator experiments cited by these authors, pilots equipped with an automatic alert system shut down an engine in response to a false alarm – a decision they themselves had declared, before the experiment, they would never make based solely on an automated alert.
The challenge is exacerbated by the characteristics specific to contemporary AI systems. Kahneman’s work on dual-process cognition – also known as System 1/System 2, the two speeds of thinking – sheds light on this mechanism. Faced with an AI system that produces output with fluency and confidence, the human mind tends to engage a fast and intuitive process (the one we use for familiar and low-risk tasks), rather than carrying out a deeper analysis of the situation, which is longer, more reflective, more logical, and therefore more cognitively demanding.
More concretely, an explanation that seems plausible triggers cognitive responses different from an explanation that really is. When AI system explanations are synthetically fluent, numerically precise, and visually formatted as authoritative outputs,they precisely eliminate skepticismthat requires ameaningful supervision.
Perhaps counterintuitively, providing more explanations does not reliably improve human judgment of AI results. A research team, in arigorous experimental study, found that the explanations produced by the AI did not systematically improve the performance of the human-AI team, and degraded it in several conditions — notably when the explanations were technically accurate but cognitively incompatible with the way the operators formed their own judgments.
More concretely, on the task of sentiment analysis, the AI explained its judgment by highlighting words it had identified as positive or negative. However, human participants evaluated the tone of a text overall, taking into account the context and overall coherence — a process that highlighting individual words cannot reproduce. Here, the AI and the human arrive at their judgment via different paths: the AI identifies local elements (a word, a sentence), whereas the human constructs a holistic judgment (the whole text, the context, internal coherence). When the explanation provided reflects the machine’s logic rather than that of human reasoning, it does not give the operator the tools to assess whether the recommendation is reliable — it simply convinces them to follow it.
Explainability is thus a necessary but insufficient condition for effective supervision. What reduces the gap between the two is metacognitive maturity.
Three implications for AI governance
If metacognitive maturity is a real and variable property of human operators, then governance frameworks that impose explainability without taking an interest in the metacognition of operators are simply incomplete. According to the work in the scientific literature — including those in explainable AI, human-automation interaction, cognitive sciences, psychology, and human and social sciences —,three implications can be statedA:
-
Transparency focused on documentation is insufficient. This is not an intuition: this is what theresearchwatchfor thirty years. Thus, documenting and explaining the behavior of a system is not enough to ensure good human decisions without involving individuals in the design processes of these explanations and this documentation and taking into account the business context at the given moment. Controlled studies have even shown that “too many explanations” can degrade the performance of the human-AI team bydrowning relevant information in the noise.
-
The metacognitive qualification of operators should be considered as a component of AI governance. This is a gap that research has begun to identify, although no framework has yet been formalized.
More concretely, regulatory texts like the AI Act require that human supervisors be “competent,” but never define what that means – and in particular, no framework evaluates what researchers call metacognitive competence, that is the ability to detect flaws in one’s own reasoning when facing an opaque system, a competence that depends on training and context, not on raw intelligence. An important clarification is needed here. Talking about the metacognitive qualification of operators is not about questioning the value or intelligence of the people supervising AI systems. It is also not about ranking humans according to their ability to “think well.” Metacognition is neither a personality trait nor a value indicator. It is a situational skill, sensitive to context, training, cognitive load, and working conditions. For example, an experienced surgeon may have excellent metacognitive calibration in their field and be just as vulnerable to automation bias as a beginner when facing an opaque AI system in a context for which they have received no specific training.
- Metacognitive skills – knowing what one understands, detecting one’s own reasoning errors, regulating one’s cognitive strategies – vary among individuals and are not evenly distributed within the population, which creates a structural risk for safety. This is a hypothesis, formulated based on studies conducted in educational psychology, which has not yet been studied in the context of AI governance. This may be the next research direction that governments should actively encourage. Indeed, while organizations best equipped with material means and human resources can meet real supervision requirements, those that do not – not because their personnel are less capable, but because the conditions enabling the development of this situational competence have not been met – will produce a superficial, insufficient compliance, generating a false security that is particularly dangerous in critical areas.
Ikram Chraibi Kaadoud does not work for, advise, hold shares in, receive funds from any organization that could benefit from this article, and has declared no other affiliation than her research institution.
–ref. AI and metacognition: knowing when one can trust, or not, the machine is not always obvious –https://theconversation.com/ai-and-metacognition-knowing-when-you-can-trust-the-machine-or-not-is-not-always-obvious-279348