AI as a black box

“A black box AI is an AI system whose internal workings are a mystery to its users. Users can see the system’s inputs and outputs, but they can’t see what happens within the AI tool to produce those outputs.”

IBM (2024)

Description and analysis

What is it?

A black box, when referring to an AI application, refers to a system whose inner workings are unclear, making it hard to interpret and trust its outcomes. The black color refers to the system’s opaqueness.

What does this metaphor suggest about AI?

This metaphor can suggest that AI is too complex for humans to understand. It  evokes a feeling of hopelessness and inevitability. This opacity also makes it difficult to know whether the outcomes of AI systems are reliable or not. However, this same complexity can also highlight the cleverness of the people developing these systems and make others accept them without questioning them.

Where does the metaphor commonly appear?

This metaphor is used whenever people are questioning the reliability of AI generated outcomes and predictions. It is also used to underline the importance of the technology by emphasizing its complexity.

What does this metaphor highlight?

This metaphor highlights the idea that AI’s internal workings are hidden from view. The visual logic implies that it can’t be seen into. In this way, it emphasizes the complexity of AI systems but also questions whether its outcomes are reliable.

What does this metaphor hide?

The metaphor can hide a lack of responsibility. When referring to AI systems as being a black box, developers and providers can evade responsibility by indicating that even they do not know how these systems work.

It minimizes human agency by framing AI behavior as an inherent mystery, It has been suggested that a better metaphor might be that of a veil, which is being drawn over human decisions and values, and invites us to lift it.

What makes (or could make) this metaphor work?

This metaphor draws on the cultural fear of the “ghost in the machine” and the intellectual tradition of cybernetics. Also behaviourism and aviation

Whose interests does this metaphor tend to serve?

The black box metaphor helps critical users advocate for more transparency. It also helps developers to evade accountability. Because it makes the lack of transparency seem like an unavoidable mystery it pre-empts scrutiny.

History

What are this metaphor’s origins (if known)? How has its use evolved over time?

The phrase ‘black box’ first emerged in 1947 as RAF slang for “navigational instruments” and was later extended to any sort of apparatus that operates in a sealed container, especially flight recorders from circa 1964. The black box concept was first formally integrated into the foundation of thinking machines and cybernetics in 1956.

Field

What field(s) does this metaphor tend to appear in?

Used broadly, especially in relation to people’s need to understand the outcomes of AI systems. 

Geographic source

If known, in which country or region did the metaphor first appear?

United Kingdom (Second World War); British and international theorists in the 1950s

Prevalence

Has this metaphor become a widely used term or it is niche?

Mainstream term, broadly used by the public and media

Additional notes

Related metaphors

Other metaphors that convey a similar meaning are AI as an iceberg.

Potential impacts

Using black box metaphor creates a sense of technological fatalism, making users feel that AI bias is an unavoidable mystery rather than a human design flaw. It can harm the public by creating “learned helplessness,” making people feel that AI is too mysterious to challenge even when it produces biased results.

Some argue that AI/LLMs are less opaque than previous black boxes, as we can see training data and we can test outputs. As Benjamin Riley said: “It’s maths not magic”. And as Leon Furze pointed out: “This metaphor is powerful because it captures both the complexity of AI systems and our limited ability to understand their inner workings. However, it can also be limiting if it leads us to view AI as completely unknowable or beyond analysis. In fact, there is excellent research trying to crack open the black box”.

Cultural nuances

German speakers frequently use the term Blackbox (or Black Box) as a loanword from English, both in everyday speech and in technical fields.

 

References

Author(s)

Who collaborated on this entry

Elja Daae, Brigitte Nerlich, Zofia Skassa

Category

Family to which each AI metaphor belongs.

Operation

This methodology considers the following families or categories:

  • Essence: What is AI? Metaphors where AI is a thing or entity (i.e. tool, brain)
  • Operation: How does AI work? Metaphors that represent AI’s processes or mechanisms (i.e. stochastic parrot)
  • Relation: How do we relate to AI? Metaphors that frame the relationship between people and AI (i.e. assistant).
  • Function: What does AI do, or what is it used for? Metaphors describing the use made of AI (tutor, translator)
  • Impact: What consequences can AI have? Metaphors that frame AI’s effects, (threat, transformative engine).

Humanity attribution

Level of human-like characteristics and agency attributed to AI. This scale goes from no / minimal anthropomorphism, through to full anthropomorphism, and beyond to deification.

1. Minimal: tool (calculator)

Other risks

What other risks can this metaphor carry?

Responsibility shift

Examples of usage

Examples of where this metaphor was used

“The Black Box Problem arises when the computing systems being developed in AI are opaque. This metaphorical way of speaking is grounded in the intuition that a system’s behavior can be explained by ‘looking inside’… they might be considered opaque in the sense that it is difficult to know exactly how these systems are programmed.”

Carlos Zednik (2019)

 

“If these black boxes are being used to make a high-stakes decision in lending, insurance, healthcare, or the judicial system, we have to decide whether we feel comfortable not knowing exactly why the decision was being made.”

Laura Blattner (2022)

 

Observatory articles
featuring this metaphor

Sorry, but nothing was found.