The Auditory Uncanny Valley (Voice & AI Assistants)

From Visual to Auditory

Mori's 1970 curve mapped human likeness against affinity for visual appearance. The same logic applies to sound: a voice is a rich signal carrying identity, emotion, and intent. When a synthetic voice approaches but fails to reach human parity, the mismatch between expectation and reality produces the same cognitive dissonance described in The Uncanny Valley (不気味の谷現象).

The key difference is temporal: where visual uncanniness is often static or movement-based, voice is inherently dynamic — it unfolds over time, making the violation of expectation a moment-by-moment experience.

The Auditory Curve

Mechanisms at Work

1. Predictive Coding & Expectation Violation

When a voice sounds almost human, the brain activates a full set of expectations: natural breath patterns, appropriate emotional inflection, correct rhythm and pacing. A neural TTS voice that stumbles on an unusual word, inserts an unnatural pause, or delivers a flat emotional response in a high-stakes context triggers a strong prediction error — the auditory equivalent of a humanoid robot's jerky movement.

2. Categorisation Ambiguity

A clearly robotic voice is safely categorised as "machine." A clearly human voice is "person." A voice that hovers in between — too expressive to be a tool, too artificial to be a person — resists easy categorisation, producing the same conflicting neural signals as a visual android.

3. Amplification by Interactivity

Mori theorised that movement amplifies the uncanny effect. In voice AI, real-time interactivity plays the same role. A static recording of a synthetic voice may be tolerable, but a conversational AI that hesitates, interrupts, laughs at the wrong moment, or changes tone mid-sentence falls much deeper into the valley. The interactive context raises the stakes on naturalness.

Real-World Examples

• Early TTS (Microsoft Sam, DECtalk): Safely on the ascending slope — clearly a machine, no discomfort.
• Modern Neural TTS (ElevenLabs, Google Duplex): So human-like that small glitches — a metallic breath, an unnatural laugh, a mis-timed pause — feel distinctly eerie. Google Duplex's restaurant reservation calls triggered public unease precisely because the voice was too human.
• AI Assistants (Siri, Alexa, ChatGPT Voice): Default voices are deliberately stylised to stay on the safe side of the valley. When users switch to more "natural" voices, minor imperfections become more noticeable and more unsettling.
• Deepfake Voice Cloning: The auditory equivalent of the Geminoid series — near-perfect replication with subtle imperfections that trigger revulsion, especially when the listener knows the voice is synthetic.
• Emotionally Expressive AI (Replika, Character.AI): Voices that sound empathetic but deliver nonsensical or contextually inappropriate responses create a jarring mismatch between vocal tone and semantic content (see The Eliza Effect).

Is the Auditory Valley Shallower?

Some research suggests the auditory uncanny valley may be less severe than the visual one. Humans are accustomed to imperfect voices — phone call static, accents, speech impediments, emotional fatigue — so the tolerance band for vocal imperfection is wider. However, the effect becomes more pronounced as the AI's conversational behaviour (timing, turn-taking, empathy) approaches human levels, because behavioural uncanniness compounds acoustic uncanniness.

Design Implications

• Deliberate stylisation: Keeping a voice recognisably synthetic (adding a slight robotic timbre or rhythmic pattern) can increase affinity by avoiding the valley entirely.
• Transparency: Users report less discomfort when they know they are speaking to an AI. Deceptive realism backfires.
• Consistency: A voice that is consistently robotic is preferred over one that is inconsistently human-like.
• Error handling: How a voice handles mistakes (hesitation, self-correction, apology) matters more as realism increases.

Backlinks

• The Uncanny Valley (不気味の谷現象) — Masahiro Mori's original robotics hypothesis serves as the direct visual blueprint for the auditory uncanny valley, plotting how affinity collapses into revulsion when artificial simulation approaches human authenticity.
• The Eliza Effect — The cognitive inverse of the auditory uncanny valley: while subtle vocal imperfections trigger visceral revulsion and hypercritical rejection, the ELIZA effect demonstrates how readily human psychology projects authentic consciousness and empathy onto crude textual programs.
• On the Psychology of the Uncanny (Zur Psychologie des Unheimlichen) — Ernst Jentsch's 1906 foundational analysis of intellectual uncertainty regarding whether an entity is alive or an automaton, directly anticipating the cognitive hesitation triggered by neural speech synthesis.
• Theory of Mind - The Cognitive Architecture of Social Intelligence — Documents the neurological mechanisms (such as rTPJ and mPFC activation) involved when human social cognition attempts to interpret conversational agents as intentional minds.

Resonant Notes

The following notes in the vault share direct thematic, sociological, and cognitive connections with The Auditory Uncanny Valley:

🔴 Cognitive Architecture & Perceptual Ambiguity

• The Uncanny Valley (不気味の谷現象) — Masahiro Mori's foundational framework establishing how human affinity plunges into revulsion when an artificial entity approaches, but fails to attain, complete human authenticity.
• On the Psychology of the Uncanny (Zur Psychologie des Unheimlichen) — Ernst Jentsch's concept of intellectual uncertainty and cognitive hesitation over the boundary between animate life and mechanical automation.
• Freud's The Uncanny (Das Unheimliche) — Sigmund Freud's psychoanalytic reading of the uncanny (unheimlich) as the return of the familiar that has undergone repression, manifesting in the eerie dread elicited by automated doubles and disembodied voices.
• Theory of Mind - The Cognitive Architecture of Social Intelligence — Analyzes the evolutionary mismatch that occurs when human social cognition encounters synthetic speech agents capable of mimicking human prosody without possessing subjective consciousness.
• Cognitive Dissonance — Leon Festinger's theory explaining the psychological tension that arises when listeners intellectually recognize a voice as synthetic while unconsciously processing its conversational cues as biological.

🟡 Synthetic Simulation & Human-Computer Interaction

• The Eliza Effect — Demonstrates the human tendency to project intentionality and understanding onto algorithmic outputs, establishing the opposite psychological pole to uncanny aversion.
• Defining a Simulacrum — Jean Baudrillard's hyperreality; synthetic voice cloning and deepfakes represent simulations detached from biological referents, blurring the boundary between authentic human expression and artificial reproduction.
• Postscript on the Societies of Control — Gilles Deleuze's diagnosis of continuous digital modulation and behavioral capture, reflecting how real-time voice assistants monitor and shape human conversational dynamics.

References

• Masahiro Mori / The Uncanny Valley / IEEE Robotics & Automation Magazine
• Karl MacDorman et al. / The Uncanny Valley in Speech / ResearchGate
• Ayse Pinar Saygin et al. / The thing that should not be: predictive coding and the uncanny valley / Oxford Academic
• Angela Tinwell / The Uncanny Valley in Games and Animation (chapter on vocal uncanniness)