Embeddings Are Not Taste: Why Vector Similarity Can't Capture What People Actually Want
Two-tower models map content into neat geometric spaces — but human preference is contextual, mood-driven, and contradictory. Where semantic retrieval ends and real discovery begins.
Modern recommendation stacks converge on the same architecture: two-tower models that learn to place users and items in a shared vector space, then retrieve nearest neighbors at inference. It’s elegant, scalable to catalogs of hundreds of millions, and foundational to nearly every major platform.
It’s also a lie — in the specific, useful sense that a map is a lie about a territory.
What the Geometry Gets Right
Embedding spaces are genuinely good at capturing stable semantic structure: genre adjacency, tonal family, production-budget tier, language. The distance between a slow-burn Scandinavian crime drama and a Korean thriller series is, in embedding terms, small — and that geometry is real.
What the Geometry Cannot Hold
| Taste dimension | Embedable? | Why it fails |
|---|---|---|
| Genre affinity | Mostly | Nearest neighbors cluster naturally |
| Mood-dependent preference | No | “Comfort watch on Sunday, thriller on Friday” is temporal, not spatial |
| Social context | No | Solo vs. family viewing selects different items for the same person |
| Deliberate exploration | No | The user who wants different today is a direction, not a point |
| Narrative novelty | Barely | Similarity optimizes for “more of the same” |
A single point in vector space is a fiction. The same viewer is multiple selves depending on time of day, social company, and emotional state — and collapsing that into one coordinate produces a recommendation that’s mediocre for everyone in the household at once.
The Fixes Platforms Are Deploying
- Session-conditioned retrieval — re-ranking by immediate context (time, device, company signals) rather than lifetime taste
- Multi-head user representations — maintaining several user vectors (mood-clusters) instead of one canonical profile
- Intent-first surfaces — explicit “something different” slots that escape the local neighborhood entirely
“Taste isn’t a location you can find with a compass. It’s a weather pattern — and the best systems forecast, not just locate.”
Deeper notes on vector retrieval and session modeling in Embedding Spaces & Semantic Search Notes.
Nearest neighbors are the floor, not the ceiling.