Elektrine lite

← Feed

@zenkat@sfba.social

2026-05-13 14:02 UTC

@simon_brooke@mastodon.scot "They do not have a semantic layer; they do not have a model of what any sequence of tokens mean" is a strawman argument. It shows a fundamental misunderstanding of how LLMs work, and why they are able to produce such remarkable results. LLM encode tokens into high-dimensional vector embeddings that reflect human meaning. eg, The embedding for "king" and "queen" are very close in vector space. Even more interesting, mathematical operations on these vectors are semantically meaningful, eg "queen" - "king" == "woman" - "man". But the encoding of meaning goes even deeper in the "self-attention" mechanism of modern Transformer architectures. They add all of the context of the tokens of a session into a huge "KV matrix", which extends the base semantics of each token with the context of all the proceeding tokens. So the embeddings for "black king to C5" become associated with chess, etc. LLMs are statistics designed to encode semantics. It's fundamental to how they work.

Replies (1)

  • @simon_brooke@mastodon.scot 2026-05-13 19:44

    @zenkat@sfba.social Frankly, I think your response shows a fundamental misunderstanding of what semantic means. Yes, certainly, decompiling an LLM will show that, for example, 'king' does cluster close to 'queen' (and also to 'rook', and thus to 'raven'). But it can't explain *why* 'king' clusters close to 'queen', or explain the context in which the association with 'rook' is, and is not, relevant.

    Open ##2702402