2026-08-23 07:53 UTC
@giacomo You've misunderstood the paper. LLMs _can_ be used as compressors, but that does not mean that LLM models _are_ compressing source data retrievably.
The reason Shannon's theorem proves this is simple. The model I used for the task in this thread is a 12B parameter model. It's much (much!) too small to store (compress) any relevant amount of source data so that it can be retrieved (decompressed) in the way you believe.
What you're referring to is known as "overfitting" and is something no LLM company wants. It means model parameters are wasted instead of used for the higher level concepts of _how_ to do things.
If you're actually interested in the topic we can continue, but considering you seem to believe you "know better" I'm not certain you're discussing in good faith here.
Replies (1)
-
@giacomo@snac.tesio.it 2026-08-23 17:59
😀 I know very well (and faced several times) curve overfitting: it's a term related to models working very well on the "training data", but badly on new samples outside that set. Using the term for LLM output reproducing verbatim fragments of the training data is just mischaracterizing the term to fool those who do not understand statistical algorithms (and neural networks in particular). As for Shannon, you seem to ignore what lossy compression is: https://en.wikipedia.org/wiki/Lossy_compression Having said that, I'm afraid that no amount of computer science or examples can change the mind of an #AI believer/addict.