Elektrine lite

← Feed

@samvines@awful.systems

2026-11-11 09:03 UTC

The scaling laws maxed out around mid 2023 with the release of GPT-4 - that was around the time that everyone got excited about “test time inference” which is where LLMs spit out reasoning streams to convince you they are “thinking”. If you look at where LLMs have improved since then it’s mainly in tasks where they can be strapped into a harness and given concrete feedback like “your program didn’t compile” which are also coincidentally use cases where data from gpt and Claude users can easily be harvested and new training data can be squeezed out of underpaid third world mturkers. The big labs have just been doing an incredible job of marketing (and particularly in Anthropic’s case, brain damaging themselves into believing their own marketing)

Replies (1)

  • @BioMan@awful.systems 2026-11-11 22:03

    These scaling laws are even funnier than that, as are more recent ones. I continue to be extremely amused by the fact that you can take a LLM ‘reasoning’ trace, replace all the ‘reasoning’ with dashes or spaces, re-run the attention mechanism, and have what comes out the other end improve by a significant fraction as much as if you let it exude text that has the appearance of having to do with the topic at hand. It’s just more opportunity for the embedding vectors to dance against each other before you translate the abstract vectors back into natural language. Hence recent frontier models doing a thing where instead of doing a long reasoning trace, they build the attention matrix, allow the vectors to push each other around… then let those new vectors push each other around again, and again, and again, multiple rounds, before you start decoding. More recently, I have seen some vague math into the scaling of the speed with which running many parallel systems with access to a common pool of text produce a legible result. If I recall, it comes to roughly: N parallel systems Finish in ~1/N^0.5 as much time At ~N^0.5x the cost So 16 parallel systems crosstalking finish complex tasks in 1/4 the time at 4x the cost (and 16x the instantaneous rate of operation per second)

    Open ##5520763