@aibutsimplee: Compression and prediction are roughly the same operation. To compress data you must predict what comes next, assigning shorter representations to likely outcomes and longer ones to surprises. Shannon entropy sets a threshold on how far this can go, measuring the irreducible randomness in a source. Cross-entropy, the loss function used to train language models, measures how well a model's predictions match reality in exactly those same bit-cost terms, so minimizing it is literally learning to compress. This puts AI training in the field of information theory, with deep roots in statistical learning and algorithmic complexity theory. The profound implication, led by researchers like Hutter and Schmidhuber, is that a model which compresses well must have internalized the underlying structure of its domain: grammar, facts, causal patterns. Compression depth stands in for understanding, which is why some researchers argue that intelligence itself is best defined as the ability to find compact models of experience. Want to learn more about AI? Join 9000+ Others in our Visually Explained AI Newsletter. Get your weekly breakdown (link in bio 🔗). C: 3blue1brown Join our AI community for more posts like this @aibutsimple 🤖 #math #deeplearning #machinelearning #computerscience #datascience

AI But Simple

Open In TikTok:

Region: US

Monday 22 June 2026 15:01:59 GMT