Hilariously, I’ve seen this happen both ways. In this case, they used ML to train a model to output Wikipedia, but I’ve also seen someone take a compression algorithm and turn it into a generative model by having it predict tokens from arbitrary inputs. I wish I still had that article on hand because it was great.
Latest 3Blue1Brown video is about this exact thing. Using language model (a predictor) as a compressor and it is possible to go the other way around (that is, using existing lossless compressor to spit out prediction)
Hilariously, I’ve seen this happen both ways. In this case, they used ML to train a model to output Wikipedia, but I’ve also seen someone take a compression algorithm and turn it into a generative model by having it predict tokens from arbitrary inputs. I wish I still had that article on hand because it was great.
Edit: found it
Latest 3Blue1Brown video is about this exact thing. Using language model (a predictor) as a compressor and it is possible to go the other way around (that is, using existing lossless compressor to spit out prediction)
Oh that’s neat.