@individualkex: 2-3x inference speedup | specdec explained #machinelearning

Noble :

You’re goated man

Jaden Stock :

I’ve seen the math for this but it still blows my mind. It actually doesn’t even matter how good the small model is, the sampling still works. The better the small model is, the further out you can go before resampling. (correct me if wrong)

reckless_dane :

swopping costly output tokens for cheaper input tokens -nice. unless your large model rejects every output of your smaller model 😂 but I’m just negatively speculating

Girth Brooks :

This may help me out quite a bit. Im building a network of mathematical kernels and each is controlled by master kernel which in turn is controlled by a Qwen model so that the local AI can do as many tasks as possible and not needing Anthropic or Open AI as much, but at the same time the system could be using this technique although it sort of already does.

Jeremy Fisher :

I wish you had explained what everything meant better because idk what’s going on

Silly :

That’s pretty cool dann

Peter :

Spec decoding is goated and people should use it more often

AV :

link please

Mxrk :

awesome!

SonOfMan :

Wes :

I hope when we get more concrete agents, one of them communicates just like you. 🥰 Besides your amazing research, your style of communication is exactly the same as my inner monolog if I could bring it to my speech

Farhan Beats :

🤩👏👏

Avg redditor :

This made me feel stupidly