@davespcs: Astra is cool ngl, this might be Larp idk #localai #homelab #170hx #strixhalo #gpu

ktonashol :

They're both shit models tho 😭 you would get better results and higher tok/s for cheaper by running an army of Luna 5.6's

shiro白 :

but well, astra is the smartest, local models are impressive nowadays but they can't compare to frontier

Huzzaifa :

Ur small models don’t compare too astra but nice llm server set up

ak.attack :

"my sonnet level llm is just like the 10T param model"

sir randy :

TikTok really just drops me off anywhere. Idk what I just read, but give me 15 minutes while I do some research I guess.

Alias :

“This might be larp idk” as we’re staring at an entirely custom-made machine made for accelerated learning

Behrad ™ :

At least 2k for the nvidia card nowadays btw. That’s 10 months of Pro 20x or Max 20x

landsnipes :

Are u not bottlenecked on the gpu connection?

LilMeap :

Allat just for you to have a context window of like 8k and for it to only be able to do extremely basic tasks because it’s a 27B param model…

▫️ :

astra still mogs btw but thats alt

Bot or Not :

It’s cool but bro be so fr. 27B compared to trillions. The best you can do is set up simple decision making N8N flows that you have to fine tune for hours. Local is power though, I’ll agree with you there.

jack :

Swear to god astra would be cheaper and better than doing all this

Vernin :

I have a 2010 T7500 with a 3070 8qb running a AI model: L3-8B-Stheno-v3.3-32K * Llama 3–based * ~8.03 billion parameters . *2,048-token stress test: 64.93 tok/s * All 33/33 layers were running on the RTX 307 * Quantization: Q4_K_M it’s pretty fast

Delta :

If you need to do this much just to “vibecode” properly, would regular coding NOT be easier, more comprehensive, and honest work?

𝓙𝖉𝖝🫆 :

Wait.. glm 5.3 flash locally on 45token/s ?? That's genuienly crazy, pls tell me more👀

vistar500 :

Astra is exciting because the ai is absorbing all data

niger :

bro astra 6 isnt even the smartest model😭

Rynn🇩🇿 :

I’d love to understand all of this it seems so useful

Dumb idiot :

cyberCafe :

Local ai is orders of magnitude smaller and less capable than sota frontier models. Local stuff is good, but you will never, ever beat the big boys.

Restrected :

I call cap on this. I’ll give you 1000$ if you can set this up for me and it works.

Gavin :

Idk how I got here I feel like I’m trying to read mandarin

Achilles :

this shi is expensive, it’s easier to just use chinese api

wendiwendi111 :

In English this time

EvilGenius :

also that isn't even anything to do with ai running on the screen. looks like a build