GitHub Awesome
Region: US
Sunday 22 March 2026 03:42:03 GMT
62322
2615
47
453
Music
Download
Comments
NForestGlade :
5 tokens per second - you can guess the next letter yourself
2026-03-22 13:26:30
83
Krispy 💡 :
5 tokens per second is crawling.😂
2026-03-22 10:55:06
69
Cody :
for anyone watching at home this isn't new technology, this is standard practice since about '95-'98 and the introduction of memory swapping, page file in Windows land. you have your hardware RAM and a service that watches it for memory that is not being accessed, if there's been any pressure for RAM then the low accessed areas are moved to the swap space on your disk, in this case modern hardware uses SSD. relatively fast in human time. if you craft a AI to use both the idea would be that you use roughly 40 gigs of that hardware RAM actively while swapping the rest of the 200 gig model from your disc just as it's needed, is it slower than having 200 gigs of RAM, yes. is it relatively performative considering what you're doing, also yes, I'd say not too bad. probably wouldn't be too snappy, but it would be very intelligent considering speed you're seeing in the hardware it's running on. thanks for coming to my Ted talk
2026-03-22 05:00:49
24
Nico :
Does it use TurboQuant? There’s been some amazing optimizations coming out lately
2026-04-09 02:59:20
0
Florin Moldoveanu :
4 tokens per second, 4 bit quantized, 209 GB... yeah that's not running the model, that is slowly killing the SSD.
2026-03-22 19:04:27
15
Carlos. :
sloooowwww
2026-03-22 04:21:00
19
Wes Bosley :
llama.cpp already does this with unified mem
2026-03-22 09:25:52
0
An avocado :
5 tokens për second.
2026-03-22 10:56:29
0
sycicnok :
claude is the developer. that's the important part.
2026-03-22 07:16:22
2
Michel M. | AI CODER :
+3.6K $ machine for 5 tokens/second ??? Really ???
2026-03-24 09:21:50
0
hojit84726 :
I'm guessing having multiple fast SSDs in raid 0 would provide better tokens ?
2026-03-22 08:45:24
0
Sveinbjörn :
5 tokens a second? Is that useable?
2026-03-22 07:28:14
0
Adam :
https://github.com/danveloper/flash-moe
2026-03-22 09:34:52
0
getklarted :
That is so sick
2026-03-24 10:43:36
0
Mr_S1n1ster :
Not even worth it for five tokens it’s a joke
2026-03-23 18:09:27
1
Ad :
Excellent for sharing 👍🙏
2026-03-22 11:40:58
0
Liu :
crazy, just by claude code...
2026-03-22 06:23:00
0
Lucas Marotta :
Maybe use something like direct storage
2026-03-22 06:38:59
1
DROP TABLE user_names; :
how is this different from AirLLM?
2026-04-22 10:34:47
0
WhatIsDeadMayNeverDIE. :
copying thing's i see 😂
2026-03-25 23:33:06
0
matetod :
Another thing to say is that SSD has that Mac, a 17GB/s beast, I'm not saying it doesn't exist in the commercial sector though, it's not exactly an easy thing and it's supported by all PCs.
2026-03-22 08:21:12
0
ectron :
thats my question why store weights in memory as a storage point .. the layers aint even taking the whole memory ... we need a better way to store weights .. thr computing layers should inly be in memory
2026-03-22 07:49:16
0
user242273687114 :
😁😁😁
2026-03-28 01:22:01
0
To see more videos from user @github.awesome, please go to the Tikwm
homepage.
