etymologynerd
Region: US
Saturday 20 December 2025 14:48:08 GMT
344058
34699
330
764
Music
Download
Comments
Sohiimfine🏳️🌈 :
I'd never considered that different languages would have different effects on how an AI language model operates, thank you for enlightening me
2025-12-20 14:56:12
4038
guudbaad :
Tokenization is the easiest step and only done on the very start.
2025-12-20 15:08:17
17
violentakiraneuron :
this is super out of date. latent space is the future. tokenization is on its way out
2025-12-20 17:29:15
8
g.i. robot lover :
what's with all the ai glazers in these comments bro 😭
2025-12-22 00:20:05
83
sup 🏳️⚧️ :
also sometimes they use Chinese characters instead of english words
2025-12-20 15:26:19
157
JeanClaudeRumHam :
DeepSeek has already figured out how to do this with English. They call it "Optical Compression." You do not need the correct spelling of word to read it. Optical Compression saves 90% the energy and only loses 3% the accuracy. That 3% is well worth it when drinking water and economic collapse is at stake with these data centers.
2025-12-20 18:49:14
112
user4kxti76kj3 :
how does this affect Japanese? since they both use Chinese characters
2025-12-20 15:55:37
18
Cebeide :
I just asked DeepSeek and it said that only 25% of their training data was in Chinese. 40% was in English.
2025-12-20 18:25:18
44
. :
None of the models tokenize based on strokes or stroke order?
2025-12-20 15:34:01
18
Barracuda GRM :
this guy never touched a Chinese LLM in his life 😆 it's december 2025 my G, literally ALL of them understand prompts in English
2025-12-20 19:14:25
27
tcums :
as a software dev this seems like something that they’ll figure out in time
2025-12-20 16:42:49
20
John :
Tokenization doesn’t make Chinese “harder” by default; transformers model relationships across tokens via attention. The real issue is that most tokenizers don’t explicitly represent radicals/strokes, so sub-character reasoning can be weaker unless you use sub-character/radical-aware tokenization or add those features during training.
2025-12-20 21:11:31
20
Chrizzz :
Nope, the opposite is true, Chinese Snart AI is WAAAY better in getting the "big picture" of a text and keeping this abstract as tokens instead of US Steamroller AI which tries to keep every word as tokens and then fails bc of huge context windows with lots of useless tokens leading to memory overflows.
2025-12-20 21:38:41
7
SunnyBreaux :
fun fact: Chinese researchers have discovered that the models are significantly better at image recognition vs. tokenization, and this led to discovering the transforming text into image files is actually way more efficient and accurate, and in theory this extends to all languages
2026-01-12 20:53:07
5
DJ Meat :
deepseek actually improved tokenization for all future LLMs by creating a vison/OCR based tokenizer. It's able to retain more accuracy while being able to process the same text using less tokens.
2025-12-21 03:29:48
36
›(⌤)‹ 𝕸🄰ℝ₵ ›(⌤)‹ :
Ai is still HORRIBLY inefficient and overall not worth existing for all the trade offs.
2025-12-20 19:24:24
5
Joseph A P Holsten :
Oh my. The tokenization isn’t the vector embedding. The tokenization of Chinese is fine, the embeddings just are way less trained than English. Sure they should reflect the radical and component symbols. But those also have phonetic and historical semantic value that isn’t necessarily in modern usage. That’s why we have to train the whole models holistically.
2025-12-20 19:12:38
15
Abbey Heffer :
Never thought I'd relate so hard to Chinese AI 😅
2025-12-20 18:12:30
11
user4815162342 :
speed yelling
2025-12-20 18:41:55
31
sir.g :
Why not first translate to some unifying language like Esperanto and tokenize from there?
2025-12-23 03:16:48
0
PDQ100 :
This seems easy to manage
2025-12-21 05:11:29
3
A :
Uh. I hate to tell you this but they are probably using English. Basically all AI is built in English.
2025-12-20 18:25:42
4
beefinoverturf :
Couldn’t help but notice the steadiness of your camera after seeing your last video
2026-01-12 16:27:37
2
kanapuro :
but i love deepseek ☹️
2025-12-31 18:45:53
2
HughJanis :
the harder part than tokenization is stemming but you don't want to open that can of worms
2025-12-21 09:13:28
2
To see more videos from user @etymologynerd, please go to the Tikwm
homepage.
