@kodekloud: Design Centralized Logging System One bad microservice should never take down your entire logging stack. But it does — for 99% of teams that skip Kafka in their pipeline. Here's the architecture that actually survives a log flood from 100+ services: 🟢 Fluent Bit reads logs from every Kubernetes node (low RAM, language-agnostic) 🟢 Kafka acts as the shock absorber for sudden error spikes 🟢 Logstash or Vector enriches each line with pod metadata + trace IDs 🟢 Elasticsearch = hot tier, 7-day searchable history 🟢 S3 = cold tier, 1-year retention, dirt cheap 🟢 Kibana for search, ElastAlert for paging on-call The Kafka buffer is the one decision that turns a fragile pipeline into a bulletproof one. Save this 📌 — you'll need it for your next system design round. Follow for more distributed systems breakdowns 👉 . . . #SystemDesign #Microservices #Kubernetes #DevOps #SRE #Elasticsearch #Kafka #FluentBit #BackendDeveloper #SoftwareEngineer #Observability #DistributedSystems #CloudEngineering #CodingLife #TechReels #ProgrammerLife
just out of curiosity, goal is to limit services but adding so many extra layers that needs maintenance can by itself become counterproductive. similar to another video you had saying not to create too many services and getting into a complex architecture
2026-05-12 15:46:07
31
Klemens :
don't have to send to Kafka first. logstash buffers the data and will throttle filebeat with backoff if it can't keep on pushing data to elastic search
2026-05-13 08:47:21
0
Gnome WarriorKing🐍🐿️🇨🇦🇸🇰 :
I know 12factor app says stdout only, but I prefer json structured log from app (directly?) to fluentd + unstructured stdout for human inspection.
2026-05-12 20:41:45
1
marekanonimowy :
Or just have JES and RMDS
2026-05-12 22:48:51
0
mohsen :
just buy grafana cloud man
2026-05-13 03:20:27
0
sothogeek :
Over engineering
2026-05-17 06:25:43
2
Tressa 🍁 :
Log analytics off AKS done
2026-05-16 01:44:10
0
vanorange :
OK bro, you can also just use log4net
2026-05-14 08:19:33
0
me :
can you guide me about central logging of postgresql write logging while setup as active standby ?
2026-05-13 00:26:27
0
juanitomint :
500+ services same but to main differences 1 all is based on opentelemetry 2 enrichment at source where the metadata is no need to process twice for enrichment (we process twice for span to metrics tho)
2026-05-13 03:11:22
0
Un :
You won't need this kind of architecture unless you want to train on that data. ElasticSearch is resource HEAVY and needs dedicated powerful VMs.
2026-05-16 19:42:40
0
YoungDev :
Container Orchestration 👍
2026-06-08 01:17:58
0
Raghu Talluri :
Could you please tell me the name of the person explaining the concept?
2026-05-13 13:22:06
0
Efemena$✞ :
Yh very good I use this I’m my architecture
2026-05-22 19:57:39
0
Mrmanny_999 :
Nice
2026-05-13 10:11:13
0
Klemens :
instead of command line elastalert use praeco, same tool but with web gui
2026-05-13 08:48:37
0
Hope :
this sounds like @JackRhysider ngl lmao
2026-05-13 18:11:44
0
To see more videos from user @kodekloud, please go to the Tikwm
homepage.