Arjay McCandless
Region: US
Monday 11 May 2026 20:52:37 GMT
124088
8979
116
279
Music
Download
Comments
schatje_dondan :
if companies cared about robots.txt ai WOULDN'T progress at such a fast rate
2026-05-11 21:51:04
15
user9762258332577858437 :
checking if robot.txt allows 😂😂😂😂
2026-05-11 21:29:08
706
ChildishSaltine :
doesn't the intern make more sense than the Jr dev? Jr's approach assumes you have auth to get anything from those api calls.
2026-05-11 21:11:03
145
Sakidō :
absolutely no one with scraping use case will ever want to check robots.txt 😂
2026-06-29 06:21:52
1
zerodoisMetal :
maintaining webscrapers is the worst job
2026-05-12 03:01:23
31
Jelloeater :
Principal / Lead Engineer... just use SmolAgents
2026-05-12 21:11:32
6
Andre B Silva645 :
i have made thousands of Web scrapers in my career, the junior approach it way better,low maintenance, high performance, low compute usage, there are cons like it takes way more time developing when you need to login,reuse sessions between bots,etc ...
2026-05-12 00:22:33
14
yolomancer :
nah. Camoufox + Browser-Use anf Gpt4o. if you're gonna do serious scrapingz throw in a residential proxy as well
2026-05-12 08:53:06
3
Evan J :
api doesnt mean you need auth all the time.
2026-06-28 14:31:50
1
Jairo Peralta Vega :
the senior response left out the stale date issue. why not create a cron job
2026-06-26 01:58:48
1
Ya mom :
Use reins on chrome web store. Made an MCP. Been scrapping like crazy
2026-05-15 04:28:13
3
princeomar :
I thought you would talk about ai..
2026-05-11 22:59:23
0
Maggick :
What about JS generated content?
2026-05-12 04:11:24
0
Eric Orr :
I don't read robots.txt cause that's only for robot eyes
2026-05-11 21:36:58
3
Fav :
robots.txt😂
2026-05-12 15:13:31
3
mandnru :
I’d add mention in the senior dev part about squid proxies and bloom filters.
2026-05-12 13:02:41
3
Robert :
etag based cache invalidation is cool, too, for final system
2026-05-11 21:15:51
3
Димка :
you forgot proxy pool for senior
2026-05-13 11:30:55
2
Hugo :
No, yes, no
2026-05-12 00:10:46
2
sam.x.11 :
so no firecrawl needed?
2026-05-11 21:01:40
0
Thakur Kharel :
Or use zyte and let them take care of it 😂
2026-05-13 04:03:20
2
JeyGDiff :
I recently had used Claude, Sagemaker and noVNC combined with logging the har file while making requests. Not only could Claude screenshot what the browser currently could see but I could also see interactively where the scraping process was (before kicking it off headlessly). This made debugging pretty clean and building the scraper much quicker.
2026-05-11 22:18:09
4
Patrick Henry :
Snr should have also add pooling & rotation of ip
2026-05-12 00:42:40
1
SMYLE :
Cool
2026-05-11 20:58:32
0
Saem's Tunes :
By that logic, I must be a SWE 🤧. Mamaaa... 😭
2026-05-11 21:01:11
1
To see more videos from user @arjay_mccandless, please go to the Tikwm
homepage.
