@lewismenelaws: What happens when one engineer deletes what 100,000 people depend on? In 2017, GitLab went down after an accidental rm -rf deleted the primary production database. 300GB gone. Backups were silently failing. Snapshots were broken. And they thought they had 5 recovery plans… until they realized they were 0/5. They ended up restoring from a staging snapshot from 6 hours earlier — and lost hours of issues, comments, and merge requests. The wildest part: they streamed the recovery live, then published a full post-mortem for the world to learn from. Test your backups. If you’ve never restored from one, you don’t have backups. You have hopes. Follow for more. #gitlab #devops #sre #incidentresponse #database

Andrés Rodríguez :

Fortunately nobody serious uses gitlab

dmme :

software engineers get paid too much

Keith David Cohen :

no-one cares about backups, redundancy, or reliability... until they do.

zack :

not the engineer's fault. it happens to everyone literally. what really failed here was the backup system.

Bukho Mabula :

Happed to me Yesterday, I tried to commit and run git push, but it returned an error about a submodule. I wanted to delete the submodule, so I ran rm -rf. All my work for the day went down the drain, and I had to start over.

WesbajaBank :

These vibe coding fails are getting more and more serious.

ɔiɿɘᎸꙅoƚɒɿƚꙄ ɔiɿƎ :

People have resigned for less than that

Patrick Lewis :

So they copy production to staging?

Nick Vella :

he could have ran TRUNCATE TABLE in the database instead

pkollitsch :

Is _that_ how we pronounce Azure? ah-shure?

Private Milo :

They were a legend

_Salvatore_ :

i once deleted WSL system file hhhhhhhh

Coding with Lewis :

Wonder what they are up to nowadays

CtrlAltDelicious :

Yeah, been through same after a ransomware attack. Luckily I found an old backup on a Google Drive I had three backups that all failed for various reasons and the image I had on Google was almost there by accident

Earl Daniel Villanueva :

coooll!

lyrics africa :

git ..

Lade_QOP👑 :

Wow

Wedi :

what happened to the engineer?

😺💤 :

pbgb :

So many places have no usable backups.

Bytemast :

I blame trim() on ssds

Michi :

worst nightmare.

brbr :

I watched the restoration live on YouTube

Rafael Bravo :

What happen when I’m taking the PlayStation……….. 😁

big_puppy :

What happened to that engineer?! 😅😅😅