Two big ones in my younger days:

Alt tabbed one too many times, clicked drop database, clicked ok, realized I’d just deleted the live user database for America’s Army. Thankfully it was the east coast site and west coast was the primary, and it was only one way replication. We shut down east coast auth and rebuilt the secondary.

Someone distracted me while typing in a vlan command on a switch, I hit enter without double checking, took out our fiber between two datacenters in the middle of a move. Took me 15 minutes to run to the DC, plug in a console cable and fix it. Took all of our customers out.

September 12, 2001. I accidentally shut down an entire production plant. Management didn’t even get mad. They closed the plant for the day, I kicked off the boot cycle (takes hours for the system to be ready for production again) and everyone went home to be with their families. Nobody’s head was on right that day anyway.

EDIT: A few years later I was testing some BigIP configs on a tertiary unit when suddenly the entire e-commerce site went down. Apparently this unit used to be a primary before being demoted and someone (not me) forgot to disable replication, so when I wiped all the rules from my “test unit” I inadvertently wiped all the rules to the production units. Technically it wasn’t my fault but it was still an “oh fuck” moment.

Shit job from 10 years ago. Getting too drunk at the office Christmas party and talking so much shit about my horrible fucking boss to a couple of influential senior managers who are close to the CFO. Freaked out all weekend about it, severe anxiety attacks. That mixed with the hangover I was just vomiting all weekend.

And then: Run into one of those senior managers early in the office on Monday morning, apologised for being inappropriate. Get a response “oh, we all know how bad she is, can’t believe you haven’t quit yet”.

Yesterday I was waiting for the Tram.

As I stood there, I turned my head to the right and witnessed how a pigeon was hit by a car.

Kinda traumatizing, especially when the cars that followed ran over and over the carcass.

I predict this will be quite tame compared to the rest of this thread but here you go anyway:

Our scene opens on a person with a bass guitar, practicing some sick slap bass perched upon an IKEA office chair. The player is lost in the zone, all mental facilities put to work making every strike with the thumb is accurate and powerful, and every pop of the fingers sends the strings into the fretboard with a gorgeous thwack! The sound is heavenly, a deep, rich, cutting tone- full of appropriate levels of CLANK and SNAP. The short riff being performed to a concert of no-one reaches its magnificent height, a slammingly heavy riff sounding like a funky machine gun, the bassists hands become a blur, the strings vibrate with precision and power, a glorious cacophony of ma-

kkrrmp.

Oh crap, I broke a string

A short, medium, or long story

So, a story?

Boss walks in: “can I see you in my office for a moment…”

Fuck corporate life and I’m sorry to everyone living in it… -former boss

My grandpa had developed the habit of falling out of his bed. The first time I was afraid that he was gonna die on the spot as I’d heard it, but it eventually became such a “regular” occurrence that I didn’t think of immediate death anymore. This particular day, he’d fallen twice. They brought him to a nearby hospital to get a check-up. I was worried sick that this time something was actually wrong, or that he might’ve broken a bone or something. Turns out he was fine! No broken bones or anything. Just one teeny tiny minor issue…

When he was brought to the hospital, he was accidentally placed in the area with people who were brought there with covid. I hadn’t been able to see him in months because of the restrictions, and even when I did go the months prior it was always with far distance, masks and in short bursts. I did everything I had been told to do to “keep him safe”, “ease up the workload in the hospitals” and all those government campaigns and all that, only for him to die because of this (seeming) serious neglect from medical professionals.

I was testing some code late at night in the test system. Rolled out the changes, log on to the admin interface and write a short news article about how one of the more hated profs at the university had died suddenly and unexpectedly.

Result looks good, roll out changes to prod, about to call it quits for the night. Think to myself: common reason people get fired, maybe delete the story from test system. Check test system, no story there… Uhoh.

Story has been live for about three hours. Hope no spiders have caught it yet, hurry to delete it and learn how to purge all evidence from database.

Turns out the shithead admin had copy and pasted the server config for the test system from live and forgotten to change the admin rewrite rules to test system. Phew…

It was my sister’s birthday and I was over her place helping set everything up for the night party and then she said “dad’s not picking up the phone”

I had the gut feeling then and there something had happened and indeed he had had a hearth attack that morning

I took down the home page of one of the top 5 websites for around 5 minutes.

There were two existing functions that were written by a different team: An encode method that took a name of something (only used internally, never shown to the user) and returned a numeric identifier for it, and a decode method that did the opposite.

Some existing code already used encode, but I had to use decode in my new code. Added the code, rolled it out to 80% of employees, and it seemed to work fine. Next day, I rolled it out to 5% public and it still seemed okay.

Once I rolled it out to everyone, it all broke.

Turns out that while the encode function used a static map built at build-time (and was thus just an O(1) lookup at runtime), decode connected to a database that was only ever designed for internal use. The DB only had ten replicas, which was nowhere near enough to handle hundreds of thousands of concurrent users.

Luckily, it’s commonplace to use feature flags changes, which is how I could roll it out just to employees initially. The devops team were able to find stack traces of the error from the prod logs, find my code, find the commit that added it, find the name of the killswitch, and disable my code, before I even noticed that there was a problem. No code rollback needed.

That was probably 7 years ago now. Thankfully I haven’t made any mistakes as large as that one again!

Always use feature flags for major changes, especially if they’re risky!

“Oh, there’s a word for what I feel? I guess I’m trans.”

I was in charge with creating, designing, configuring, and running an e-commerce site on Shopify as well as responding to customer emails and orders. This was for a start-up. First orders would get a promo code for a 15% discount on a next purchase.

CEO tells me we need to get rid of stock fast as possible, so make a storewide discount on certain models. I go and do that.

We receive a bulk order of 50 of the discounted units while also using the 15% off promo code we sent out to early purchasers. Realized I should have put in a rule that doesn’t allow discounts to stack…

CEO and other members all unanimously voted to get rid of me a couple of hours later.

“We need to talk.”

A few years ago I was responsible for provisioning accounts in an application with a shitty admin UI. The process for doing so was to create the account and then give the account access it required by clicking in the list of available access and then clicking the arrow to move it over to the assigned list. Or you could just double click on it. To remove access you did the same thing from the assigned list to the available list. Or you could go into a particular item and remove the accounts from it that way. One day I was working through terminating a department that had been laid off when I noticed that on one of the largest access items the account list only had names up to “j” in the alphabet. Apparently my mouse had developed a double click problem and I had managed to select the bottom half of that list. I did not have access to any logs or anything that would allow me to reverse that change and it was a Saturday so I was basically fucked until Monday morning when I could contact the application support and they could hopefully fix it.