STILL

One day, I decided to summarise my personal engineering philosophy that I’ve built up over the decade and a half of getting paid for making software. And obviously, I wanted to use a catchy mnemonic acronym because mnemonic acronyms make everything better. So what I came up with is STILL.

Which stands for STupid Irresponsible Lazy Loser.

That’s who I am, and my entire professional path from a junior to a medior to a senior to a staff engineer was all about discovering my personal shortcomings and learning how to leverage them as my most powerful tools.

So, without further ado, let me begin my act of public self-humiliation with the fact that I’m a sore loser.

Loser

Look, some people are just lucky. They always pick the right stocks when investing. They find pouches of gold on their way to the grocery store. Their cars never break down in the middle of nowhere. They marry decent people at their first weddings.

Also, their algorithms never run into corner cases, their databases never crash, and their users always do what they’re supposed to do.

I’m painfully jealous of those people because I’m not like them. I’m a loser.

Let me tell you a story. Once, I’ve been working on a piece of UI. Nothing fancy, really, just a bunch of controls with a bunch of callbacks attached to them, usual stuff. Also, those callbacks shared a blob of in-memory data. So when multiple events happened to be handled in parallel, there could be races over the shared model, the model could get corrupted, and then things would get wacky. I knew it was a shortcut, and it wasn’t even that hard to fix.

But then I thought, fuck it. On my shitty cheap workstation, it took ten milliseconds to handle a click. On average, sometimes a tad more, but never above a hundred. If you do the math, this means that users will have to click at least ten times per second to trigger a bug, and humans just ain’t that fast. It’s gonna be okay. I kept the code as is, and we shipped it to the customer.

Guess what. It took the users less than a week to break it, and by “break it,” I mean reliably reproduce the bug. I don’t know how exactly they did it, but they did. So I had to redo the feature properly, and my boss had to explain to the customer that one of his devs was a moron.

And this just happens to me all the bloody time. Users do weird shite; third-party services do weird shite; infrastructure collapses; data formats are inconsistently inconsistent; assumptions break. Whatever you can come up with, I likely have an anecdote about that happening to me or someone I know personally.

What I learned from this, and many, many, many other incidents, is that when I find myself thinking, “oh, I hope this thing will work,” my professional duty is to cut through the bullshit and simply treat “this thing” as if I have a bug report already. Abandon all hope and instead just hammer my software into shape when I don’t rely on hope, luck, or fortune but can rigorously logically prove that it will do what is expected.

Likewise, when I realise I’m making an assumption, I look into eliminating it. When this option is unavailable, I at least add enough assert’s and logger.error()’s to ensure that breakage will not go unnoticed.

• If the algorithm has known corner cases, get rid of them.
• If things must happen in the correct order, apply the appropriate synchronisation patterns.
• If data structure can be in an inconsistent state, reduce the number of ways it can be mutated.
• If a function’s inputs can be garbage, design the type model that prevents it from happening.
• If the database is flaky, either switch to the backup, retry a fixed amount of times, or, if neither helps, just crash (and then text the database admins).
• If the third-party service is screwing with you, dump all necessary troubleshooting details, rollback all outstanding transactions, and then crash.
• If anything else can go wrong, make sure it doesn’t. Or else make sure it fails in a controlled, consistent and observable way.

Look, I know it’s hard to be a full-time paranoid. It’s not how devs are commonly trained. Typically, they’re taught to code the happy flow and then either bullshit or weep their way out of the rest. And then they come to work for the managers who just want to crank out features as fast as possible, which also doesn’t help.

Oh, by the way, let me tell you a story. Once, we were hiring a front-ender, and I was tasked with making a simplistic webservice that candidates could use for their take-home assignment. And it was doing precisely what the API spec said, except that every endpoint had a kink:

if random.random() < 0.25:
    raise InternalServerError("Because fuck you, that's why")

And let me tell you, this was brutal. As it turned out, many candidates thought the most appropriate reaction to the backend messing with them was to hide under the desk and cry until someone came to hug them.

Don’t be like those candidates. Be a mindful loser.

Stupid

Look, some people are just clever. They recite their favourite philosophers at cocktail parties. They solve equations of quantum chromodynamics for the fun of doing it. They’re always available to condescend about unsophisticated books and dumb blog posts you read, such as the one you’re reading right now.

Also, they can easily navigate ultra-complicated architectural landscapes. And, no matter the problem, it never takes them more than ten minutes to invent a perfect solution to which they’re ready to commit for years to come.

I’m totally envious because, as a nerd, I’m supposed to be just as brainy, and I’m not. I’m merely a mediocre idiot.

My stupidity comes in various shapes and forms.

Oftentimes, when I face a problem to solve, I have absolutely no idea how to do it. So I might sketch a prototype to try out how things may fit together and then trash it entirely because it doesn’t make any sense. Then start over from scratch and end up with something where, maybe, a third or a quarter looks decent, and the rest is garbage to be discarded yet again. When a problem is particularly untrivial, I might need three or five takes over the course of a few weeks until I arrive at the solution that I find decent. Sorry, but that’s what it takes for a simpleton like me.

Oftentimes, I have only a vague understanding of the problem I’m solving. And, if I try to fill the gaps with “what I think is right,” it will likely be wrong. That’s because, well, I’m a fool. So I go and ask questions until I have a solid picture in my little head. I don’t give a fuck how naïve those questions might seem. I’m a fool, I’ve nothing to lose.

Also, I often struggle to understand complicated, obscure and arcane things in general.

Let me tell you a story. Once, I worked on a feature, and I thought I did a pretty good job because I ended up with roughly a hundred lines of code that were self-contained, took me two hours to write, and could be easily scrapped and redone if such a need arises.

Then I asked for a review, and the feedback was, “No, that’s not how we do things here, this is too simple.” The reviewers insisted that the proper way was to chop the code into pieces and spread it over five classes in three different processes.

I was awestruck. I thought these people must be so good if they could instantly foresee all the performance and maintainability implications of such design and intuitively analyse all the outage scenarios. I must be an impostor here.

Spoiler alert. They weren’t, and they couldn’t. In fact, up to 70% of the issues we had in that project were either directly caused or seriously aggravated by “arbitrarily distributed monolith” architecture. Oh, and as usual in such projects, there was a massive emotional investment in this abomination, so radical reengineering was entirely out of the question.

Actually, I just realised one thing… If any of my ex-colleagues recognise this and get upset with my trash talk, I can always dismiss it by saying I meant a different project where it was even worse. I’ve been developing backends for fifteen years. Of course, I’ve seen it more than once.

Anyway, so from all this experience, I developed a bunch of heuristics to which I adhere, such as:

• It’s okay to say, “I don’t know.” It’s also okay to say, “I don’t understand.”
• Simple is good. Complicated is bad. Complex might be the necessary evil and must be dealt with appropriately. Complex for the sake of “Look, mommy, I’m so smart, I have 35 levels of inheritance in my class hierarchy,” must be physically punished.
• Asking dumb questions is not embarrassing. Shipping garbage software is. Also, team outings often get embarrassing quickly, but I digress.
• If it doesn’t fit into your head (whatever “it” is, it might be a function, module, protocol, or anything), simplify. If you hit the limit of how deeply you can simplify, then decouple and isolate.
• Not every piece of code you write has to hit production. Just like not every woman you sleep with has to meet your wife. Sometimes, to have a little fun experiment, to try out a few new things, and then call it a day is the right thing to do.
• “Smart hacks” are never smart.
• When someone around you says, “This isn’t meant to be understood by average peasants,” discuss your concerns with management and/or update your CV.
• When someone around you says, “My code is perfect, the problem must be elsewhere”, in a non-ironic way and doesn’t get punched in the face, skip the management and start answering recruiters’ calls straight away.

Look, I know it feels so good to be smart. That’s how devs are commonly trained pretty much since primary school. Be the first one in the class to solve the puzzle, get your instant gratification, get your little dopamine rush. By the time you get out of the uni and start your career, your ego is hardwired into “I must always look smart no matter what,” and it’s a tough thing to resist. It’s not impossible, and another time I’ll write about how, but yes, it’s hard.

And yet, you must never forget that unrestrained “I must always look smart no matter what” is one of the most powerful techniques to turn a humble brain fart into an epic clusterfuck that survivors will remember for years.

The only technique that can rival it is hard work.

Irresponsible + Lazy

Look, some people are really hard-working. They can log 50–60 hours per week while producing high-quality, meaningful output that manages to combine attention to fine details with a firm grip on the overall context.

I’m not envious of them as much as I’m just plainly scared. Like, are they even humans, machines, or aliens?! I mean, I can operate in this mode for three, four, maybe six months. After that, I’ll need three, four, maybe six months to lay on my couch, sip Primitivo, listen to early Genesis, and recover from severe burnout.

For common people, hard work usually comes in one of four flavours:

• Overwork
• Monkey work
• Shit work
• Firefighting

I’m lazy and irresponsible, so I stay away from all four of these as much as possible.

Also, no funny stories in this segment of the essay because none of these are particularly fun.

Regarding overwork, I already wrote an essay on that topic some time ago, and I don’t think I’ve got much to add, so just click the link.

Monkey work is what allows you to look busy without requiring much brainpower. Say, you have a function copy-pasted ten times with slight variations, and you changed one, so now you go through the other nine and update them as well. Or, you “fixed” the copy-paste problem with a dumbshit abstraction layer, and instead of ten similar functions, you have ten blobs of the BehaviourInjectorFactoryBuilder boilerplate. Or when you log into the production database at 3:30 AM every full moon to manually run seventeen magical maintenance queries. Or when you make five unit tests for every trivial line of application code. Or anything else that makes you look busy on an hourly scale but looks hilarious in the monthly or quarterly context.

• I’m too lazy and too sloppy to keep ten similar chunks of code in perfect synchronicity. That’s why I refactor (which might take three or five attempts to do right, see the previous segment of this essay).
• Also, I’m too irresponsible to wake up every full moon night. That’s why I automate.
• Also, I’m too lazy to wrestle with inadequate tooling. Or take people who say, “I always use unconfigured vi because IDEs are too easy” seriously.

Shit work is precisely what it says on the tin. It’s really not hard to produce a huge pile of features. Provided you don’t give a flying fuck about reliability, maintainability, compatibility, being a liability, or any other -bility whatsoever. In a certain sense, it’s the logical inverse of monkey work. One is being obsessive about minute details and totally ignorant about the overall context. While the other makes perfect sense in the grand scheme of the project portfolio, but the implementation quality is so poor that the end result is worthless.

• Doing the work badly and then fixing it is (in total) virtually always harder than doing it right the first time. That’s why a hard-working person might naturally lean towards the former option, as it allows them to showcase their dedication and commitment. Me, I don’t give a shit about any of this, I’m lazy, and I take the easy route.
• Cutting corners to meet the deadlines means you’re setting yourself up for a crapfest after release. I’m too lazy, too irresponsible, too dumb and too unlucky to prefer that as my modus operandi.
• Prototyping isn’t shit work. Unless you decide to ship your prototype as-is, then it retroactively becomes one.
• Ignorance of quality standards is often root-caused by cleverness and/or reliance on luck. Being a dumb loser really helps a lot here.

Firefighting is… Well, first of all, firefighting is extremely fun and heroic. All kids want to be firefighters. I say that because my nephews watch cartoons about firefighters and want fire trucks for their birthdays. I’ve never heard of cartoons about fire safety inspectors.

So, yeah, firefighting is fun. Jump into an issue, find the problem, come up with a hotfix, roll it out, save the day, feel good, go home late. The next day you get a new issue, jump into it, and the cycle of life goes on.

I mean, look, if you have a one-off freak accident caused by a unique alignment of unfortunate events, then, sure, the right thing to do is to patch it up and move on. If, however, you have similar problems popping up again and again, and you keep patching them up without looking into the root causes, that’s precisely monkey work. Even when it comes to psychological mechanics, it’s quite similar: you’re tunnel-visioned on the problem at hand, and you’re oblivious to the overall context. And that’s very bad because your heroic problem-solving might be hiding systemic problems with software or processes, such as overly brittle and complex architecture, or organisation that encourages shit work, or anything else entirely, which you don’t see because you’re too busy putting out the dumpster fires.

• Fix the problem. Then find the root cause. Then fix the root cause.

Well, that is pretty much my engineering philosophy in a nutshell. If there are some things you found insightful, that’s great. If there are some things you can relate to, fantastic, let’s hug and cry. If you had a few good laughs, that’s also not bad.

Now, if you want me to teach you how to do it, I’ve absolutely no idea. I mean, it’s easy to say, “Don’t be afraid to look foolish” or “Don’t be afraid to look like a slacker,” but you… Okay, not necessarily you, but many people have been conditioned into that for decades, and it’s virtually impossible to undo it all with a handful of quirky essays.

Virtually impossible doesn’t mean I’m not gonna try, and I have a few quirky essays in the pipeline, so come again later. And for now, have a good day.