← Writing
The Second Law of Data
Data engineering craft

The Second Law of Data

The second law of thermodynamics is the one that ruins your day if you think about it too long. It says, roughly, that in a closed system entropy always increases. Disorder is not an accident or a failure but the default, the direction everything drifts when no work is being done against it. Your coffee cools. The tidy room becomes the messy room and never, on its own, the other way. Order is expensive. Disorder is free. The universe is running slowly downhill and the only thing that ever climbs is what something spends energy to lift.

Data systems obey this law too, and I don't mean that as a flourish. I mean it almost literally.

Leave a data warehouse alone (genuinely alone: no maintenance, no attention, just running) and it does not hold still. It decays. Schemas rot: a field that meant one thing in 2022 quietly means something adjacent by 2024 because three teams reinterpreted it and none of them updated the doc, if there ever was a doc. Definitions drift: `active_user` means one thing to marketing and another to finance, and both queries run fine, return numbers, and disagree, and nobody knows which number is in the board deck. Sources go stale. Dead tables accumulate like sediment, and nobody deletes them because nobody's sure who's still reading them, so they sit there radiating false confidence. The whole thing gets warmer, more uniform, more useless: the heat-death of your own warehouse, arriving one un-refactored decision at a time.

Here is the part that reorganizes how you see the job. None of that decay requires a mistake. Nobody has to do anything wrong. Entropy is what happens when everyone does nothing. The rot is the resting state. The warehouse tends toward disorder the way the coffee tends toward room temperature, and for the same reason: order was the thing that cost energy, and energy stopped being spent.

Which means the work of a data engineer is less like construction than it sounds, and more like refrigeration. You are not building a monument that stands once and endures. You are running a pump against a gradient that never turns off. The deprecation nobody wanted to do. The definition dragged back into agreement across two teams who'd learned to live with disagreeing. The lineage documented, the dead table finally killed, the contract enforced so the drift can't start. This is not glamorous and it does not photograph well and it is the actual job: spending energy, continuously, to hold back a decay that is patient and free and always winning a little.

I've noticed the engineers who understand this are calmer about it than the ones who don't. The ones who think a system is done when it's built spend their careers baffled and betrayed, watching their beautiful architectures degrade and taking it personally, as though the universe had singled them out. The ones who understand the second law expect the drift. They budget for it. They know that a data platform isn't a thing you finish; it's a fire you tend. And a tended fire is a very different relationship than a finished house.

Entropy comes for the warehouse. It was always coming. The craft was never in pretending otherwise. It was in showing up, every day, to spend the work that holds the heat-death off one more day.

That's the job. Somebody has to run the pump.