Long Short-Term Memory
Every system encodes the state of the world at the time of its design.
This is just an overcooked way of saying that all designs have starting assumptions, and these assumptions have a way of working themselves into the fabric of the design in a way that is sometimes impossible to change and sometimes merely very hard.
As the 1990s came to a close, the working assumption was that a server was a critical piece of Big Iron™. High spec, high reliability, high price. It was against this backdrop that Sun Microsystems built the superlative business of its day, closing deals for millions of dollars worth of E10Ks over a round of golf. Google realized that there was no iron big enough to run their service, realized that if you can run on two machines you can run on ten, realized that if you can run on ten you can run on nine for a while, realized that if the cluster is reliable, the individual servers don't matter. They bought a shitload of cheap Linux PCs and invented the modern cloud.
Sun was trapped in the old world. Their technology, their leaders and their skillsets, their culture, everything was built around this assumption that servers are big, expensive, precious objects. They couldn't adapt, and they died. Everyone knows about these cases where the world shifts, the incumbent sees it coming a mile away, and it doesn't matter.
Yet catastrophic failures like Sun are the exception. Usually the earth moves at its typical geological pace, and the house on top just shifts and strains. The floors are wonky, the walls are forming cracks, the doors are harder to open and close,1Can you tell that I live in San Francisco? but the house keeps standing. That daylight in the doorjamb is the telltale sign of a big rigid system struggling but not quite failing to adapt to new conditions.
Most things that work poorly are not the result of malice or incompetence. They're just life with a system that is out of joint, straining to break free from its original assumptions. This overhang between assumptions and reality is the opportunity to do better. The art is in knowing whether that crack in the door is one you can walk through.
Sometimes the big old building can do a little retrofitting and get a whole new lease on life. We ran out of IPv4 addresses fifteen years ago, but NAT works pretty well and now we have an order of magnitude more connected devices than we should be able to address with 32 bits. The migration to IPv6 will probably never complete, and we probably wouldn't even start if we knew what we know today. It was based on an assumption that never came true. It turns out IPv4 can scale if you know how to use it.
A workable incremental solution doesn't mean that the incumbent is safe. Even the most dramatic cases of disruption start to look pretty iterative when you zoom in. Google didn't have to invent those cheap Linux PCs, they just had to have the ingenuity to realize they could be used differently. Android used Linux too.2And iOS on Mach. I've tried to emphasize the gray areas in this essay, but I can offer you one piece of rock-solid, 100% guaranteed, unambiguously correct advice: there is never, ever an opportunity to replace Unix. Even the latest revolutionary, ChatGPT, was quite famously built on prior art.
So the door is ajar. Now what? How can you know if you can make it all the way through? You fundamentally cannot. Once you know for sure, the door is all the way open. By then, you're going to be fighting your way in: other aspiring entrants will have noticed. The incumbent will have too. Everyone has read The Innovator's Dilemma. They know they have to adapt or die. Without the fear of death, they won't have the activation energy to do it.