The Best Place to Build a Subway

This is a great (if short) parable on SF’s infrastructure challenges, and how they relate to software construction. A couple of highlights:

Many times, engineers don’t realize that their projects (and their paychecks) exist only because of the preexisting and complex systems that impose constraints on the new work.

You can’t easily build infrastructure where it’s crowded, and you can’t afford to build infrastructure where it’s not crowded.

O’Farrell created a massive 120-foot-wide Market Street paralleling Mission Street in the southern part. This was a ridiculously large waste of land for a street, especially in a town of 600 people. The locals of the town were furious. That land was valuable and was being taken from them for no value at all.

Construction on Market Street [BART] started in 1967 and continued for almost 10 years. Unfortunately, for 10 years Market Street was a mess, and lots of businesses went bankrupt. Still, the 120 foot width of Market Street opened up the possibility that construction with cut-and-cover could be attempted. Without that generous width, we likely wouldn’t have BART today.

One especially challenging part of evolving a complex system is keeping it going while it’s being changed. Years back, any new version of software had to be sent in a box, and after installing, it would run better on the data stored on disk. By the early 1980s, I was worried about wide area network distributed transactions and how I could compatibly evolve the protocol. It was not unusual for this to take three releases of planned messaging changes, with each release being six months apart.

Starting with a clean slate may seem to be more desirable. There are fewer constraints. There’s also an increased chance that your software project will not take root and will not matter to anyone. The best hope is to build something that has an appropriate investment in infrastructure based on the economics. While doing that, try to have the insight to leave especially wide roads, perhaps 120 feet wide, even if they don’t matter much now.