I never used to think of myself as a software engineer. That’s because I didn't do anything that I thought related to "engineering."
I’m in awe of the progress that SpaceX has made in developing their StarShip: A completely reusable spacecraft that is designed, ultimately, to allow humans to live on other planets. The rubber meets the road in this lofty ambition in trying to build engines that are powerful enough, structures that are strong and light enough, and computer control-systems that are effective enough—not to mention infrastructure, processes, new levels of logistics, and everything else necessary to achieve a vision on this scale.
The fascinating thing is that you can watch this learning take place live, on YouTube. This is real design engineering in action: experimentation, exploration, failure, and success. Most engineers don't ever expect to get things right the first time. If you get it right the first time, you aren't learning, you are just fulfilling your expectations.
How might we apply this type of engineering to software?
My definition of engineering is "the application of an empirical, scientific approach to finding efficient solutions to practical problems." The science part is important. We need to iterate so that we can have more opportunities to learn. We need to gather feedback so that we can reflect on the results. We need to work incrementally so that we can compartmentalize the problems in front of us and explore them efficiently. We need to organize our work into a series of small, safe experiments so that we can build our knowledge up in a more productive way. We need to capture data from real world systems so that we can learn empirically what really works and what doesn't.
These are the tools of scientific reasoning. These are the tools of real "engineering.”
Software development is an exercise in learning, creativity, and discovery. We have used words like "Software Engineering" and "Technical Discipline" without really thinking about what these words mean and how they relate to software development.
We think of the software industry as one of progress. But much of that progress is illusory. Software books from the 1970s, ’80s and ’90s often discuss the same problems that we face today but in different terms. The Mythical Man Month written by Fred Brooks recommended that we should work iteratively and collaboratively in small teams—and it was published more than 50 years ago.
The major programming paradigms, including both object-oriented programming and the currently fashionable functional programming, were recognized and largely defined even earlier than that. Lisp, the original functional programming language, was invented in 1958 and is still used in the form of Clojure, Scheme, and other dialects.
It is too glib to say that nothing changes. We do make progress as an industry. But it is slower than we may think.
In most disciplines, "engineering" simply means "the stuff that works." It is practical, but based on fact and grounded in scientific rationalism. I argue that "engineering" is the difference between our modern, high-tech civilization (only a few centuries old, at most) and the hundreds of thousands of years of human experience that preceded it.
If we want to benefit from iteration, feedback, incrementalism, experiment, and empiricism in software engineering, we need to take some of the ideas of science and find practical, real world ways to better apply them.
First we need to make progress in a series of small steps. This allows us to benefit from iteration and gather that valuable feedback.
Next we need to make progress as a series of experiments. We need to find ways to design our code that are more deterministic so that we can carry out experiments to see what works and what doesn't.
Like scientists, we need to control variables. This can be helped enormously by compartmentalizing the systems that we create so that we can evaluate the pieces. This idea of modularity is at the heart of a more incremental approach to design and development.
In this respect, software has an enormous advantage over every other discipline. The target of our experiments is software-executed on computers, and this software is our product, not just a simulation of it. Computers are the perfect experimental platform. We can experiment on our code in the form of targeted tests. Automated tests can be run in their millions, in parallel, and give us results in seconds, if we work at it. This is a freedom available to no other discipline.
Finally we need to collect information and gather feedback from our systems in production so that we can learn what our users like, what works, and where we have messed up. This empirical discovery is at the heart of modern software development techniques like canary releasing and A/B testing.
How engineering improves software
One of the lessons we are starting to learn as an industry is that when we apply this kind of thinking we make better software faster, the organizations that employ us are more successful, and we have more fun because this is real, creative work.
Different people talk about this in different terms. Many people call this approach DevOps, I prefer the term "continuous delivery" because it is more descriptive. But the names don't really matter. This is "Engineering for Software."
My take on continuous delivery is consciously centered on applying scientific reasoning to solving problems in software. We create deployment pipelines that allow us to iterate fast, aiming to create a production-releasable outcome multiple times per day. This encourages us to work in much smaller steps. Each commit to the pipeline is an experiment that explores a hypothesis about our change. We design our systems to be better compartmentalized so that we can test and deploy them effectively. We automate everything that we can so that we can move fast, but even more importantly, so that we can control the variables and achieve repeatable and reliable results.
The result of this is that teams that practice this kind of discipline measurably produce higher-quality software more quickly than teams that don't.
These are the kinds of results we would expect from a genuine engineering discipline. After all, engineering is “the stuff that works.” If continuous delivery is a genuine engineering discipline for software development—and I believe it is—then it is one of the most significant discoveries in our industry over the past couple of decades. We now have "engineering for software." Now we need to get the message out and start applying it even more widely. I am more proud than ever to call myself a Software Engineer. It’s finally an accurate description of what I do.
