The Problem With Design
Somehow, some people have gotten the idea that design is somehow inimical to Agile. That’s not true. It’s always helpful to think about things before you do them. The question is not whether or not you should think in advance, it’s how far in advance should the thinking occur. Hours? Days? Months?
The problem with shorter periods is that you may get something wrong and have to redo it. Nonetheless, the most Agile shops tend to design in short (a few hours max) bursts literally just before implementing. However, they use design principles that can handle incremental changes (domain-focused designs, for example). They create the simplest design that works for the story at hand, then incrementally modify that design to accommodate the next story. Incremental design is a discipline in it’s own right. You don’t just go with whatever pops into your head first. You can easily learn how to do it, however. (Take one of my workshops! :-)).
The (considerable) plus side of this approach is that you have the minimal (and simplest) effective design. That simplicity speeds up current development and maintenance, and accelerates the speed of changes. Down-the-line payback is considerable.
The problems with doing your design too early far outweigh the problem of occasional refactoring in an incremental approach.
First, the odds of getting something wrong if you’re thinking too far in advance are very high. It’s just not possible to plan in advance for everything. You learn by releasing code to actual customers and getting feedback. Too-far-in-advance designs can’t incorporate that knowledge, so are never optimal (or at least I’ve never seen an optimal one). Put another way, you’ll have to refactor these larger designs, too, because they’re never right. Unlike an incremental design, that refactoring is likely to be much more difficult. The benefits of too-far-in-advance designs are usually illusory.
You can try to overcome the refactoring problem with a “flexible” design that can accommodate whatever you throw at it. Unfortunately, those designs are inherently large and complex, and they easily take three or four times longer to implement than a simple solution. You are effectively guessing about how the design might evolve, but knowing that your guesses are guesses, you end up planning for many eventualities that never happen, and waste time building code that’s never used.
The best case in this scenario is that unnecessary work pushes out your release date by 2-4x. The cost of that delay is way larger than the cost of the extra work, and the extra time required for development is typically longer than the time required to do rewrites in the incremental-design scenario. You haven’t really gained much.
The next problem is sunk cost. A large complex design, once in place, becomes politically impossible to change, even if it’s fundamentally wrong. Consequently, people put up with it taking two or three times longer than necessary to make any required changes because they don’t want to change the underlying architecture that causes that delay. That’s a problem that lives with you for the rest of the life of the program. Everything is harder. All changes are slower. In a worst-case scenario, that can make agility impossible. Changes become so hard that you just don’t make them.
The final problem is the sheer scope of a larger design, and the fact that the pieces typically interact with one another. If you need to make a change in a corner of the design, it’s often the case that that change ripples out to the rest of the system, again making small changes very expensive. People often solve that problem with kluges that attempt to adapt a part of the design only, but those kludges accrete over time until you have an unmaintainable mess.
So, in general, it’s not a good idea to design too early. Big, up-front designs rarely give you the benefit you imagine. Instead, learn how to design incrementally, and do that a the last responsible moment.
Wouldn’t your final generalization be more accurate if you said, “it’s not a good idea to design too much too early? I have long considered the process of good design as being analogous to the opening of a game of Go: sure, you do not know what the board is really going to look like after 20 more moves, but you know that if you plan strategically, the stones that seem weak, wandering and lost in the wilderness during the opening will spring to life later when you need them to in the game.
Similarly in software design, if you make the right decisions early on, it becomes much easier to accommodate changes even wider than ‘refactoring’ changes when the need arises later on. But also similarly, some decisions should not be made during the opening, they must be put off until the middle game.
But please note that this “right decision early on” cannot be and must not be an example of the ‘flexibility’ you mention. The flexibility you mention that causes only trouble for the project sounds like (for example) adding extra code for more generality instead of finding more generality by getting rid of special cases. Planning for more generality by eliminating special cases is good at any time in the development cycle, planning for more generality by adding code to handle cases (whether truly special or not) is an activity that belongs later in the process — if at all.
Finally, I have to admit that although I called the above the way I have long considered design to be, I have rarely seen it work so smoothly. Usually, the political pressures you mention intervene and ruin it. Or worse, yet, the rest of the team is bound and determined to insist on an interpretation of Agile principles that, as you say at the start, deny that design is “inimical to Agile”.
Hi Matt, My only reservation with your “not too much too early” is that, in the wrong hands, that could still lead to too much :-). It’s surprising how little you need to get started. I’m assuming, though, that the people getting started are all architects at some level (and if they aren’t, the the Architect isn’t doing his/her primary job—education). People who are just flailing around will make bad decisions. So, maybe there is a first step, but it’s not designing, it’s learning how to design.
This strategy may not work for every domain/industry. Consider an automobile company or an avionics company. They have to do large up-front architecture considering features that might have to be implemented 3-4 years down the line. In general, the architecture needs to be able to accommodate growth in features and content for 5 years in automotive industries. There are hardware decisions that will be impacted by choosing to go with minimum effective design. Frequent refactoring isn’t going to work great with systems that involve hardware/software.
The investments are too huge to incrementally keep evolving design. You have to factor some future needs and estimate up front.
I am not sure the principles used in designing in web domain or phone apps can be applied to critical system design like avionics, reactors, grids etc. In my opinion, these need big up-front architecture. But the big up-front architecture can be developed by architects/designers using agile principles i suppose.
The consideration must be based on the industry and the type of work.
Instead what Organizations seem to try doing is to use some generalized version of Agile/Scrum everywhere much to the distress of the engineers/designers.
Also, I believe poorly thought out minimum effective design and this byte sized work is what is leading to the daily software updates that people are fed up of.
In my opinion, every industry/organization must decide what is AGILE in the context of their business. Unfortunately, managements and CONSULTANTS seem to be interested in 2-week Sprint SCRUM solutions and a myriad of tools (that increase developer distress ) that promise instant data/trend about the state of the projects.
Please don’t consider this as a critique of everything agile. These are my observations based on how people seem to use agile methodologies these days.
But I know people who are building avionics in a fully agile, incremental way (at GE avionics). Even aircraft are built incrementally (using things like full-size true-to-life cockpit simulators and flight tests on full-size prototypes). Joe Justice applied Scrum to automobile development years ago (look him up on YouTube, there are many videos), and Tesla proves you wrong every time they release a nightly update. Your assertion is proven false by reality.
I think the subtlety here has to do with how much of the underlying problem domain is well understood, and how much of it can be expressed in terms of definite rules. If something is well understood (e.g., there exist time-tested scientific models, a body of reproducible knowledge, etc.), it can be designed (or, rather, specified) up-front to a significant extent. But then you have to embed or manifest those theoretically well understood aspects into a real-world thing (application, car, aircraft), and perhaps also introduce a change or a novelty, and also deal with things that the real world throws at you – that’s where agility and design skills come in.
In other words, agile lets you learn about and adapt to those aspects of the domain you don’t understand well yet – and there are always aspects that are like that. What’s particular to the software industry is that, generally speaking, it’s a given that the domain is not well understood in the above sense – in part because developers have to learn about most aspects of it, in part because the knowledge about it is not necessarily available in a formalized or “digestible” form (but sort of in the heads of domain experts), and in part because things change over time. Does that make sense? [sorry if I posted twice, it didn’t seem to work the first time I tried]
I found the article to be quite enlightening. The article serves as a compelling reminder to prioritize user requirements for effective and impactful design solutions.