Somehow, some people have gotten the idea that design is somehow inimical to Agile. That’s not true. It’s always helpful to think about things before you do them. The question is not whether or not you should think in advance, it’s how far in advance should the thinking occur. Hours? Days? Months?
The problem with shorter periods is that you may get something wrong and have to redo it. Nonetheless, the most Agile shops tend to design in short (a few hours max) bursts literally just before implementing. However, they use design principles that can handle incremental changes (domain-focused designs, for example). They create the simplest design that works for the story at hand, then incrementally modify that design to accommodate the next story. Incremental design is a discipline in it’s own right. You don’t just go with whatever pops into your head first. You can easily learn how to do it, however. (Take one of my workshops! :-)).
The (considerable) plus side of this approach is that you have the minimal (and simplest) effective design. That simplicity speeds up current development and maintenance, and accelerates the speed of changes. Down-the-line payback is considerable.
The problems with doing your design too early far outweigh the problem of occasional refactoring in an incremental approach.
First, the odds of getting something wrong if you’re thinking too far in advance are very high. It’s just not possible to plan in advance for everything. You learn by releasing code to actual customers and getting feedback. Too-far-in-advance designs can’t incorporate that knowledge, so are never optimal (or at least I’ve never seen an optimal one). Put another way, you’ll have to refactor these larger designs, too, because they’re never right. Unlike an incremental design, that refactoring is likely to be much more difficult. The benefits of too-far-in-advance designs are usually illusory.
You can try to overcome the refactoring problem with a “flexible” design that can accommodate whatever you throw at it. Unfortunately, those designs are inherently large and complex, and they easily take three or four times longer to implement than a simple solution. You are effectively guessing about how the design might evolve, but knowing that your guesses are guesses, you end up planning for many eventualities that never happen, and waste time building code that’s never used.
The best case in this scenario is that unnecessary work pushes out your release date by 2-4x. The cost of that delay is way larger than the cost of the extra work, and the extra time required for development is typically longer than the time required to do rewrites in the incremental-design scenario. You haven’t really gained much.
The next problem is sunk cost. A large complex design, once in place, becomes politically impossible to change, even if it’s fundamentally wrong. Consequently, people put up with it taking two or three times longer than necessary to make any required changes because they don’t want to change the underlying architecture that causes that delay. That’s a problem that lives with you for the rest of the life of the program. Everything is harder. All changes are slower. In a worst-case scenario, that can make agility impossible. Changes become so hard that you just don’t make them.
The final problem is the sheer scope of a larger design, and the fact that the pieces typically interact with one another. If you need to make a change in a corner of the design, it’s often the case that that change ripples out to the rest of the system, again making small changes very expensive. People often solve that problem with kluges that attempt to adapt a part of the design only, but those kludges accrete over time until you have an unmaintainable mess.
So, in general, it’s not a good idea to design too early. Big, up-front designs rarely give you the benefit you imagine. Instead, learn how to design incrementally, and do that a the last responsible moment.