There is one overarching principle that governs the need to refactor and optimize, both in waterfall and Agile: YAGNI (You Ain't Gonna Need It). A second principle is the corollary of the first: "Premature optimization is the root of all evil", the coding equivalent of the general proverb "the enemy of excellence is perfection".
Let's take the priciples and apply them. You have the requirement to build an ETL algorithm that takes a file of a particular type, extracts its information, then puts that information into a database. Your goal for this week (for our purposes it doesn't matter whether you're in an Agile or SDLC shop) is to get that done.
You're a smart fellow, and you have been given a glimpse of the big picture. You know that this is not the only type of file for which the project will need an ETL. So, you consider implementing this ETL algorithm to also work on another type of file, which has only minor differences. Doing this would violate YAGNI. Your job is not to develop the algorithm for that other file; it is to develop the algorithm for the one file that is needed by the end of the week. To meet that goal and pass the acceptance tests, you need to develop that algorithm and make it work correctly. You "ain't gonna need" the additional code to make it work with the other file. You may think it will save you time to incorporate it now, and you might be right, but you might also be terribly wrong; the algorithm for the other file might need to be used in an area of the system your code can't be used, or the requirements for the new file might be different than for yours in ways you don't know (in Agile, those requirements may not exist yet). In the meantime, you've wasted time and unnecessarily increased the complexity of your algorithm.
Now, it's next week, and as a dubious reward for your excellent work on the first algorithm, you have been given the task of creating the algorithms for two new file types. Now, you DO need additional code to make your algorithm work with more files. You may extend your existing algorithm using a template method pattern that will use a basic pattern with file-specific individual steps, or you may simply derive a common interface from your existing algorithm, develop two new ones that follow the interface, and plug them into an object that can choose which algorithm to use.
While developing, you know you have a requirement that the system be able to process 10KB of raw data per second. You do a load test and find your initial draft algorithm handles 8KB/s. Well, that's not going to pass the AATs. You take a look and see that there's some O(my God)-complexity loop structure in your algorithm; you streamline it and get 12KB/s. "Pretty good", you think, "but if I had that poor a loop in the code, what else can I shave off?". buzz You just violated the "premature optimization" rule. Your code works, and passes all requirements. You're "done", until such time as the requirements are updated to require 15KB/s. If and when that happens, THEN you pull the code back up and look for things to improve.
Follow this simple process while developing, whether in Agile or in traditional SDLCs: "On the first pass, make it work. On the second pass, make it pretty. On the third pass, make it SOLID." What this means is, when you first create a line of code, make that code do its job correctly and bug-free, but don't pay too much attention to design rules within this code, as for all you know right now you'll never touch this area again. The next time you visit that line of code, you just proved yourself wrong; it's no longer a one-off piece of the system. Refactor it for readability, conciseness of code, and/or DRY principles (you may have copy-pasted some code to do something five times; refactor that into a loop and/or a method call). The third time you are working in or around that line of code, it should be considered a key area of the system, and you should refactor it for SOLID structure, perform easy-to-spot optimization enhancements, etc. Follow this, and you'll find that the areas that need refactoring receive it, but at the same time no time is wasted.
It sounds like you are describing what Steve McConnell called Waterfall with Subprojects. In this methodology, you waterfall through conceptualization, requirements engineering, and architectural design. Then, for every major component, you then proceed through a detailed design, coding, and testing phase. At the end, you integrate the components in a system testing phase.
Typically, this is done by multiple teams at the same time, each working on a separate component. However, because you were working alone, it probably resembled a more iterative approach. The key difference between Waterfall with Subprojects and a true iterative approach is when you do the integration. In Waterfall with Subprojects, it comes at the completion of all subprojects. With a truly iterative approach, it happens continuously and you are fully integrated at the end of every iteration.
Best Answer
Spiral is a cycling waterfall. Thats the definition of spiral. That's what Boehm and other proposed when they invented it.
Mostly true.
Yes.
The feedback loop is the fundamental difference.
Waterfall demands Big Requirements Up Front (BRUF). It demands Big Design Up Front (BDUF) before any real coding can begin.
Spiral and Agile methods relaxe this demand.
Don't make it sound so minor. It's not a little tweak. It's a fundamental change in the volume of requirements (and design) and how those requirements (or design) are used.
In waterfall, you can't really start without all the requirements. In many cases, this is an intellectual impossibility. It's hard to visualize all the ramifications of a new way of doing business and new software to empower that new way of doing business.
In Spiral or Agile, you don't have all the information. You have enough to get started.
Many folks want "Spiral" to be "Waterfall". They want a defined schedule based on a complete understanding. In order to stop that foolishness, many folks try not to use the "Spiral" word because it doesn't encourage getting started and delivering software right now with incomplete requirements.