There appears to be this notion among the project management team that stating that "it works" means it should then be deemed 100% complete. Most programmers know that isn't always the case. If I'm trying alternative approaches to get a piece of functionality working, that doesn't necessarily mean I found the best solution, or it won't require some rework after reviewing with other developers. I'll often get done with something, step back, and then ask myself what I can do better after the business rules are satisfied. Should this "I can do better" time actually fit somewhere within the timeline? I'm of the opinion that the best approach is that you always leave code better than when you found it (to a degree), which could mean post launch refactoring. However, project teams are often extremely uncomfortable with this approach because again, if it works, and has been tested, then why fix it?
Agile vs Waterfall – Where to Fit Code Refactoring and Optimization
agileoptimizationrefactoringwaterfall
Related Solutions
Spiral is a cycling waterfall. Thats the definition of spiral. That's what Boehm and other proposed when they invented it.
Even in each Spiral or Agile cycle, there are still step-by-step tasks/objectives that need to be finished...
Mostly true.
Is the main difference the introduction of a feedback loop into the process of development earlier on and throughout the process.
Yes.
is there a fundamental difference in the actual step-by-step completion of a task?
The feedback loop is the fundamental difference.
Waterfall demands Big Requirements Up Front (BRUF). It demands Big Design Up Front (BDUF) before any real coding can begin.
Spiral and Agile methods relaxe this demand.
like it's really just compressing and iterating
Don't make it sound so minor. It's not a little tweak. It's a fundamental change in the volume of requirements (and design) and how those requirements (or design) are used.
In waterfall, you can't really start without all the requirements. In many cases, this is an intellectual impossibility. It's hard to visualize all the ramifications of a new way of doing business and new software to empower that new way of doing business.
In Spiral or Agile, you don't have all the information. You have enough to get started.
Many folks want "Spiral" to be "Waterfall". They want a defined schedule based on a complete understanding. In order to stop that foolishness, many folks try not to use the "Spiral" word because it doesn't encourage getting started and delivering software right now with incomplete requirements.
I don't think you'll really get anywhere without reworking the way things are done right now.
One example is that during development we only focus on new features and ignore existing bugs and even introduce new bugs.
I am fairly certain that this is a grave mistake. Try to explain that bug fixing becomes more difficult and more expensive the longer it is postponed. I don't think there is any excuse for postponing bug fixes for so long.
All agile methods I know emphasize that the software should be technically ready to ship at the end of each cycle/iteration (or even continuously). That means any bug that you wouldn't want to ship must be fixed before any new features is started.
Try to work with people, and explain your concerns.
Finally, if you cannot get any kind of agreement on such basics, it may be best to move on.
Edit
Based on your amended question: I agree that bug fixing should be handled just like new features. I'd normally have single tasks for "small" bugs (up to a few hours) and stories with subtasks for "big" bugs (multiple days, possible to split into tasks). If your colleagues do not agree, point out that the advantages of agile methods apply to bug fixing just like to new features.
As to wheter to use an agile tracking tool or a bug database: Ideally, the two are integrated. If they are not, then yes, you'll have to track work in both. It's annoying, but I've done it myself, and didn't find it a big problem. You track bugs as usual, and just copy&paste the bug id into the title of a task/story when you schedule a bug to be worked on. The task is just a link to the bugtracker (eays if both are web-based), and all discussion is in the bug db. If you link consistenly, you could probably even run reports across both systems.
Finally, it you believe you cannot change the way development is organized, you'll have to work within these constraints. Things like introducing some agile methods into the bug fixing phase will probably help.
Still, I am convinced that splitting development into a "feature phase" and a "bugfixing phase" is a grave, possibly fatal mistake. It may not be so serious for your company (every situation is different), but if it is, and if your organization is unable to change this, then your organization may be so dysfunctional that you'd be better off elsewhere. But that is your call to make... good luck!
Edit 2
Finally, if company policies dictate "features first, bug fixing later", maybe you can sidestep them, e.g. by redefining bug fixes as new features (the difference is often a matter of opinion anyway). To some extent, you can try to introduce new ideas "from below" that way. You know, "if you can't beat them, join them".
Best Answer
There is one overarching principle that governs the need to refactor and optimize, both in waterfall and Agile: YAGNI (You Ain't Gonna Need It). A second principle is the corollary of the first: "Premature optimization is the root of all evil", the coding equivalent of the general proverb "the enemy of excellence is perfection".
Let's take the priciples and apply them. You have the requirement to build an ETL algorithm that takes a file of a particular type, extracts its information, then puts that information into a database. Your goal for this week (for our purposes it doesn't matter whether you're in an Agile or SDLC shop) is to get that done.
You're a smart fellow, and you have been given a glimpse of the big picture. You know that this is not the only type of file for which the project will need an ETL. So, you consider implementing this ETL algorithm to also work on another type of file, which has only minor differences. Doing this would violate YAGNI. Your job is not to develop the algorithm for that other file; it is to develop the algorithm for the one file that is needed by the end of the week. To meet that goal and pass the acceptance tests, you need to develop that algorithm and make it work correctly. You "ain't gonna need" the additional code to make it work with the other file. You may think it will save you time to incorporate it now, and you might be right, but you might also be terribly wrong; the algorithm for the other file might need to be used in an area of the system your code can't be used, or the requirements for the new file might be different than for yours in ways you don't know (in Agile, those requirements may not exist yet). In the meantime, you've wasted time and unnecessarily increased the complexity of your algorithm.
Now, it's next week, and as a dubious reward for your excellent work on the first algorithm, you have been given the task of creating the algorithms for two new file types. Now, you DO need additional code to make your algorithm work with more files. You may extend your existing algorithm using a template method pattern that will use a basic pattern with file-specific individual steps, or you may simply derive a common interface from your existing algorithm, develop two new ones that follow the interface, and plug them into an object that can choose which algorithm to use.
While developing, you know you have a requirement that the system be able to process 10KB of raw data per second. You do a load test and find your initial draft algorithm handles 8KB/s. Well, that's not going to pass the AATs. You take a look and see that there's some O(my God)-complexity loop structure in your algorithm; you streamline it and get 12KB/s. "Pretty good", you think, "but if I had that poor a loop in the code, what else can I shave off?". buzz You just violated the "premature optimization" rule. Your code works, and passes all requirements. You're "done", until such time as the requirements are updated to require 15KB/s. If and when that happens, THEN you pull the code back up and look for things to improve.
Follow this simple process while developing, whether in Agile or in traditional SDLCs: "On the first pass, make it work. On the second pass, make it pretty. On the third pass, make it SOLID." What this means is, when you first create a line of code, make that code do its job correctly and bug-free, but don't pay too much attention to design rules within this code, as for all you know right now you'll never touch this area again. The next time you visit that line of code, you just proved yourself wrong; it's no longer a one-off piece of the system. Refactor it for readability, conciseness of code, and/or DRY principles (you may have copy-pasted some code to do something five times; refactor that into a loop and/or a method call). The third time you are working in or around that line of code, it should be considered a key area of the system, and you should refactor it for SOLID structure, perform easy-to-spot optimization enhancements, etc. Follow this, and you'll find that the areas that need refactoring receive it, but at the same time no time is wasted.