How to Implement a Software Development Process in an Existing Project

I think two techniques are key:

• Develop a complete simulator or test-environment for the hardware, so that you can develop the software as if you have real hardware. Don't skimp or take shortcuts here: developing a good simulator will pay off.

• Write lots of unit tests against the simulator.

Once you have these things going, and people are confident that the simulator and unit tests will give an accurate idea of how things will work with the hardware, it will be easier to adopt other agile techniques (short iterations, relentless refactoring, etc.).

Yes... If...

I'll only say yes if you can drop features from your project to hit the deadline.

Otherwise you will only be getting a portion of the benefits from an Agile Approach.

Arbritrary Deadlines don't work

Timescales are tight, and reputations have been staked on delivering on time.

This worries me, this smacks of an arbritrary deadline created from optimistic best guess analysis. Arbritrary deadlines do not work, and in fact have been proven to work in detriment to a project.

Joel Spolsky wrote a great essay on Evidence Based Scheduling

3) Simulate the future

Rather than just adding up estimates to get a single ship date, which sounds right but >gives you a profoundly wrong result, you’re going to use the Monte Carlo method to >simulate many possible futures. In a Monte Carlo simulation, you can create 100 possible >scenarios for the future. Each of these possible futures has 1% probability, so you can >make a chart of the probability that you will ship by any given date.

Be prepared to drop features to save time

Putting that aside having a fixed timeframe for an Agile Project is fine, however you need to be able to prioritize your most valuable features in your software product and then drop your not so valuable features to reach your time milestone.

This should happen every single sprint, the only features that get merged into a "release" at the end of a sprint are features that are fully tested and functioning. Features that are not fully tested and functioning get dropped for the next sprint or next "release". Release doesn't mean deliverable to the customer, but after every sprint you have a fully working (albeit in a limited fashion) software product that is fully tested.

Use the best features of Agile

The idea of using agile is that your features between sprints is not set in stone. In a normal agile project the customer has input at every sprint and may request a new feature be added as a higher priority than a feature requested 3 sprints ago.

Avoid a project race (if at all possible)

In your case this probably isn't going to happen, you already have the feature set out in your existing projects, and in addition it also sounds like you are going to have a race condition whereby you are going to be racing your current maintenance cycle for your existing projects with your new project so that you can replace it.

• http://butunclebob.com/ArticleS.UncleBob.CleanCodeArgs

If you have been a programmer for more than two or three years, you have probably been significantly slowed down by someone else’s messy code. If you have been a programmer for longer than two or three years, you have probably been slowed down by your own messy code. The degree of the slow-down can be significant. Over the span of a year or two, teams that were moving very fast at the beginning of a project can find themselves moving at a snail’s pace. Every change they make to the code breaks two or three other parts of the code. No change is trivial. Every addition or modification to the system requires that the tangles, twists, and knots be “understood” so that more tangles, twists, and knots can be added. Over time the mess becomes so big and so deep and so tall, they can not clean it up. There is no way at all.

As the mess builds, the productivity of the team continues to decrease, asymptotically approaching zero. As productivity decreases, management does the only thing they can; they add more staff to the project in hopes of increasing productivity. But that new staff is not versed in the design of the system. They don’t know the difference between a change that matches the design intent, and a change that thwarts the design intent. Furthermore, they, and everyone else on the team are under horrific pressure to increase productivity. So they all make more and more messes, driving the productivity ever further towards zero.

Eventually the team rebels; and they inform management that they cannot continue to develop in this horrific code base. They demand a redesign. Management does not want to expend the resources on a whole new redesign of the project, but they cannot deny that productivity is terrible. Eventually they bend to the demands of the developers, and authorize the grand redesign in the sky.

A new tiger team is selected. Everyone wants to be on this team because it’s a green-field project. They get to start over and create something truly beautiful. But only the best and brightest are chosen for the tiger team. Everyone else must continue to maintain the current system.

Now the two teams are in a race. The tiger team must build a new system that does everything that the old system does. Not only that, they have to keep up with the changes that are continuously being made to the old system. Management will not replace the old system until the new system can do everything that the old system does, on the day of the change.

This race can go on for a very long time. I’ve seen it take 10 years. And by the time it’s done, the original members of the tiger team are long gone, and the current members are demanding that the new system be redesigned because it’s such a mess.

If you have experienced even one small part of the story I just told, then you already know that spending time keeping your code clean is not just cost effective; it’s a matter of professional survival.

An Alternative - clean up your existing architecture

Robert Martin as quoted above expresses an alternative to a project rewrite in his talk about software craftsmanship. If you haven't seen this before its well worth watching all the way through.

• http://www.infoq.com/presentations/craftmanship-ethics

I also recommended some methods to perform gradual refactoring as opposed to a complete rewrite in my answer to a similar question.

• What is the best/ethical way to estimate code clean-up?

Getting Started

In the past where I have started large greenfield projects (unsurprisingly it was also SOA). I have found that the initial architecture prototyping and bedding down of the framework took a bit of time where I may redesign my approach, scrap my ideas half a dozen times over as I prototype and test out theories and best design practices.

Don't waste time writing specifications that will only be thrown away

Using a big up front design in a new architecture will waste time and effort so taking an agile approach of functional requirements and specifications over a fully fleshed out specifications document is essential to not waste time and throw away bad ideas that looked great on paper but didn't work out in real life.

Initial Prototyping

This initial prototyping stage I have found generally doesn't produce anything that is deliverable to the customer until its actually finished and working. However what you should end up with at the end should be a well thought out architecture, with easily configurable modules and refactoring the existing applications into the new architecture should lend itself quite easily to an agile methodology with customer visible results.

Followed by a ramp up in velocity

Once your prototype is finished and your architecture bedded down you will find that you will have a transition period where you will need to educate the rest of your team as to how best to migrate or refactor your existing projects to fit in with the new framework. So you can expect a slow ramp up speed followed by increased velocity nearing the end of the project as you get more developers online and implementing your design.