First, I realize this question may by somewhat long and vague and I apologize for this. This is probably a basic problem with a short name to anybody who's "got it", but as I find myself lacking in this regard, please bear with me in describing the problem.
I've been doing programming in this way or the other since I was about 11 years old. This means that I've been mostly teaching myself everything from the start. I received a technical education, but not strictly in Computer Science (I graduated with a degree in Photonic Engineering). We had programming courses of course, but this was mostly basic stuff for me and I didn't learn much new things. I've kept educating myself along the way for the joy of it and always knew I would pursue a career in programming, but all my projects were quite small at that time. I had no trouble keeping them in my mind and maintaining them.
Now, I find myself the lead on a team, but not in a corporate environment – I work for the university developing scientific software (in C++) for engineering applications. Suddenly the project is growing (relatively) big and I've trouble wrapping my mind around it most of the time. I'm losing a lot of time and effort on two things mostly:
- When I have to return to a section of code I've not worked on for a while, I have difficulty remembering how it worked. I spend a lot of time reviewing the header files for the relevant classes and reading the comments I placed along the way in the source files. I wish there was some form of "schematic" I could glimpse at and regain the picture more easily;
- When I introduce changes, sometimes I realize half-way that what I'm trying to do will break things somewhere else (or worse, it shows up only at runtime as a surprise). I revert and start doing it differently, only to find out I neglected the influence on some other component. I wish there was some "architecture diagram" where I could see how things get done, how what I'm trying to do will influence other components and a way for me to plan in detail before I start implementing changes.
Most of the people I work with have similar stories as my own – strong technical orientation and sometimes great skills, but with no way of organizing their work. However, their projects are usually much smaller than mine so they cope somehow. Anyway, what that means for me is that I'm on my own and I have nobody to learn the good practices from.
I took up a postgraduate course in managing IT and while I find it quite satisfying, it's mostly targeted at non-programmers, teaching about project management methodologies, budget/schedule estimations, enterprise architecture etc. – not software design and planning as such. That's OK, I'm trying to learn that stuff too. Of course, some tools (like UML) and the types of sofware development processes (cascade, iterative, agile…) were introduced, but obviously not in great detail and I've a hard time deciding what I should pick and use (and to what extent).
I've been reading many questions and answers about software design on SO – there are many about doing it using this or that particular tool or methodology, and if I was convinced that UML documentation would solve my problems – I'd pick it up and start using it. But some people swear by it, others say it's useless. I'm looking for an answer on a higher level of abstraction – are there ways to solve the two problems I'm having, and how do you personally do it? What should I learn to be able to do it, possibly without being bound to one particular tool? These come and go out of style from time to time, and I expect their applicability varies depending on the type of the project.
Thanks a lot for reading, I was unable to say what I mean more briefly (lacking in software design experience and vocabulary).
Best Answer
Imagine how the poor guy who comes after you is going to feel -- he doesn't even have the benefit of having once known how your code works. Instead of trying to decipher your code, you should be reviewing the documentation that you wrote for the module in question. That documentation should offer a reasonably accurate view of what the module does any why. It's not "the module starts by initializing three arrays using a triple for loop...", but instead: "this module retrieves the data collected by the Fabulotron's main sensor, rearranges it into standard Neopolitan (chocolate, vanilla, strawberry) format, and delivers it to the Analysis module."
In a perfect world, you'd have a design document that sets out the various modules in the system and describes their respective responsibilities, and each of your modules could just refer back to that document to explain what they do: "This module provides the Fabulotron data collection service as detailed in section 4.8 of the design document: http://fabulotron.org/design/section4-8.html." If you don't have something like that, start writing down an overview of each module as you work on it. You don't need to write a book -- a few paragraphs are often enough to get you oriented.
That might be an indication that your modules are too interconnected. The more independent you can make your modules/classes/units, the less likely it will be that you'll run into this kind of problem. Try to make the interfaces between modules as explicit as possible, and try to limit them to just what needs to be there. An interface is a contract -- if you know that some module lives up to its obligations as specified in the interface, you don't need to know anything else about it. That prevents changes in the module you're working on from affecting other modules.
Using standard parts can help in this respect. C++ provides standard parts in the form of the Standard Template Library, and using those parts where appropriate lets you work at a higher level of abstraction. If you've written your own code to manage data structures like lists, you and those that follow you will constantly have to read the source to figure out what's going on. If you instead use standard parts provided by the STL, any C++ programmer will quickly be able to tell what your code is doing without digging into your data management routines.
Another kind of standard part comes from design patterns. They're standard concepts that can be used as a shorthand to explain how the relationship between two objects works.