How to document code?
You already have a hint: look at how Java API is documented.
More generally, there is no unique set of rules which apply to every project. When I work on business-critical large-scale projects, the documentation has nothing to do with the one I would write for a small open source library, which, in turn, has nothing to do with the documentation of my medium-scale personal project.
Why many open source projects are not documented well?
Because most open source projects are made by people who contribute to those projects because it's fun. Most programmers and developers consider that writing documentation is not fun enough to be done for free.
Why many closed-source projects are not documented well?
Because it costs a huge amount of money to (1) write good documentation and to (2) maintain it.
The immediate cost (cost of writing the documentation) is clearly visible to the stakeholders: if your team asks to spend the next two months documenting the project, it's two additional months of salary to pay.
The long term cost (cost of maintaining the documentation) becomes noticeable pretty easy too to the managers, and is often the first target when they must lower the cost or shorten the delays. This causes an additional problem of outdated documentation which quickly becomes useless, and is extremely expensive to update.
The long term savings (savings from not having to waste a few days exploring the legacy code just to understand a basic thing which should have been documented years ago) are, on the other hand, difficult to measure, which confirms the feeling of some managers that writing and maintaining documentation is a waste of time.
What I often observe is that:
At the beginning, the team is willing to document a lot.
Over time, pressure of deadlines and lack of interest make it more and more difficult to maintain the documentation.
A few months later, a new person who joins the project practically can't use the documentation, because it doesn't correspond at all to the actual system.
Noticing that, management blames developers for not maintaining the documentation; developers ask to spend a few weeks updating it.
If management grants a few weeks for that, the cycle repeats.
If management refuses, based on previous experience, it only increases the bad experience, since the product lacks documentation, but a few months were spent writing and maintaining it.
Documentation should be a continuous process, just like testing. Spend a week simply coding a few thousands of LOC, and getting back to tests and documentation is very, very painful.
How to encourage the team to write documentation?
Similarly to the ways to encourage people to write clean code, to do regular refactoring, to use design patterns or to add enough unit tests.
Lead by example. If you write good documentation, your pairs might start doing it too.
Do systematic code reviews, including formal code reviews targeted at inspecting the documentation.
If some members of the team are particularly antipathetic to good documentation (or documentation at all), discuss the subject with them privately, to understand what are the impediments which prevent them from writing better documentation. If they blame the lack of time, you see the source of the problems.
Make the presence or the lack of documentation measurable for a few weeks or months, but don't focus on that. For example, you may measure the number of lines of comments per LOC, but don't make it a permanent measure, otherwise, developers will start writing long but meaningless comments just to get rid of low scores.
Use gamification. This comes together with the previous point.
Use positive/negative reinforcement.
(See the comment by SJuan76) Treat the lack of comments as errors. For example, in Visual Studio, you can check an option to generate XML documentation. If you also check that all warnings are treated as errors, the lack of a comment at the top of a class or a method will halt the compilation.
As for the three previous points, this one should be used with caution. I used it for a while with a particularly tough team of beginner programmers, and it ended up with StyleCop-compliant comments like that:
/// <summary>
/// Gets or sets the PrimaryHandling.
/// </summary>
public Workflow PrimaryHandling { get; set; }
which were, hm..., not particularly helpful.
Remember: nothing automated can help you pinpoint bad comments when programmers wants to screw with you. Only code reviews and other human tasks will help.
Are there any good examples of when minimal or no documentation is needed?
Documentation explaining the architecture and the design is not needed:
For a prototype,
For a personal project written in a few hours to accomplish a task, while being pretty sure this project won't be maintained any longer,
For any project where it is obvious, given the small size of it, coupled with the particularly clean code, that you will spend more time writing documentation than all the future maintainers exploring the code.
In-code documentation (code comments) is not needed according to some developers when the code is self-documenting. For them, the presence of comments is, except in the rare cases, not a good sign, but a sign that the code wasn't refactored enough to be clear without the need for comments.
I feel that we should have a documentation review after a project is delivered.
If your project is delivered at least once per week, it's the way to go. If your project is not agile and is delivered at the intervals of six months, then do more regular reviews.
Generated documentation's audience is developers who are trying to use the code. Meaning the code is a reusable library or framework.
If code is named and structured properly, documentation is not as important. I have found that the most important documentation is at the class or module (namespace, package, etc) level. The least important documentation is at the function or method level.
In order to make the documentation meaningful and useful to developers, be sure to describe the purpose of classes and how they are used together. If I have an e.g. PrintJob class, I know what it is. But how does it interact with a PrintManager or PrinterProperties? What is the process of taking an object and printing it? How do these objects work together? I would expect to see a short code example showing the class interactions.
Generated documentation must also include pre- and post-conditions in plain English at the function level where appropriate. If your print() function expects a job to be in a certain state, tell me. I do not want to dig around in the code to decipher what it needs. Once the printing is done, can I reuse the job to print a second copy or is it now invalid?
Finally, omit documentation for entities that truly are simple and do not need it. If you have a List, you do not need to document the size() function. I understand it will be a nonnegative integer, and based on the data type, I know how big it can be. This reduces fluff. When I am scanning documentation, I know that if I see text, it is important. It is not documenting something that is common-sense to a developer.
Best Answer
You should strive to become irreplaceable not by writing code noone else understands, but by gathering more experience and knowledge than others. The former way makes you a developer everyone tries to avoid working with, as they will fear and loath maintaining code you wrote. The latter way you become a sought out team member, whom managers want to have in their team.
In my experience writing clean, well documented (and preferably self-documenting) code has always paid off. In fact I took almost every opportunity to help and teach others (as well as learning from them when they knew something better), and I hardly ever felt in danger of getting replaced by someone less capable than me. In fact, this usually helped the whole team work better and solve problems faster, which is every manager's dream - and a sensible manager doesn't want to replace members of a good team.