Organize documentation in accordance with target audience needs.
In your case, primary audience are apparently software developers. The parts to consider here are to address different "sub-audiences" of this one:
- Hello World.
Those willing to quickly get the taste of it, just build and run sample application to see how it looks like.
Think of the audience like one addressed by MySQL "15 minutes rule":
...a user to be able to have MySQL up and running 15 minutes after he finished downloading it.
- Fundamentals.
For the guys willing to quickly learn things necessary to start producing working software.
- Advanced topics.
For developers already well familiar and practiced with fundamentals, interested to know what is there beyond. Mainstream topics that have not been covered in Fundamentals.
- Style Guide / Recommended Practices.
Subjective advice and guidance for those interested in the way how you recommend to do things.
The question does not indicate whether this could have a substantial audience in your case, so the options to consider are to include it as a part / appendix of Advanced topics or even drop it completely.
- Quirks.
Obscure topics, outside of mainstream, that might be of interest to pretty limited fraction of your audience. For example, if you have legacy line, or stuff like upgrade / migration / interoperability with legacy - put it here. If there are some side effects for some function at particular "exotic" environment, it goes in this part.
Your secondary audience are maintainers of the manual. These guys can make or break how things work for your primary audience, so you better take care of their needs and issues.
What if something in the manual is questionable / ambiguous? What if it turns out that thorough explanation of particular concepts would make that manual too hard to read? What if it turns out that particular version of the manual has mistakes?
Two things to consider for maintainers are:
- Technical/formal specification.
Whenever there is a questionable / ambiguous / hard to explain topic in the manual, the reader can be referred to particular paragraph in the specification for a strict and clear, "official" statement on that. Strict and complete (and most likely boring) description of language syntax would better go there.
Paramount considerations for Specification are technical accuracy and completeness, even if these come at the expense of readability.
- Online supplement.
Just reserve some URL and put it in the beginning of every document you release, so that the guys wondering what the hell they have just read could go there (instead of pestering manual maintainers) and find the mistake explained.
Errata > Fundamentals > Release 2.2 > Typo at page 28, second sentence starts with luck, read lock instead.
Concepts such as locking strategies, performance related details should be included (possible partially) in where you expect target audience need it.
E.g. manual maintainers would apparently be interested in a complete, accurate description of concurrency and locking in the formal specification - put it there. Readers of Fundamentals or Advanced topics might be interested in an overview / introduction / guide extracted from specification and tailored to their needs etc.
It could be helpful to arrange a comparative study of reference documentation provided for other languages like yours. Aim this study at leveraging experience gained by those who did it before and learning how to avoid mistakes they made.
The last but not the least, consider setting up documentation development in a way similar to software development. I mean things like version control, regular releases, issue tracking, quality assurance etc. You may want to make some compromises though if it turns out that your technical writer(s) are not really comfortable with stuff like that. We don't want to get crappy content "in exchange" for perfect development process, do we?
You sound like a Software Engineer.
A Project Manager is more focused on the status of the overall project and allocating resources in an effective way to ensure that milestones are being met and in proper time. They also remove roadblocks and help the project resources focus on their specific job functions.
Writing and maintaining design and technical documentation is an important part of being a software engineer and is appropriate for your role. Too much of a good thing can be bad though (See Analysis Paraylsis ).
He considers the design to be paramount, coding is "just writing the design down"
If the project manager considers the design documents to be paramount then he expects the documents as a deliverable in the process. It is not a wasted time on your part if he feels they are important enough to allocate 80% of your time to it.
it shouldn't take too long, and "all the code should be written before the hardware is ready".
This is wishful thinking and quite an antiquated Waterfall style view of how software development works. It invariably is never that clean no matter how much design and preparation that you devote. On that note however, you should have clear hardware specs and proper test environments and mock hardware simulations for your code to interact with but again, the real world is messy.
Project management is incredibly easy in a perfect world. The world is not perfect however no matter how much you wish it to be so making real project management incredibly difficult. This is why they are paid the big bucks.
(2) Doesn't understand the difference between a Central & Distributed Version control.
Why should he care? How does it affect the milestones? It doesn't.
3) Doesn't understand code, and wants to understand every bug and its proposed solution
His goal is for working software and yours should be too. He does not need to understand the code but he does need to understand the issues that are preventing working software. Once the two of you see eye to eye on this basic level then your mutual goal will help you work together more effectively.
(4) Believes verification should be done by developer, and validation by the Tester.
What is wrong with this sentiment? Testers in the quality assurance role should be concerned with validating features and requirements. Developers should make every effort to verify and test their work before it goes to testing.
I don't really like creating documents, I want to solve problems and write code.
If you would rather be a simple programmer then perhaps you should talk to your boss about this and see if there is a better role for you on the project. Somebody needs to write and maintain the technical documentation, so perhaps one of the other developers can help you with some of these tasks so you aren't spending 80% of your time writing documentation.
In my experience, creating design documents only helps to an extent, its never the solution to better or faster code.
This is mostly true... but only if all ten developers are spending 80% of their time writing technical documentation that nobody will ever read. This is an enormous management anti-pattern that I have lived in before. If you find that you are doing most of the documentation for the team then it hardly seems fair that you are denied the right to perform more coding work.
The fact of the matter is that the best technical documentation is the code itself.
I feel the boss doesn't really care about making better products, but only about being a good manager in the eyes of the management
I feel that he does care because the product is his ward and if projects and features are not completed and clients aren't happy then upper management will not care for him very much. The problem is that your perspective on the necessary quality improvements don't match with his, and upper management, and the clients perception of what they feel is important.
Try to understand what is truly important for the product, because while you can increase the efficiency of a process 3 fold, if you spend an entire week doing this then it is at the cost of another important feature that the client is demanding.
We all want to look good to the eyes of our superior. There is nothing wrong with that, it is human nature to be self serving. This is a fact of life.
Best Answer
I usually use a wiki and just spam stuff up there as I go through the discovery process. Wiki because you get search and history as well as team editing functions. The exact tool is less important than having functioning search and getting everyone using it. Expect it to be very rough at first, but encourage people to make those rough notes as they go because that's all you're going to get at first.
A few things that help a lot:
I have started with a MediaWiki virtual machine image in the past because it's very quick and easy to get started, so people saying "it's too hard" don't get any initial traction.
If your language/environment supports it using JavaDoc/NDoc type tools to extract the comments as you add them is a good low-level approach. But that's less useful than the high-level documentation, and even though the tools kind of support adding higher-level stuff, creating it from nothing using only those tools is unnecessarily laborious.