How to make a large codebase easier to understand

documentationlarge-scale-project

Suppose that I am developing a relatively large project. I have already documented all my classes and functions with Doxygen, however, I had an idea to put a "programmer's notes" on each source code file.

The idea behind this is to explain in layman's terms how a specific class works (and not only why as most comments do). In other words, to give fellow programmers an other-view of how a class works.

For example:

/*
 * PROGRAMMER'S NOTES:
 *
 * As stated in the documentation, the GamepadManager class 
 * reads joystick joystick input using SDL and 'parses' SDL events to
 * Qt signals.
 *
 * Most of the code here is about goofing around the joystick mappings.
 * We want to avoid having different joystick behaviours between
 * operating systems to have a more integrated user experience, since
 * we don't want team members to have a bad surprise while
 * driving their robots with different laptops.
 *
 * Unfortunately, we cannot use SDL's GamepadAPI because the robots
 * are interested in getting the button/axes numbers, not the "A" or
 * "X" button.
 *
 * To get around this issue, we created a INI file for the most common 
 * controllers that maps each joystick button/axis to the "standard" 
 * buttons and axes used by most teams. 
 *
 * We choose to use INI files because we can safely use QSettings
 * to read its values and we don't have to worry about having to use
 * third-party tools to read other formats.
 */

Would this be a good way to make a large project easier for new programmers/contributors to understand how it works? Aside from maintaining a consistent coding style and 'standard' directory organization, are there any 'standards' or recommendations for these cases?

Best Answer

This is awesome. I wish more software developers took the time and effort to do this. It:

States in plain English what the class does (i.e. it's responsibility),
Provides useful supplementary information about the code without repeating verbatim what the code already says,
Outlines some of the design decisions and why they were made, and
Highlights some of the gotchas that might befall the next person reading your code.

Alas, many programmers fall into the camp of "if code is written properly, it shouldn't have to be documented." Not true. There are many implied relationships between code classes, methods, modules and other artifacts that are not obvious from just reading the code itself.

An experienced coder can carefully craft a design having clear, easily-understandable architecture that is obvious without documentation. But how many programs like that have you actually seen?

Related Solutions

Using Vim when coding a large-scale application

One aspect of experienced programmers who move from an IDE to a console / xterm environment, is finding a replacement for the indexing of source code objects (function names, variables). I believe the general term used for Microsoft's Visual Studio is Intellisense or something like that.

In the Unix/Linux world, such as vim, one tool used if ctags or the popular multiple language Open Source implementation, exuberant ctags. It is not vim specific, and is supported by a number of Unix, Linux, MS Windows, Mac OS text editors, including Emacs, CRiSP, vile & a number of other vi clones, nedit, gedit, JED, UltraEdit, BBEdit, and DreamWeaver (some of these are via third-party plugins).

Beyond that, good design and thoughtful decomposition, organization of larger projects makes the project manageable in that there are only 1-2 obvious potential places to look for any given bit of information (typedef or class definitions, etc.).

Also I use multiple instances of vim (often via view for read-only viewing of source files), as well as a limited use of multiple edit buffers per vim instance (primarily for moving or refactoring code between files). I find using only a few source files open at a time can help in its own small way, to keep me focused on the task on hand.

Algorithm Execution – How and Where to Run Algorithms on Large Datasets?

The nice thing about the PageRank algorithm is that it can be solved iteratively in a distributed way, within the MapReduce framework. However, the working data for Pagerank on ~5M nodes and ~50M edges should fit perfectly well in 4GB ram, never mind 48GB....

Specifically, you don't need to store all data for each web page in memory -- instead, you should digest your input database so that the working data for the PageRank solver refers to nodes by index. Even with no particular effort at optimization, each node should take no more than 32 bytes, and each edge no more than 16 bytes, for in-memory space of less than 1GB.

A demonstration example for this kind of datastructure, in C++/STL:

std::vector<float> old_rank, new_rank;  // rankings for each node
std::vector<int> end_edge;  // index after final edge for each node
std::vector<int> edge_dest;  // destination node index for each edge
std::vector<float> edge_weight;  // fractional weight for each edge

...

void pagerank_iteration(float base_value, float scale_value) {
    new_rank.fill(0.0);
    int edge = 0;  // loop variable:  current edge index
    for(int node=0; node<first_edge.size(); ++node) {  // loop over nodes
        while(edge < end_edge[node]) { // loop over edges of current node
            int dest_node = edge_dest[edge];
            new_rank[dest_node] += edge_weight[edge] * old_rank[node];
            ++edge;
        }
    }
    assert(edge == edge_dest.size());

    for(int node=0; node<new_rank.size(); ++node) {  // add scale/offset
        new_rank[node] = base_value + scale_value * new_rank[node];
    }
}

Running on a single PC is easier than running it on a cloud service, because you don't need to use a network-capable framework (although it might be a good idea to keep that possibility in mind). The relatively small sizes you are describing can be solved easily with an ad-hoc single-threaded algorithm, and you can either use Hadoop locally or roll your own MapReduce using threads or inter-process communication if you want more cores.

Best Answer

Related Solutions

Using Vim when coding a large-scale application

Algorithm Execution – How and Where to Run Algorithms on Large Datasets?

Related Topic