Read first: For a definition of the two terms ("profiling" and "benchmarking") and the need for distinguishing them, please read this answer to a previous question.
I have to admit that until I saw Winston Ewert's answer, I have never thought of the need to distinguish the two techniques. I simply think that "profiling" can be applied at different "scale levels" of software, and that when it is applied on the higher level, the profiling code in the lower levels ought to be turned off in order to lower the aggregate overhead.
After I reflected on the answer, it might have explained why I fell prey to micro-optimization in my previous project.
In an effort to optimize during that project, I implemented a low-overhead profiler (inserted into the source code) which is good at generating accurate profiling results at the millisecond level. I then spent all days tinkering with it, and optimized a lot of code based on the profiler's result. In the end, I was successful in reducing the computation part of the project from several seconds to less than a fraction of a second.
The next thing I learned, to my horror: when the optimized module was used in a larger project, I/O and data conversion completely dominated the module's computation time. The non-computation part is in the range of 1-2 seconds, making my optimization efforts moot.
To this date, I still haven't got a chance to do a true "benchmarking", though I am going to give it a try very soon.
Given that "Did you do profiling?" has become the cliche on both StackOverflow and Programmers.SE, is there a danger that my kind of ignorance is actually prevalent among fellow developers? Does this ignorance lead to micro-optimizations all over the places?
Best Answer
From what I've seen the "did you profile?" question always comes after "why does this run so slow?" so the "benchmarking" has been done and the result was "too slow" and now we're trying to figure out why it's running so slowly so we go and "profile" it.
Real life is usually more complicated. How fast your software is depends on the architectural decisions you make, algorithms you choose, whether or not you've correctly identified and dealt with various bottlenecks and system constraints. Getting stuck optimizing a system that isn't designed for performance is an easy trap to fall into and can suck away huge amounts of time for little reward. On the other hand, not every software has high performance as a requirement.
Profiling and optimizing before you benchmark, i.e. before you know whether or not the performance is adequate is truly falling into the premature optimization scenario.
I like this quote from Wikipedia: