Design – Is premature optimization really the root of all evil

When you're basing it off of experience? Not evil. "Every time we've done X, we've suffered a brutal performance hit. Let's plan on either optimizing or avoiding X entirely this time."

When it's relatively painless? Not evil. "Implementing this as either Foo or Bar will take just as much work, but in theory, Bar should be a lot more efficient. Let's Bar it."

When you're avoiding crappy algorithms that will scale terribly? Not evil. "Our tech lead says our proposed path selection algorithm runs in factorial time; I'm not sure what that means, but she suggests we commit seppuku for even considering it. Let's consider something else."

The evil comes from spending a whole lot of time and energy solving problems that you don't know actually exist. When the problems definitely exist, or when the phantom psudo-problems may be solved cheaply, the evil goes away.

Steve314 and Matthieu M. raise points in the comments that ought be considered. Basically, some varieties of "painless" optimizations simply aren't worth it either because the trivial performance upgrade they offer isn't worth the code obfuscation, they're duplicating enhancements the compiler is already performing, or both. See the comments for some nice examples of too-clever-by-half non-improvements.

The bigger increase in performance definitely comes from hardware.

In terms of software, one of the biggest changes in the past 30 years is that we don't write nearly as much low level code as we used to. For example, software now relies on automatic compiler optimizations as opposed to hand written assembly, and makes extensive use of existing frameworks and patterns which have matured in the past few decades. On the other hand, software has become increasingly complex, and there have been corresponding performance hits.

However, hardware capabilities have improved mostly in accordance to Moore’s Law, and CPU speeds and memory bandwidth have increased hundreds of times over the past 30 years. Manufacturing processes have improved, allowing components to become smaller and faster because more transistors can be packed together. One of the biggest things which has sped up computers is memory access and usage of caching. CPU cache sizes are now bigger than total RAM used to be, and low level programs have shifted to make better use of this. Also, when 64 bit CPUs became commonplace, a corresponding instruction set (i.e. x86-64, the use of which might still qualify as “software”) was required to take proper advantage of this. In that way, it is a combination of improvements in hardware, that are taken better advantage of by shifts in software design.

In short, the biggest incremental strides in performance come from hardware – however changes to software are often required to make optimal use of new hardware. Either one doesn’t really work without the other!