Stack vs Heap – Why Use a Heap if Stack is More Efficient?

Efficiency and CPU architecture. I remember when the Alpha chips came out, they could only read memory on 32 bit boundaries. If you wanted to read byte 3, you'd load a 32-bit number then have to rotate the result to isolate the byte you want. A traditional byte-by-byte string compare was quite inefficient in this architecture.

That being said, I'm not sure what modern processors do. It's been a while since I had to deal with this.

Any time you have an application which has a performance-intensive critical path, you should be concerned how you treat memory. Most end-user client-side applications don't fall into this category because they are primary event-driven and most events come from interactions with the user, and that doesn't have that many (if any at all) performance constraints.

However, a lot of back-end software should have some focus on how the memory is handled because a lot of that software can scale up to handle higher number of client, larger number of transactions, more data sources.... Once you start pushing the limits, you can start analyzing how your software users memory and write custom allocation schemes tailored to your software rather than rely on a completely generic memory allocator that was written to handle any imaginable use case.

To give you few examples... in my first company I worked on a Historian package, software responsible for collecting/storing/archiving of process control data (think of a factory, nuclear power plant or oil refinery with 10's of millions of sensors, we'd store that data). Any time we analyzed any performance bottleneck that prevented the Historian from processing more data, most of the time the problem was in how the memory was handled. We've gone through great lengths to make sure malloc/free were not called unless they were absolutely necessary.

In my current job, I work on surveillance video digital recorder and analysis package. At 30 fps, each channel receives a video frame every 33 milliseconds. On the hardware we sell, we can easily record a 100 channels of video. So that's another case to make sure that in the critical path (network call => capture components => recorder management software => storage components => disk) there isn't any dynamic memory allocations. We have a custom frame allocator, which contains fixed-size buckets of buffers and uses LIFO to reuse previously allocated buffers. If you need 600Kb of storage, you might end up with 1024Kb buffer, which waste space, but because it is tailored specifically for our use where each allocation is very short-lived, it works out very well because the buffer is used, free and reused for next channel without any calls to heap API.

In the type of applications I described (moving lots of data from A to B and handling large numbers of client requests) going to the heap and back is a major source of CPU performance bottlenecks. Keeping heap fragmentation to a minimum is a secondary benefit, however as far as I can tell most modern OSes already implement low-fragmentation heaps (at a minimum I know Windows does, and I would hope others do as well). Personally, in 12+ years working in these types of environments, I've seen CPU usage issues related to heap quite frequently, while never once have I seen a system that actually suffered from fragmented heap.