Frame Pointer Explanation in Assembly and Stack

The red zone is, purely and simply, an optimization that can save instructions. It means that it's no longer necessary for the emitted code for every function to subtract from the stack pointer to make local storage like so

sub XXX, %rsp 

at the beginning of every function call, even if they are not leaf functions. Often times the code emitted from the compiler can use the temporary space in the red zone below the stack pointer without needing to save it and before calling other functions. This is a useful optimization to have available.

If you no longer have to sub from the stack pointer, the emitted code can use rsp as the base pointer, a job normally reserved for rbp, and the emitted code can use rbp as another general purpose register.

This ultimately means the prologue and epilogue of each function call can save two instructions that would save and restore rbp:

(gnu assembler)

pushq %rbp       # prologue [ two instructions not necessary ]
movq %rsp,%rbp

.... [code]

movq %rbp,%rsp   # epilogue [ two instructions not necessary ]
popq %rbp        

Note that in gcc you can pass the -mno-red-zone flag if you don't want it (but the x86-64 ABI requires it). The Linux kernel does not need to be ABI compliant and thus all kernel code is compiled with -mno-red-zone.

Furthermore, accessing memory beyond the stack pointer is not dangerous if that is the expected mode of operation. It's only dangerous and can lead to corruption when it's unplanned, and unexpected. When the emitted code does it, it knows what it is doing.

It depends on the VM, to be honest.

It is a bit easier to write a VM in an OO language that operates on data structures kept in memory rather than on some form of linearized bytecode. If you ever implement a toy programming language you might decide to do it this way, at least initially (I have done this before), in which case the VM is probably just a polymorphic "run()" method. A lot of scripting languages work this way, because the code will be generated at the same time in which it is run, so there is no need to maintain the bytecode for later.

If you compile all the way to a linearized bytecode (like how Java compiles), the way you handle operation calls will depend on the language. If you do not allow recursion (most old programming languages originally did not), you can simply inline the operation calls directly into the bytecode. This causes code bloat but is "safe" otherwise.

If you compile all the way to a linearized bytecode, but you want to allow recursion, then you'll need to emulate a call stack. Each frame of the stack will hold a reference to the location in the code segment where that operation was invoked (for operation-based frames). It will also hold copies of all variables local to that frame. Objects to which those variables refer may be held on the stack or in a separate heap segment, depending on the design of the programming language and (more likely) the code optimizer being used by the compiler.