Assembly – How to Convert Binary/Hex to Assembly Instructions

for the very first assembler ever written (i.e. in history), wouldn't it need to be written in machine code

Not necessarily. Of course the very first version v0.00 of the assembler must have been written in machine code, but it would not be sufficiently powerful to be called an assembler. It would not support even half the features of a "real" assembler, but it would be sufficient to write the next version of itself. Then you could re-write v0.00 in the subset of the assembly language, call it v0.01, use it to build the next feature set of your assembler v0.02, then use v0.02 to build v0.03, and so on, until you get to v1.00. As the result, only the first version will be in machine code; the first released version will be in the assembly language.

I have bootstrapped development of a template language compiler using this trick. My initial version was using printf statements, but the first version that I put to use in my company was using the very template processor that it was processing. The bootstrapping phase lasted less than four hours: as soon as my processor could produce barely useful output, I re-wrote it in its own language, compiled, and threw away the non-templated version.

Often times NOP is used to align instruction addresses. This is usually encountered for example when writing Shellcode to exploit buffer overflow or format string vulnerability.

Say you have a relative jump to 100 bytes forwards, and make some modifications to the code. The chances are that your modifications mess up the jump target's address and as such you'd have to also change the aforementioned relative jump. Here, you can add NOPs to push the target address forward. If you have multiple NOPs between the target address and the jump instruction, you can remove the NOPs to pull the target address backward.

This would not be a problem if you are working with an assembler which supports labels. You can simply do JXX someLabel(where JXX is some conditional jump) and the assembler will replace the someLabel with the address of that label. However, if you simply modify the assembled machine code(the actual opcodes) by hand(as it sometimes happens with writing shellcode), you also have to change the jump instruction manually. Either you modify it, or then move the target code address by using NOPs.

Another use-case for NOP instruction would be something called a NOP sled. In essence the idea is to create a large enough array of instructions which cause no side-effects(such as NOP or incrementing and then decrementing a register) but increase the instruction pointer. This is useful for example when one wants to jump to a certain piece of code which address isn't known. The trick is to place the said NOP sled in front of the target code and then jumping somewhere to the said sled. What happens is that the execution continues hopefully from the array which has no side-effects and it traverses forwards instruction-per-instruction until it hits the desired piece of code. This technique is commonly used in aforementioned buffer overflow exploits and especially to counter security measures such as ASLR.

Yet another particular use for the NOP instruction is when one is modifying code of some program. For example, you can replace parts of conditional jumps with NOPs and as such circumvent the condition. This is a often used method when "cracking" copy protection of software. At simplest it's just about removing the assembly code construct for if(genuineCopy) ... line of code and replacing the instructions with NOPs and.. Voilà! No checks are made and non-genuine copy works!

Note that in essence both examples of shellcode and cracking do the same; modify existing code without updating the relative addresses of operations which rely on relative addressing.