If I use my machine [Ubuntu 16.04 64 bit, kernel 4.4] for a while, QEMU needs the kernel caches to be dropped, otherwise, it won't manage to allocate the RAM.
Why does it happen?
This is a sample run:
~$ free -m
total used free shared buff/cache available
Mem: 15050 5427 3690 56 5931 4803
Swap: 0 0 0
~$ sudo qemu-system-x86_64 -m 10240 # and other options
qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory
~$ echo 3 | sudo tee /proc/sys/vm/drop_caches
3
~$ free -m
total used free shared buff/cache available
Mem: 15050 1799 9446 56 3803 9414
Swap: 0 0 0
~$ sudo qemu-system-x86_64 -m 10240 # and other options
qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory
~$ echo 3 | sudo tee /proc/sys/vm/drop_caches
3
~$ free -m
total used free shared buff/cache available
Mem: 15050 1502 10819 56 2727 10784
Swap: 0 0 0
~$ sudo qemu-system-x86_64 -m 10240 # and other options
# Now QEMU starts
Best Answer
Not all cached data can be discarded immediately. For example, cached dirty pages have to be written back to disk before they can be removed from RAM. You have no swap, so until those writes complete, there simply isn't enough available space for QEMU.
You really should add a reasonable amount of swap. You can't expect the memory manager to do a good job with one hand tied behind its back.