TLDR; version of why Magento gives those instructions: you want things to be writable during installation (700 directories and 600 files), then afterwards assign ownership to the web server running user and restrictive permissions (500 directories, 400 files) that allow read only except for media/ and var/ directories.
This will work on a server that has been set up specifically with security in mind. Web servers can have a lot of variation in their setup especially shared hosts.
Your system sounds like it needs group and/or global read permissions for the web server to read your login user owned files. Check to see who owns the var/cache/ sub folders and the files they contain, you probably will find it's different.
From the question, you didn't get to the next step. They then have the After Installation settings which are even more restrictive.
Running recommendation is:
500 for directories
400 for files
for media/ and var/
700 for directories
600 for files.
The key to understanding all this is the need to know the server user
On a dedicated server, in the instructions they tell you how to find the server user by checking the apache2.conf or httpd.conf file for the User config line.
Typically, this will be something like nobody, www-data
And so with this bit of information on a dedicated server you then assign all directories and files to be owned by the server user
chown -R {web-server-user-name-here} .
On a hosted system, if you're using Apache MPM-ITK or litespeed, the web server will run with your login name as the server user.
Once the ownership is set properly then you change all the directories and files as follows:
find . -type f -exec chmod 400 {} \;
find . -type d -exec chmod 500 {} \;
find var/ -type f -exec chmod 600 {} \;
find media/ -type f -exec chmod 600 {} \;
find var/ -type d -exec chmod 700 {} \;
find media/ -type d -exec chmod 700 {} \;
chmod 700 includes
chmod 600 includes/config.php
Now comes the massive headache part. Any time you upload files you no longer have write permissions except where you have allowed them (var/, media/) so every time you want to do maintenance outside these folders, you must change everything back by:
find . -type d -exec chmod 700 {} \;
find . -type f -exec chmod 600 {} \;
And on the dedicated server, also probably change ownership to the login user name so you will have permission to write stuff.
Also, if you used Magento Connect (on dedicated server, leave ownership as the web server user), anything it installs will be given 777 permissions.
Because you have to remember to undo and redo the process every time you change something, or are running on a system where the web server needs group/global read permissions, the following permissions have probably become the defacto standard among lesser technically skilled website owners:
find . -type f -exec chmod 644 {} \;
find . -type d -exec chmod 755 {} \;
chmod o+w var app/etc
chmod 550 mage
chmod -R o+w media
While these permissions will get your site to work
find . -type f -exec chmod 644 {} \;
find . -type d -exec chmod 755 {} \;
chmod 550 mage
They are certainly not recommended for you to use such wide open permissions. Before you start changing permissions you should understand what you are changing and why. The above example gives a lot of privileges to all your files and folders.
You should experiment with your permissions and to keep them as restrictive as possible. For example, most of your files can run with 440 (You can test)
Note from Magento suggestions here http://devdocs.magento.com/guides/m1x/install/installer-privileges_after.html
find . -type f -exec chmod 400 {} \;
find . -type d -exec chmod 500 {} \;
find var/ -type f -exec chmod 600 {} \;
find media/ -type f -exec chmod 600 {} \;
find var/ -type d -exec chmod 700 {} \;
find media/ -type d -exec chmod 700 {} \;
chmod 700 includes
chmod 600 includes/config.php
You see that you regular files are not writable at all, only your media and var folders. If you are on a shared server you may have to change the second bit to something other than "0"
If the above example does not work then you can try:
find . -type f -exec chmod 440 {} \;
find . -type d -exec chmod 550 {} \;
find var/ -type f -exec chmod 640 {} \;
find media/ -type f -exec chmod 640 {} \;
find var/ -type d -exec chmod 750 {} \;
find media/ -type d -exec chmod 750 {} \;
chmod 750 includes
chmod 640 includes/config.php
It is important that you don't leave your site wide open to the world!
Best Answer
This was due to the my NGINX configuration:
I was able to fix it by removing the above block and replacing it with:
As recommended on the Magento Wiki.