These days, a system may have multiple interfaces, each with multiple addresses, and each address may even have multiple DNS entries associated with it. So what does a "system hostname" even mean?
Many applications will use the system hostname as a default identifier when they communicate elsewhere. For example, if you're collecting syslog messages at a central server, the messages will all be tagged with the hostname of the originating system. In an ideal world you would probably ignore this (because you don't necessarily want to trust the client), but the default behavior -- if you named all your systems "localhost" -- would result in a bunch of log messages that you wouldn't be able to associate with a specific system.
As other folks have pointed out, the system hostname is also a useful identifier if you find yourself remotely accessing a number of system. If you've got five windows attached to a systems named "localhost" then you're going to have a hard time keeping them straight.
In a similar vein, we try to make the system hostname matches the hostname we use for administrative access to a system. This helps avoid confusion when referring to the system (in email, conversations, documentation, etc).
Regarding DNS:
You want to have proper forward and reverse DNS entries for your applications in order to avoid confusion. You need some forward entry (name -> ip address) for people to be able to access your application conveniently. Having the reverse entry match is useful for an number of reasons -- for example, it helps you correctly identify the application if you find the corresponding ip address in a log.
Note that here I'm talking about "applications" and not "systems", because -- particularly with web servers -- it's common to have multiple ip addresses on a system, associated with different hostnames and services.
Trying to maintain name to ip mappings in your /etc/hosts file quickly becomes difficult as you manage an increasing number of systems. It's very easy to for the local hosts file to fall out of sync with respect to DNS, potentially leading to confusion and in some cases malfunction (because something tries to bind to an ip address that no longer exists on the system, for example).
Names resolved from DNS are case insensitive. This is important to prevent confusion. If it was case sensitive then we would have eight variants of .com (.com, .Com, .cOm, .COm, .coM, .CoM, .cOM, and .COM). Country codes would have four.
If name resolution is case-sensitive for Ping it is not being done by DNS.
Best Answer
The Internet standards (Request for Comments) for protocols mandate that component hostname labels may contain only the ASCII letters 'a' through 'z' (in a case-insensitive manner), the digits '0' through '9', and the hyphen ('-'). The original specification of hostnames in RFC 952, mandated that labels could not start with a digit or with a hyphen, and must not end with a hyphen. However, a subsequent specification (RFC 1123) permitted hostname labels to start with digits. No other symbols, punctuation characters, or white space are permitted.
Hostname is Case Insensitive.