OK, I've worked out what's going on. Leonidas is right, it's not just the hash that gets encrypted (in the case of the Cipher class method), it's the ID of the hash algorithm concatenated with the digest:
DigestInfo ::= SEQUENCE {
digestAlgorithm AlgorithmIdentifier,
digest OCTET STRING
}
Which is why the encryption by the Cipher and Signature are different.
Use SHA-2. Always. SHA-1 shall be reserved to situations where SHA-2 cannot be used due to interoperability issues with legacy code.
There is no performance issue until actual measures have been performed. Hash functions are fast. In most situations, hash function performance has only negligible impact; even detecting it could prove difficult. Security comes first. Since weaknesses have been found in SHA-1, using it nonetheless requires some robust justification. Using SHA-256 will not be questioned; this is the "default choice". But if you use SHA-1, prepare to be criticized.
Note that there are four functions known as "SHA-2": SHA-224, SHA-256, SHA-384 and SHA-512. SHA-224 and SHA-256 are the same function, save for an internal parameter (the "initial value") and the output size (SHA-224 output size is 28 bytes, whereas SHA-256 offers 32 bytes); they have the same performance characteristics. Similarly, SHA-384 and SHA-512 are the same function performance-wise. SHA-512 uses 64-bit arithmetic operations and is faster than SHA-256 on platforms which offer 64-bit opcodes; on 32-bit platforms, SHA-256 will be faster (note: on 32-bit x86 with native code, it is possible to use the SSE2 opcodes and their 64-bit computing abilities, hence some native code implementations of SHA-512 will be faster than SHA-256 in 32-bit mode; the OpenSSL code does that; but, as far as I know, the SHA-512 implementation in .NET is "managed code"). Also, all the SHA-* functions have some basic granularity, because they process data by chunks: for SHA-256, chunks are 64-byte long, whereas SHA-512 uses 128-byte chunks; when hashing very short data elements, the higher SHA-512 granularity correspondingly lowers its performance. Finally, SHA-256 (on a 32-bit platform) is likely to yield smaller code (i.e. use less L1 cache on the CPU).
So, when in doubt, use SHA-256. If you plan on using SHA-1 then you should doubt.
If you want to use a hash function for a non-cryptographic usage (i.e. the weaknesses are not a problem for you) then, instead of SHA-1, consider MD4.
Best Answer
With RSA, as specified by PKCS#1, the data to be signed is first hashed with a hash function, then the result is padded (a more or less complex operation which transforms the hash result into a modular integer), and then the mathematical operation of RSA is applied on that number. The result is a n-bit integer, where n is the length in bits of the "modulus", usually called "the RSA key size". Basically, for RSA-1024, n is 1024. A 1024-bit integer is encoded as 128 bytes, exactly, as per the encoding method described in PKCS#1 (PKCS#1 is very readable and not too long).
Whether a n-bit RSA key can be used to sign data with a hash function which produces outputs of length m depends on the details of the padding. As the name suggests, padding involves adding some extra data around the hash output, hence n must be greater than m, leaving some room for the extra data. A 1024-bit key can be used with SHA-512 (which produces 512-bit strings). You could not use a 640-bit key with SHA-512 (and you would not, anyway, since 640-bit RSA keys can be broken -- albeit not trivially).