Please consider long and hard if you can't get around implementing your own cryptography
The ugly truth of the matter is that if you are asking this question you will probably not be able to design and implement a secure system.
Let me illustrate my point: Imagine you are building a web application and you need to store some session data. You could assign each user a session ID and store the session data on the server in a hash map mapping session ID to session data. But then you have to deal with this pesky state on the server and if at some point you need more than one server things will get messy. So instead you have the idea to store the session data in a cookie on the client side. You will encrypt it of course so the user cannot read and manipulate the data. So what mode should you use? Coming here you read the top answer (sorry for singling you out myforwik). The first one covered - ECB - is not for you, you want to encrypt more than one block, the next one - CBC - sounds good and you don't need the parallelism of CTR, you don't need random access, so no XTS and patents are a PITA, so no OCB. Using your crypto library you realize that you need some padding because you can only encrypt multiples of the block size. You choose PKCS7 because it was defined in some serious cryptography standards. After reading somewhere that CBC is provably secure if used with a random IV and a secure block cipher, you rest at ease even though you are storing your sensitive data on the client side.
Years later after your service has indeed grown to significant size, an IT security specialist contacts you in a responsible disclosure. She's telling you that she can decrypt all your cookies using a padding oracle attack, because your code produces an error page if the padding is somehow broken.
This is not a hypothetical scenario: Microsoft had this exact flaw in ASP.NET until a few years ago.
The problem is there are a lot of pitfalls regarding cryptography and it is extremely easy to build a system that looks secure for the layman but is trivial to break for a knowledgeable attacker.
What to do if you need to encrypt data
For live connections use TLS (be sure to check the hostname of the certificate and the issuer chain). If you can't use TLS, look for the highest level API your system has to offer for your task and be sure you understand the guarantees it offers and more important what it does not guarantee. For the example above a framework like Play offers client side storage facilities, it does not invalidate the stored data after some time, though, and if you changed the client side state, an attacker can restore a previous state without you noticing.
If there is no high level abstraction available use a high level crypto library. A prominent example is NaCl and a portable implementation with many language bindings is Sodium. Using such a library you do not have to care about encryption modes etc. but you have to be even more careful about the usage details than with a higher level abstraction, like never using a nonce twice. For custom protocol building (say you want something like TLS, but not over TCP or UDP) there are frameworks like Noise and associated implementations that do most of the heavy lifting for you, but their flexibility also means there is a lot of room for error, if you don't understand in depth what all the components do.
If for some reason you cannot use a high level crypto library, for example because you need to interact with existing system in a specific way, there is no way around educating yourself thoroughly. I recommend reading Cryptography Engineering by Ferguson, Kohno and Schneier. Please don't fool yourself into believing you can build a secure system without the necessary background. Cryptography is extremely subtle and it's nigh impossible to test the security of a system.
Comparison of the modes
Encryption only:
- Modes that require padding:
Like in the example, padding can generally be dangerous because it opens up the possibility of padding oracle attacks. The easiest defense is to authenticate every message before decryption. See below.
- ECB encrypts each block of data independently and the same plaintext block will result in the same ciphertext block. Take a look at the ECB encrypted Tux image on the ECB Wikipedia page to see why this is a serious problem. I don't know of any use case where ECB would be acceptable.
- CBC has an IV and thus needs randomness every time a message is encrypted, changing a part of the message requires re-encrypting everything after the change, transmission errors in one ciphertext block completely destroy the plaintext and change the decryption of the next block, decryption can be parallelized / encryption can't, the plaintext is malleable to a certain degree - this can be a problem.
- Stream cipher modes: These modes generate a pseudo random stream of data that may or may not depend the plaintext. Similarly to stream ciphers generally, the generated pseudo random stream is XORed with the plaintext to generate the ciphertext. As you can use as many bits of the random stream as you like you don't need padding at all. Disadvantage of this simplicity is that the encryption is completely malleable, meaning that the decryption can be changed by an attacker in any way he likes as for a plaintext p1, a ciphertext c1 and a pseudo random stream r and attacker can choose a difference d such that the decryption of a ciphertext c2=c1⊕d is p2 = p1⊕d, as p2 = c2⊕r = (c1 ⊕ d) ⊕ r = d ⊕ (c1 ⊕ r). Also the same pseudo random stream must never be used twice as for two ciphertexts c1=p1⊕r and c2=p2⊕r, an attacker can compute the xor of the two plaintexts as c1⊕c2=p1⊕r⊕p2⊕r=p1⊕p2. That also means that changing the message requires complete reencryption, if the original message could have been obtained by an attacker. All of the following steam cipher modes only need the encryption operation of the block cipher, so depending on the cipher this might save some (silicon or machine code) space in extremely constricted environments.
- CTR is simple, it creates a pseudo random stream that is independent of the plaintext, different pseudo random streams are obtained by counting up from different nonces/IVs which are multiplied by a maximum message length so that overlap is prevented, using nonces message encryption is possible without per message randomness, decryption and encryption are completed parallelizable, transmission errors only effect the wrong bits and nothing more
- OFB also creates a pseudo random stream independent of the plaintext, different pseudo random streams are obtained by starting with a different nonce or random IV for every message, neither encryption nor decryption is parallelizable, as with CTR using nonces message encryption is possible without per message randomness, as with CTR transmission errors only effect the wrong bits and nothing more
- CFB's pseudo random stream depends on the plaintext, a different nonce or random IV is needed for every message, like with CTR and OFB using nonces message encryption is possible without per message randomness, decryption is parallelizable / encryption is not, transmission errors completely destroy the following block, but only effect the wrong bits in the current block
- Disk encryption modes: These modes are specialized to encrypt data below the file system abstraction. For efficiency reasons changing some data on the disc must only require the rewrite of at most one disc block (512 bytes or 4kib). They are out of scope of this answer as they have vastly different usage scenarios than the other. Don't use them for anything except block level disc encryption. Some members: XEX, XTS, LRW.
Authenticated encryption:
To prevent padding oracle attacks and changes to the ciphertext, one can compute a message authentication code (MAC) on the ciphertext and only decrypt it if it has not been tampered with. This is called encrypt-then-mac and should be preferred to any other order. Except for very few use cases authenticity is as important as confidentiality (the latter of which is the aim of encryption). Authenticated encryption schemes (with associated data (AEAD)) combine the two part process of encryption and authentication into one block cipher mode that also produces an authentication tag in the process. In most cases this results in speed improvement.
- CCM is a simple combination of CTR mode and a CBC-MAC. Using two block cipher encryptions per block it is very slow.
- OCB is faster but encumbered by patents. For free (as in freedom) or non-military software the patent holder has granted a free license, though.
- GCM is a very fast but arguably complex combination of CTR mode and GHASH, a MAC over the Galois field with 2^128 elements. Its wide use in important network standards like TLS 1.2 is reflected by a special instruction Intel has introduced to speed up the calculation of GHASH.
Recommendation:
Considering the importance of authentication I would recommend the following two block cipher modes for most use cases (except for disk encryption purposes): If the data is authenticated by an asymmetric signature use CBC, otherwise use GCM.
The short answer to your question is: SHA-1 is as secure as you can get. MD5 would be fine too, even MD4; but it could make some investors nervous. For public relations, it is best to use a "better" hash function, e.g. SHA-256, even if you truncate its output to 160 or 128 bits (to save on storage cost). Some of the SHA-3 round-2 candidates appear to be faster than SHA-1 while being arguably "more secure"; yet they are still a bit new, so sticking to SHA-256 or SHA-512 would be a safer route right now. It would make you look professional and cautious, which is good.
Note that "as secure as you can get" is not the same as "perfectly safe". See below for rather lengthy explanations.
About known attacks:
The known attacks on MD4, MD5 and SHA-1 are about collisions, which do not impact preimage resistance. It has been shown that MD4 has a few weaknesses which can be (only theoretically) exploited when trying to break HMAC/MD4, but this does not apply to your problem. The 2106 second preimage attack in the paper by Kesley and Schneier is a generic trade-off which applies only to very long inputs (260 bytes; that's a million terabytes -- notice how 106+60 exceeds 160; that's where you see that the trade-off has nothing magic in it).
The rest of this message assumes that the hash function you use (e.g. SHA-1) is a "black box" with no special property that the attacker may use. That's what you have right now even with the "broken" hash functions MD5 and SHA-1.
About rainbow tables:
The "rainbow attack" is actually cost-sharing of a dictionary or brute force attack. It is a derivative from the time-memory trade-off first described by Hellman in 1980. Assuming that you have N possible passwords (that's the size of your dictionary, or 2n if you consider brute-forcing a hash function with an output of n bits), there is a time-sharing attack in which you precompute the N hashed passwords and store them in a big table. If you sort the hash outputs, you can get your password in a single lookup. A rainbow table is a smart way to store that table with a much reduced space. You store only N/t hashed passwords, and you crack passwords with O(t2) lookups. Rainbow tables allow you to virtually handle precomputed tables much larger than what you can realistically store.
However, rainbow or not, the attacker still has to run the full attack at least once. This can be viewed as several successive optimization layers:
- The brute-force / dictionary attack has cost N for cracking each password.
- With a pre-computed table, the attacker pays that cost N once and can thereafter attack many passwords with very small extra cost per password.
- If the pre-computed table is a rainbow table, then N can be somewhat bigger, because storage cost is reduced. The bottleneck on N becomes the CPU power that the attacker can muster, not the size of his harddisks.
If N is large enough that the CPU-cost of hashing N passwords is ludicrous, then such an attack is not feasible, regardless of whether rainbow tables are used or not. This means that a (preimage-resistant) hash function with an output of 80 bits or more is enough to make brute-force attack infeasible.
About salts:
Salts are a way to defeat pre-computations. In the description above, the salt brings back the attacker to step 1: salting prevents the attacker from sharing the O(N) cost between several attacked passwords. Pre-computed tables, a fortiori rainbow tables, are no longer feasible.
You want salting because when the hashed data consists in passwords, i.e. something which fits within the brain of a random human being, then N can be quite low: humans are really bad at choosing and remembering passwords. This is what "dictionary attacks" are about: that's using a reduced space of potential passwords (the "dictionary") under the assumption that many user passwords will be in that specially selected space.
Hence salting will at least prevent the attacker from using pre-computed tables, in particular pre-computed rainbow tables. This assumes that the attacker will be able to break one password or two; we do not want him to break 1000 other passwords with little extra overhead.
Also, salting is good for public relations.
About SHA-1 cost:
The elementary cost of SHA-1 is about hashing a 64-byte block. That's how SHA-1 works: data is padded, then split into 64-byte blocks. The cost of processing a single block is about 500 clock cycles on an Intel Core2 system, and that's for a single core. MD5 and MD4 are faster, count about 400 and 250 cycles, respectively. Do not forget that most modern CPU have several cores, so multiply accordingly.
Some salting schemes prescribe huge salts; e.g. what enters the hash function is actually 40000 successive copies of a single 128-bit salt, followed by the password itself. This makes password hashing more expensive (by a factor of 10000 with my example), both for the legitimate user and for the attacker. Whether this is a good idea depends on the setup. For login on a desktop system, this is good: the user will not even notice that it took 10ms to hash his password, instead of 1µs; but the cost for the attacker has risen by a very noticeable factor 10000. On shared servers with thousands of clients per second, the aggregate cost may become prohibitive. Conceptually, raising the bar by the same factor for the legitimate user and the attacker is not ultimately good security; but it can be a worthwhile idea in some specific situations.
About online attacks:
All of the above is about defeating offline attacks. An offline attack is an attack where the attacker has all the data he needs in order to "test" passwords; e.g. the attacker could get a copy of the database holding the hashed passwords. In an offline attack, the attacker is limited only by his computational resources. Conversely, an online attack is an attack where each guess by the attacker must go through an honest verifier (e.g. the attacker simply tries to log in on the attacked system). Online attacks are thwarted by enforcing limits on how many passwords can be tried per second. Extreme examples are smartcards which shut down after three wrong PINs.
Usually, for password security, it pays off much more to arrange the system for not letting an attacker build an offline attack. That's what Unix systems do: the hashed passwords, which used to be in the world-readable /etc/password file, are now in the /etc/shadow file which is protected against read access, except by a few privileged applications. The assumption here is that if the attacker can read /etc/shadow, then he probably has enough control over the system that he does not really need passwords anymore...
Best Answer
Predictable IVs can be exploited by chosen plain text.
Pretend that Eve is a DBA at an insurance company. The company collects medical histories from beneficiaries that include a lot of true/false check boxes about medical conditions. This company also happens to its own health insurance provider. Eve realizes that Alice could be blackmailed if she can discover that Alice has a particularly embarrassing medical condition. However, the value in each of these fields is encrypted, so even though Eve is the DBA, she only has access to the cipher text.
In CBC, the IV is XORed (noted by "⊕" below) with the plain text, then run through the block cipher: C1 = Ek(IV ⊕ P1).
Since Eve is a beneficiary of the insurance company, she can choose the plain text for her own medical record, and since she is the DBA, she can examine anyone's cipher text. In addition to using predictable IVs, the sloppy application developer did a poor job of validating the application inputs. If Eve can predict the IVs that will be applied to her (IVeve) and Alice's (IValice) records in advance, she can choose the plain text for her own record like this: Peve = IVeve ⊕ IValice ⊕ "false"
The application encrypts this plain text like this:
Ceve = Ek(IVeve ⊕ Peve) = Ek(IVeve ⊕ (IVeve ⊕ IValice ⊕ "false"))
The IVeve ⊕ IVeve cancels out, which means that Ceve = Ek(IValice ⊕ "false")
Now Eve can compare Ceve and Calice. If they are different, she knows that Alice must have entered "true" for that medical condition.
Making IVs unpredictable thwarts this attack, and an easy way to make them unpredictable is to choose them randomly after the plain text has been supplied.