Php – How to upload files directly to S3 using PHP and with progress bar

DISCLAIMER: This answer was written in 2008.

Since then, PHP has given us password_hash and password_verify and, since their introduction, they are the recommended password hashing & checking method.

The theory of the answer is still a good read though.

TL;DR

Don'ts

• Don't limit what characters users can enter for passwords. Only idiots do this.
• Don't limit the length of a password. If your users want a sentence with supercalifragilisticexpialidocious in it, don't prevent them from using it.
• Don't strip or escape HTML and special characters in the password.
• Never store your user's password in plain-text.
• Never email a password to your user except when they have lost theirs, and you sent a temporary one.
• Never, ever log passwords in any manner.
• Never hash passwords with SHA1 or MD5 or even SHA256! Modern crackers can exceed 60 and 180 billion hashes/second (respectively).
• Don't mix bcrypt and with the raw output of hash(), either use hex output or base64_encode it. (This applies to any input that may have a rogue \0 in it, which can seriously weaken security.)

Dos

• Use scrypt when you can; bcrypt if you cannot.
• Use PBKDF2 if you cannot use either bcrypt or scrypt, with SHA2 hashes.
• Reset everyone's passwords when the database is compromised.
• Implement a reasonable 8-10 character minimum length, plus require at least 1 upper case letter, 1 lower case letter, a number, and a symbol. This will improve the entropy of the password, in turn making it harder to crack. (See the "What makes a good password?" section for some debate.)

Why hash passwords anyway?

The objective behind hashing passwords is simple: preventing malicious access to user accounts by compromising the database. So the goal of password hashing is to deter a hacker or cracker by costing them too much time or money to calculate the plain-text passwords. And time/cost are the best deterrents in your arsenal.

Another reason that you want a good, robust hash on a user accounts is to give you enough time to change all the passwords in the system. If your database is compromised you will need enough time to at least lock the system down, if not change every password in the database.

Jeremiah Grossman, CTO of Whitehat Security, stated on White Hat Security blog after a recent password recovery that required brute-force breaking of his password protection:

Interestingly, in living out this nightmare, I learned A LOT I didn’t know about password cracking, storage, and complexity. I’ve come to appreciate why password storage is ever so much more important than password complexity. If you don’t know how your password is stored, then all you really can depend upon is complexity. This might be common knowledge to password and crypto pros, but for the average InfoSec or Web Security expert, I highly doubt it.

(Emphasis mine.)

What makes a good password anyway?

Entropy. (Not that I fully subscribe to Randall's viewpoint.)

In short, entropy is how much variation is within the password. When a password is only lowercase roman letters, that's only 26 characters. That isn't much variation. Alpha-numeric passwords are better, with 36 characters. But allowing upper and lower case, with symbols, is roughly 96 characters. That's a lot better than just letters. One problem is, to make our passwords memorable we insert patterns—which reduces entropy. Oops!

Password entropy is approximated easily. Using the full range of ascii characters (roughly 96 typeable characters) yields an entropy of 6.6 per character, which at 8 characters for a password is still too low (52.679 bits of entropy) for future security. But the good news is: longer passwords, and passwords with unicode characters, really increase the entropy of a password and make it harder to crack.

There's a longer discussion of password entropy on the Crypto StackExchange site. A good Google search will also turn up a lot of results.

In the comments I talked with @popnoodles, who pointed out that enforcing a password policy of X length with X many letters, numbers, symbols, etc, can actually reduce entropy by making the password scheme more predictable. I do agree. Randomess, as truly random as possible, is always the safest but least memorable solution.

So far as I've been able to tell, making the world's best password is a Catch-22. Either its not memorable, too predictable, too short, too many unicode characters (hard to type on a Windows/Mobile device), too long, etc. No password is truly good enough for our purposes, so we must protect them as though they were in Fort Knox.

Best practices

Bcrypt and scrypt are the current best practices. Scrypt will be better than bcrypt in time, but it hasn't seen adoption as a standard by Linux/Unix or by webservers, and hasn't had in-depth reviews of its algorithm posted yet. But still, the future of the algorithm does look promising. If you are working with Ruby there is an scrypt gem that will help you out, and Node.js now has its own scrypt package. You can use Scrypt in PHP either via the Scrypt extension or the Libsodium extension (both are available in PECL).

I highly suggest reading the documentation for the crypt function if you want to understand how to use bcrypt, or finding yourself a good wrapper or use something like PHPASS for a more legacy implementation. I recommend a minimum of 12 rounds of bcrypt, if not 15 to 18.

I changed my mind about using bcrypt when I learned that bcrypt only uses blowfish's key schedule, with a variable cost mechanism. The latter lets you increase the cost to brute-force a password by increasing blowfish's already expensive key schedule.

Average practices

I almost can't imagine this situation anymore. PHPASS supports PHP 3.0.18 through 5.3, so it is usable on almost every installation imaginable—and should be used if you don't know for certain that your environment supports bcrypt.

But suppose that you cannot use bcrypt or PHPASS at all. What then?

Try an implementation of PDKBF2 with the maximum number of rounds that your environment/application/user-perception can tolerate. The lowest number I'd recommend is 2500 rounds. Also, make sure to use hash_hmac() if it is available to make the operation harder to reproduce.

Future Practices

Coming in PHP 5.5 is a full password protection library that abstracts away any pains of working with bcrypt. While most of us are stuck with PHP 5.2 and 5.3 in most common environments, especially shared hosts, @ircmaxell has built a compatibility layer for the coming API that is backward compatible to PHP 5.3.7.

Cryptography Recap & Disclaimer

The computational power required to actually crack a hashed password doesn't exist. The only way for computers to "crack" a password is to recreate it and simulate the hashing algorithm used to secure it. The speed of the hash is linearly related to its ability to be brute-forced. Worse still, most hash algorithms can be easily parallelized to perform even faster. This is why costly schemes like bcrypt and scrypt are so important.

You cannot possibly foresee all threats or avenues of attack, and so you must make your best effort to protect your users up front. If you do not, then you might even miss the fact that you were attacked until it's too late... and you're liable. To avoid that situation, act paranoid to begin with. Attack your own software (internally) and attempt to steal user credentials, or modify other user's accounts or access their data. If you don't test the security of your system, then you cannot blame anyone but yourself.

Lastly: I am not a cryptographer. Whatever I've said is my opinion, but I happen to think it's based on good ol' common sense ... and lots of reading. Remember, be as paranoid as possible, make things as hard to intrude as possible, and then, if you are still worried, contact a white-hat hacker or cryptographer to see what they say about your code/system.

Introduction

To browse and select a file for upload you need a HTML <input type="file"> field in the form. As stated in the HTML specification you have to use the POST method and the enctype attribute of the form has to be set to "multipart/form-data".

<form action="upload" method="post" enctype="multipart/form-data">
    <input type="text" name="description" />
    <input type="file" name="file" />
    <input type="submit" />
</form>

After submitting such a form, the binary multipart form data is available in the request body in a different format than when the enctype isn't set.

Before Servlet 3.0, the Servlet API didn't natively support multipart/form-data. It supports only the default form enctype of application/x-www-form-urlencoded. The request.getParameter() and consorts would all return null when using multipart form data. This is where the well known Apache Commons FileUpload came into the picture.

Don't manually parse it!

You can in theory parse the request body yourself based on ServletRequest#getInputStream(). However, this is a precise and tedious work which requires precise knowledge of RFC2388. You shouldn't try to do this on your own or copypaste some homegrown library-less code found elsewhere on the Internet. Many online sources have failed hard in this, such as roseindia.net. See also uploading of pdf file. You should rather use a real library which is used (and implicitly tested!) by millions of users for years. Such a library has proven its robustness.

When you're already on Servlet 3.0 or newer, use native API

If you're using at least Servlet 3.0 (Tomcat 7, Jetty 9, JBoss AS 6, GlassFish 3, etc), then you can just use standard API provided HttpServletRequest#getPart() to collect the individual multipart form data items (most Servlet 3.0 implementations actually use Apache Commons FileUpload under the covers for this!). Also, normal form fields are available by getParameter() the usual way.

First annotate your servlet with @MultipartConfig in order to let it recognize and support multipart/form-data requests and thus get getPart() to work:

@WebServlet("/upload")
@MultipartConfig
public class UploadServlet extends HttpServlet {
    // ...
}

Then, implement its doPost() as follows:

protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    String description = request.getParameter("description"); // Retrieves <input type="text" name="description">
    Part filePart = request.getPart("file"); // Retrieves <input type="file" name="file">
    String fileName = Paths.get(filePart.getSubmittedFileName()).getFileName().toString(); // MSIE fix.
    InputStream fileContent = filePart.getInputStream();
    // ... (do your job here)
}

Note the Path#getFileName(). This is a MSIE fix as to obtaining the file name. This browser incorrectly sends the full file path along the name instead of only the file name.

In case you want to upload multiple files via either multiple="true",

<input type="file" name="files" multiple="true" />

or the old-fashioned way with multiple inputs,

<input type="file" name="files" />
<input type="file" name="files" />
<input type="file" name="files" />
...

then you can collect them as below (unfortunately there is no such method as request.getParts("files")):

protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    // ...
    List<Part> fileParts = request.getParts().stream().filter(part -> "files".equals(part.getName()) && part.getSize() > 0).collect(Collectors.toList()); // Retrieves <input type="file" name="files" multiple="true">

    for (Part filePart : fileParts) {
        String fileName = Paths.get(filePart.getSubmittedFileName()).getFileName().toString(); // MSIE fix.
        InputStream fileContent = filePart.getInputStream();
        // ... (do your job here)
    }
}

When you're not on Servlet 3.1 yet, manually get submitted file name

Note that Part#getSubmittedFileName() was introduced in Servlet 3.1 (Tomcat 8, Jetty 9, WildFly 8, GlassFish 4, etc). If you're not on Servlet 3.1 yet, then you need an additional utility method to obtain the submitted file name.

private static String getSubmittedFileName(Part part) {
    for (String cd : part.getHeader("content-disposition").split(";")) {
        if (cd.trim().startsWith("filename")) {
            String fileName = cd.substring(cd.indexOf('=') + 1).trim().replace("\"", "");
            return fileName.substring(fileName.lastIndexOf('/') + 1).substring(fileName.lastIndexOf('\\') + 1); // MSIE fix.
        }
    }
    return null;
}

String fileName = getSubmittedFileName(filePart);

Note the MSIE fix as to obtaining the file name. This browser incorrectly sends the full file path along the name instead of only the file name.

When you're not on Servlet 3.0 yet, use Apache Commons FileUpload

If you're not on Servlet 3.0 yet (isn't it about time to upgrade?), the common practice is to make use of Apache Commons FileUpload to parse the multpart form data requests. It has an excellent User Guide and FAQ (carefully go through both). There's also the O'Reilly ("cos") MultipartRequest, but it has some (minor) bugs and isn't actively maintained anymore for years. I wouldn't recommend using it. Apache Commons FileUpload is still actively maintained and currently very mature.

In order to use Apache Commons FileUpload, you need to have at least the following files in your webapp's /WEB-INF/lib:

• commons-fileupload.jar
• commons-io.jar

Your initial attempt failed most likely because you forgot the commons IO.

Here's a kickoff example how the doPost() of your UploadServlet may look like when using Apache Commons FileUpload:

protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    try {
        List<FileItem> items = new ServletFileUpload(new DiskFileItemFactory()).parseRequest(request);
        for (FileItem item : items) {
            if (item.isFormField()) {
                // Process regular form field (input type="text|radio|checkbox|etc", select, etc).
                String fieldName = item.getFieldName();
                String fieldValue = item.getString();
                // ... (do your job here)
            } else {
                // Process form file field (input type="file").
                String fieldName = item.getFieldName();
                String fileName = FilenameUtils.getName(item.getName());
                InputStream fileContent = item.getInputStream();
                // ... (do your job here)
            }
        }
    } catch (FileUploadException e) {
        throw new ServletException("Cannot parse multipart request.", e);
    }

    // ...
}

It's very important that you don't call getParameter(), getParameterMap(), getParameterValues(), getInputStream(), getReader(), etc on the same request beforehand. Otherwise the servlet container will read and parse the request body and thus Apache Commons FileUpload will get an empty request body. See also a.o. ServletFileUpload#parseRequest(request) returns an empty list.

Note the FilenameUtils#getName(). This is a MSIE fix as to obtaining the file name. This browser incorrectly sends the full file path along the name instead of only the file name.

Alternatively you can also wrap this all in a Filter which parses it all automagically and put the stuff back in the parametermap of the request so that you can continue using request.getParameter() the usual way and retrieve the uploaded file by request.getAttribute(). You can find an example in this blog article.

Workaround for GlassFish3 bug of getParameter() still returning null

Note that Glassfish versions older than 3.1.2 had a bug wherein the getParameter() still returns null. If you are targeting such a container and can't upgrade it, then you need to extract the value from getPart() with help of this utility method:

private static String getValue(Part part) throws IOException {
    BufferedReader reader = new BufferedReader(new InputStreamReader(part.getInputStream(), "UTF-8"));
    StringBuilder value = new StringBuilder();
    char[] buffer = new char[1024];
    for (int length = 0; (length = reader.read(buffer)) > 0;) {
        value.append(buffer, 0, length);
    }
    return value.toString();
}

String description = getValue(request.getPart("description")); // Retrieves <input type="text" name="description">
    

Saving uploaded file (don't use getRealPath() nor part.write()!)

Head to the following answers for detail on properly saving the obtained InputStream (the fileContent variable as shown in the above code snippets) to disk or database:

• Recommended way to save uploaded files in a servlet application
• How to upload an image and save it in database?
• How to convert Part to Blob, so I can store it in MySQL?

Serving uploaded file

Head to the following answers for detail on properly serving the saved file from disk or database back to the client:

• Load images from outside of webapps / webcontext / deploy folder using <h:graphicImage> or <img> tag
• How to retrieve and display images from a database in a JSP page?
• Simplest way to serve static data from outside the application server in a Java web application
• Abstract template for static resource servlet supporting HTTP caching

Ajaxifying the form

Head to the following answers how to upload using Ajax (and jQuery). Do note that the servlet code to collect the form data does not need to be changed for this! Only the way how you respond may be changed, but this is rather trivial (i.e. instead of forwarding to JSP, just print some JSON or XML or even plain text depending on whatever the script responsible for the Ajax call is expecting).

• How to upload files to server using JSP/Servlet and Ajax?
• Send a file as multipart through xmlHttpRequest
• HTML5 drag'n'drop file upload to Java Servlet

Hope this all helps :)