Validating allowed characters or validating disallowed characters

formsSecurityvalidation

I've always validated my user input based on a list of valid/allowed characters, rather than a list of invalid/disallowed characters (or simply no validation). It's just a habit I picked up, probably on this site and I've never really questioned it until now.

It makes sense if you wish to, say, validate a phone number, or validate an area code, however recently I've realised I'm also validating input such as Bio Text fields, User Comments, etc. for which the input has no solid syntax.

The main advantage has always seemed to be: Validating allowed chars reduces the risk of you missing a potentially malicious character, but increases the risk the of you not allowing a character which the user may want to use. The former is more important.

But, providing I am correctly preventing SQL Injection (with prepared statements) and also escaping output, is there any need for this extra barrier of protection? It seems to me as if I am just allowing practically every character on the keyboard, and am forgetting to allow some common characters.

Is there an accepted practice for this situation? Or am I missing something obvious?

Thanks.

Best Answer

Any implementation trying to detect "malicious characters" is flawed, when you look at the combined properties of such an implementation:

  • A "valid" subset of a character set is not so easy to define. Newline is a control character, and you definitely want to allow that in comments. You'd have work cut out for days or weeks to create a sensible subset of Unicode (and combinations of characters) which could be considered "valid" across the globe.
  • An "invalid" subset is also difficult, if you're making anything even remotely complex. For example, you don't want literal quotes in SQL, but you also don't want literal ampersands or inequality signs in HTML, or backslash in JavaScript. If you have a series of input and output languages, the only way to be sure is to escape user input/output for each one, and test that the escaping works.
  • The set is valid only for a single version of a single character set, so it's not future-proof.
  • You still need to test the full character range to see if there are any security holes.
  • If you're not careful with what you accept, you'll end up annoying users, and they will leave. If you're lucky, one in a thousand will file a bug report.

I'd go so far as to say that validating allowed characters reduces security, because it encourages sloppy implementation (lack of testing/escaping). If you escape where necessary you can instead just test the "nasty" characters, and if they work, you have pretty much guaranteed that other nasty characters will also be harmless to the system.

All this is of course not to say that some characters are nonsensical in some fields, such as two in a numeric field. But even this is often not trivial:

  • Different languages have different decimal and thousand separators. 1,000 == 1 in much of Europe, and 1'000 is a valid way to write 1000 in some places. You don't want to tell those users their way of writing is wrong.
  • Leading zeros, plus signs, hash and asterisk are all valid characters in a phone number. Some countries include an inland prefix ((0) in Switzerland) which you have to use within the country, and have to exclude when using a language code.
  • Contrary to many sloppy web developers, email addresses can contain dash and plus characters, and a whole lot of others which are routinely rejected as if to prove that the company policy is that they would do just fine if it wasn't for all those pesky customers.
  • Names can contain quote characters (Gerard 't Hooft), numbers (John Doe the 5th), punctuation (John Doe, M.D.), and SQL: Robert`); DROP TABLE Students;--
Related Topic