The vast majority of text LCD displays for screens 4×40 or smaller use a controller based on the Hitachi HD44780. It's a pretty nice design in a lot of ways, so its popularity is understandable. One aspect of it puzzles me, however.
The display controller is designed to operate with either 5×8 or 5×10 fonts; according to the 44780 data sheet, the part is available with two stock ROM configurations–one with 248 5×8 character shapes, designated A02, and one with 160 5×8 shapes and 32 5×10 shapes, designated A00. The 5×10 character shapes are only really useful on single-line displays which have a 10-high LCD matrix, and such displays seem to be very rare (I think I've seen data sheets for a handful of them, if that, and I don't know that I've ever actually seen one in person).
I am puzzled, then, by why it seems that every LCD display I've seen based on an HD44780-compatible controller, uses font pattern A00 which has many blank character cells, and whose glyphs for a number of characters (including 0xE4, 0xE6, 0xE7, 0xEA, 0xF0, 0xF1, and 0xF9) are essentially useless with anything other than a single-line 5×10 display. I would have expected that at least some vendors would have at least changed the definitions for those characters to look good on a 5×8 display, but I'm unaware of any having done so. I would also think that for many applications, ROM pattern A02 would be more useful than ROM pattern A00, but all the displays I've ever seen for sale use ROM pattern A00.
To be sure, in a lot of applications, the ability to define custom glyphs substantially mitigates the limitations of the built-in character set, but the display can only show eight different custom glyphs at a time. Having built-in glyphs for things like arrows would mean that applications needing arrows could have more custom glyphs left over for other purposes.
If nothing else, it would seem logical for displays to have an option to fill in character ranges 0x10-0x1F and 0x80-0x9F with useful characters. If a particular controller is only going to be used with 5×8 or 5×16 glass (would seem pretty likely), it should be a simple matter to use the 5×10-font control bit to enable or disable those characters. That would make the controller compatible with software that would expect all of those characters to appear as blank, but would allow software that knows what it is to exploit those other characters.
Best Answer
Economy of scale.
Buy 100,000 chips that will do for all displays at say $0.10 each, or buy 50,000 of one chip at, say, $0.15 and 50,000 of another chip at $0.15 each.
(figures purely fictional - for illustration only)
You do the maths.
While it's not 100% perfect for every display, it does mean that they are cheaper, which is good for all of us.