The user mode program always uses system calls to communicate with the hardware. The device file exists solely so that the program can use well-known, simple, standard system calls to get the job done rather than complicated device-specific ones.
Basically, reading and writing a device file involves exactly the same system calls open(), read(), write()` etc. as reading from a vanilla text file. Since many programmers already know how to do this, this makes controlling a device relatively easy - you only have to know the device node name and the protocol for talking to it, rather than the special verbs for programming your snazzy TV tune card.
With a user-space driver you can in principle allow the user space program to use a more complicated way of talking to the driver, but why would you want to? Device files bring a great reduction in complexity for a moderate loss in expressivity, so they are usually preferred. That way only driver programmers have to delve into the intricacies of the specific card.
Where there's an operating system involved, programs don't talk to device drivers, at least not directly. Programs talk to abstractions that, unbeknownst to them, eventually end up talking to device drivers by way of one or more layers of abstraction.
I'm going to skip the complexities of modern operating systems and use CP/M, a microcomputer operating system developed 45 years ago, as an example. CP/M was a layer cake with three layers:
Program. The top layer is a program that does something useful (word processing, playing Space Invaders) by doing computation and I/O. Let's say that at some point the program wants to display the letter 'A' for the user to see. CP/M provides an abstraction known as the console, which is where the user interacting with the program should be looking. The conventional way to do send a character there is with a few assembly instructions:
LD C,2 ; Load 2 into register C
LD E,65 ; Load the ASCII code for 'A' into register E
CALL 5 ; Call CP/M's routine for getting things done
(If you're not familiar with them, registers can be thought of as variables that live in the processor.) We'll get to what the magic numbers 2 and 5 are all about in a minute. The takeaway here is that all the program knows is that there's a console and there's a way to write to it. It doesn't know or care about anything beyond that. This is the first of two abstractions that CP/M uses for I/O.
BDOS. The address 5 the program called is the entry point for the next layer, the Basic Disk Operating System or BDOS. The BDOS provides a whole raft of numbered functions that are like ordering by number from a restaurant menu. You tell it you'd like console output by loading the C register with the function number (2 for console output) and the E register with the character to be sent. Console output is a very simple operation, and the BDOS doesn't really have to do much with it other than call the next layer.
BIOS. The BIOS, or Basic Input/Output System is the layer where all of the hardware-specific code lives. In modern systems, this would be considered a set of device drivers. Like the BDOS, the BIOS provides calls for a standard set of very primitive operations that the BDOS uses to do its business. One of those operations is called CONOUT, which takes care of getting the character the program asked to write two layers above through to whatever hardware does it. (Unlike PCs, things weren't homogeneous back then. Everybody's system had different ways of making it happen.) Console output is a simple pass-through for the BDOS, but doing something more complex like creating a file on a disk might require many BIOS calls to manipulate the media. Again, because the BIOS has a standard, abstract interface, the BDOS always knows how to get what it wants and doesn't care how the BIOS does it.
You're probably wondering why there are two abstractions (program-to-BDOS and BDOS-to-BIOS) instead of just one. The answer is that CP/M and its BDOS could be provided in binary form to computer manufacturers, they'd write a custom BIOS with device drivers for their hardware, bolt the two together and ship it as the OS for their systems. This was a big deal because the BDOS was maintained by one organization and therefore always was a known quantity to user programs, making it possible to run the same applications on a very wide (for the time) variety of hardware. This is why operating systems exist and we don't just write programs that twiddle the hardware directly.
Everything I've described here applies to modern operating systems, too. Unix, for example, abstracts everything as files. It gives programs the same set of system calls (open(), write(), close(), etc.) to communicate whether it's a disk drive or serial port. The set of decisions and abstractions is much more complex, but it still eventually boils down to picking out what device driver code at the bottom layer needs to be run to make the operation happen.
Best Answer
In a modern operating system, device drivers serve two purposes:
It is important to remember these two purposes when you consider what is and isn't either necessary or advisable when you consider using device drivers. Now, breaking down your question into details:
Right. This is because of the "potentially dangerous operations" part. Communicating with a serial port, depending on the exact hardware you're running on, requires:
Typically, these resources are shared between multiple ports, so if your process could do either of these directly, it would be able to interfere with any other process that might be using other ports. Therefore, an intermediary is required that will make sure it only performs actions that won't cause problems with other parts of the system. The device driver is such an intermediary.
This is not usually necessary. Operating systems are supplied with serial port drivers, so you don't need to create one yourself. In general, user applications should never need to supply device drivers: device drivers are a system function that are associated with the hardware they run on, not the applications that want to use it. This relates to the "abstraction layer" purpose: there are multiple types of serial port, and the device driver knows in detail how to talk to the actual type that your computer has installed. Your application only needs to know how to talk to the device driver, and your operating system provides a standard interface that allows that without needing to know which exact driver is in use.
This isn't usually necessary. You only need one device driver in order to achieve both purposes:
That said, there is one case that is similar to the situation you describe, where you may want to have an application-specific device driver which sends data to the hardware-specific device driver. That is if you have a device that you connect to the computer via a serial port, but which you want the application to logically consider as separate to the serial port -- perhaps because you may want to be able to produce a range of devices that have the same behaviour, but maybe there's a serial port version, a USB version, a parallel port version, etc. In that case, it may be worth creating a second driver that provides a second abstraction layer (in this case abstracting the details of how the external device is connected rather than how the port it is connected to works). One example of this (now somewhat relegated to history, thankfully) is the way you used to be able to get mice that connected via serial ports. These would have a driver that implemented the standard mouse protocols and which talked to the serial port driver to actually communicate.