Using multiple connections per user could be problematic because you will hit the servers connection limit earlier when you have a lot of users.
TCP, however, has the advantage that packets are always received in whole and in order. You can use that to implement your own application-level session handling system which works in a preemtive ("time-sharing") multi-threading way.
When there are multiple file downloads taking place at the same time, you could split each file transfer into individual packets and prefix each packet with an identifier which says to which file transfer it belongs. Then you put the packets of each transfer into a queue and process these queues in round-robin order. That means when you have 3 file transfers A, B and C consisting of 3, 5 and 7 packets each, you would send these 15 packages in the order ABCABCABCBCBCCC. Make sure that you are able to add a new queue at any time between sends so that you can enter a new transfer while other transfers are running.
On the other side you read the prefix of each packet to find out to which file transfer it belongs and append it to the corresponding input stream.
A good API to manage all this on Java is non-blocking IO. You should use that anyway on a server which handles a lot of users, because that way you avoid to create individual threads for all users.
Why Rewrite What you can Get Off the Shelf?
Why not use RedDwarf Server (formerly Project DarkStar)?
RedDwarf Server is an open source middleware solution for developing the server-side of massively multiplayer online games. It is the official community fork of Project Darkstar, an open source project supported and managed by Sun Microsystems. - from RedDwarf's Wikipedia article
RedDwarf's domain seems down today (2013-06-12), but there's still a wiki, and they are migrating to a GitHub repo.
RedDward presents its philosophy and goals as:
- Make server-side game code reliable, scalable, persistent, and fault-tolerant in a manner that is transparent to the game developer.
- Present a simple single-threaded event-driven programming model to the developer. The developer should never have his or her code fail due to interactions between code handling different events.
(from the RedDwarf Tutorial)
Not that this doesn't mean that server-code is single-threaded, but that it is abstracted from the game developer's perspective. The RedDwarf Tutorial provides a lot more information on what RedDwarf can do and clarifies many of its design decisions.
One point of concern for you, though, was that the multi-node capability wasn't fully implemented last time I check (ca. 2011).
From Scratch
That shouldn't stop you from attempting to do most of these things from scratch, if you value the learning experience. However, this is a major effort and will be rather time-consuming, and as, you noted in your question, some of these issues are rather low-level in the tech stack to deal with and will greatly increase the complexity of the code you'll have to maintain.
But anyways, regarding your 3 options, your first one seems the best to me, if you were to go for a home-made implementation. Option 2 (using HTTP servlets) seems only adapted for some games, though I guess it could be a relatively decent alternative to implement something yourself while still delegating to the a middleware, and you could benefit from many web server additions for handling the load (cache modules, etc...). Option 3 (using JMS + JEE) indeed seems overengineered, but it's hard to know for sure without knowing what you have in mind.
And if you're here to try to learn, then obviously Option 1 will cover a lot of ground. But it's going to be a rather uphill battle.
Best Answer
Java's ServerSocket returns a new Socket object for each client that connects. This Socket object is the same type as is used in a client, so supports exactly the same options. You can simply set the keepalive option there.