Short answer: you can't. Ports below 1024 can be opened only by root. As per comment - well, you can, using CAP_NET_BIND_SERVICE, but that approach, applied to java bin will make any java program to be run with this setting, which is undesirable, if not a security risk.
The long answer: you can redirect connections on port 80 to some other port you can open as normal user.
Run as root:
# iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080
As loopback devices (like localhost) do not use the prerouting rules, if you need to use localhost, etc., add this rule as well (thanks @Francesco):
# iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-ports 8080
NOTE: The above solution is not well suited for multi-user systems, as any user can open port 8080 (or any other high port you decide to use), thus intercepting the traffic. (Credits to CesarB).
EDIT: as per comment question - to delete the above rule:
# iptables -t nat --line-numbers -n -L
This will output something like:
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 REDIRECT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 redir ports 8088
2 REDIRECT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 redir ports 8080
The rule you are interested in is nr. 2, so to delete it:
# iptables -t nat -D PREROUTING 2
Take a look at tc. (man tc)
Your first problem is uniquely identifying each "user".
Will each user be using a different IP? If so, tc will let you share the interface fairly and divide available resources per IP.
Your second objective is much trickier. It seems to me like you're talking about multiple vhosts on a web server. You can process the logs in batches (every hour or so) to track total bytes transferred by vhost and then use that to "choke" the IP using tc.
Differentiating between local and remote traffic should be manageable using tc as long as you know all your internal subnets ahead of time.
Good luck.
Best Answer
I managed to figure this out.
I wrote an LD_PRELOAD library that overrides send, recv, read, write family of functions and logs these operations on sockets.
The source code is very experimental and not secure, but anyway, I put it on SourceForge:
https://sourceforge.net/projects/netacct/