To speak to each of your concerns:
I've been deploying Java runtime environment releases for a few years now as software installation assignments from Group Policy. I disable updater functionality as a transform to the MSI and deploy updates, as necessary, through mandatory upgrades. If machines need to keep an older JRE (because some application requires it), I use security groups to keep machines from receiving newer upgrades. (Fortunately, I have not had to do this frequently.)
I build transforms to Sun's MSI using Microsoft's Orca tool. It might be nice to have a tool like Adobe's "Customization Wizard", but I can do everything I need with Orca.
I haven't had occasion for users to "manually configure certain settings", but I'd handle it one of two ways. If it's a matter of some users needing some settings that are different that the "norm", I'd either deploy a Group Policy "preference" to set that setting (assuming it's in the user portion of the registry), or an Administrative Template to change the setting (assuming it's in the computer portion of the registry). If it's required that hte user be allowed to change the setting on-demand, I'd grudgingly alter the permissions on the registry to allow the user (really, a security group containing the user) to do so. Grudgingly.
If an app requires its own JRE I'd be apt to tie the installation of that JRE in with the script / GPO that deploys the application and treat the two as a unit. That's the simplest way I can think of to deal with it.
I'm having a hard time recalling what settings live under "Program Files", but I'd grudgingly grant permission to a security group containing user accounts that need to modify these settings, if that was required. I'd probably also hold my head in my hands and curse Sun.
Until Sun gets their act together re: enterprise deployment and management of the JRE, I think it's likely that all of us are going to have hacky workarounds to deal with it. It's frustrating, but sadly typical. It seems like the vast majority developers have no concept of what it's like to do sysadmin work. <sigh>
I also ran into the same problem. After reading a lot of docs I came up with the following "solution":
To workaround the name clashes I renamed User Private Groups by adding a "g" character at the beginning. For example: User=erik, Group=erik. Now on active directory I named the group "gerik". This way I can continue concentrating on AD migration without thinking about the User Private Groups problem right now.
Slowly stop using User Private Groups as this does not seem to be the way to go - at least when using Active directory. This can be done by creating a new group like "unixgrp" or using "Domain Users", but I do not like "Domain Users" because that name is so long when displaying files with "ls".
be careful when migration from User Private Groups to a common group like "unixgrp". Do not just change the group of all files to "unixgrp". If for example a user has group write permissions on a file owned by his user private group and you change the group to "unixgrp", then all users in "unixgrp" will also have write access to that file. So some kind of script has to modify permissions the right way... have fun with that!
I admit that the only real solution whould be to somehow support User Private Groups when using active directory. But I do not know how...
Best Answer
You can take a look into OpenLDAP, or another directory services solution, but I have yet to find an 'all in one' solution such as ActiveDirectory - there always seems to be pieces, such as GPOs, which you need to sort out yourself.
Take a look at:
RedHat Directory Server
Apache Directory Server
OpenLDAP
eDirectory
OpenDS