This is implemented and tested for Windows Server 2008 R2, it's however not extensively tested. There are major licensing issues with this setup.
It's possible to have Citrix farm and RDS farm functionality on the same servers. This implementation is done with DNS round robin, connection broker load balancing, redirection with ip address. Maintenance on servers require to logon with mstsc /v:IP /admin to get the correct server, or remotely manage with admin tools. At the moment the RD Connection Broker is a s.p.o.f.
Issues has been seen that if there are no Citrix licenses available, this also affects the RDS farm and user can not logon there either, this could probably however just be an configuration issue (not been investigated). The RDS servers are "per user". Regarding Citrix this is configured for concurrent.
This means that each RDP session towards a server also running the Citrix application will also take a Citrix license. There are therefor no purpose off trying to run CTX and RDS on same server and use RDS for saving licenses for those users that don't need CTX functionality
Citrix uses ICA protocol, RDS uses RDP protocol. You'll need to configure for both if you for example don't want new users to connect to the server due to maintenance.
They won't be aware of each other regarding sessions, meaning, a user (the same) connecting to the Citrix farm and then to the RDS farm will get two separated sessions.
They won't be aware of each other regarding load balancing, but Citrix will however be aware of the server load based on the configuration, take that in account.
RDS is also somewhat, ineffective, then load balancing based on the Connection Broker service.
RDS farm (2008 R2) cannot be selective with what applications and such is available per server, therefore only add those servers that are exactly the same as the Citrix servers. For example only servers that delivers a whole desktop and all have the same applications.
For the RDS farm you'll need to handle the certificates (a certificate containing both the farm name and each server within the farm) if you don't reconfigure RDP for RDP Security Layer.
Extending this answer/Corrections to this answer is encouraged.
You seem to have setup Remote Desktop licensing wrong, and seem to be confused about how it works.
There are two RDS licensing modes. Per device, or per user. Your server has to be in one mode or the other, so you can't mix and match the way you want to.
By default, Windows Server is in per device mode and allows two concurrent RDS connections. (The licensing requirements or implications around that are not on-topic here.) If you choose to use per user mode instead, (in order to allow more than 2 concurrent RDP sessions, being a frequent reason) the server needs to be able to contact a licensing server that has a user CAL for each user that will use RDS. Again, the licensing part of this is not on topic here, but it sounds like your issue is either that this server is not configured with a licensing server to contact, or the licensing server it is contacting does not have any licenses assigned to it.
Best Answer
No. Per user licensing is per user, not per user per server. It doesn't matter which server they connect to.