In our instance, our problem was solved by sysctl parameters, one different from Maciej.
Please note that I do not speak for the OP (buecking), I came on this post due to the problem being related by the basic detail (no multicast traffic in userland).
We have an application that reads data sent to four multicast addresses, and a unique port per multicast address, from an appliance that is (usually) connected directly to an interface on the receiving server.
We were attempting to deploy this software on a customer site when it mysteriously failed with no known reason. Attempts at debugging this software resulted in inspecting every system call, ultimately they all told us the same thing:
Our software asks for data, and the OS never provides any.
The multicast packet counter incremented, tcpdump showed the traffic reaching the box/specific interface, yet we couldn't do anything with it. SELinux was disabled, iptables was running but had no rules in any of the tables.
Stumped, we were.
In randomly poking around, we started thinking about the kernel parameters that sysctl handles, but none of the documented features was either particularly relevant, or if they had to do with multicast traffic, they were enabled. Oh, and ifconfig did list "MULTICAST" in the feature line (up, broadcast, running, multicast). Out of curiosity we looked at /etc/sysctl.conf. 'lo and behold, this customer's base image had a couple of extra lines added to it at the bottom.
In our case, the customer had set net.ipv4.all.rp_filter = 1. rp_filter is the Route Path filter, which (as I understand it) rejects all traffic that could not have possibly reached this box. Network subnet hopping, the thought being that the source IP is being spoofed.
Well, this server was on a 192.168.1/24 subnet and the appliance's source IP address for the multicast traffic was somewhere in the 10.* network. Thus, the filter was preventing the server from doing anything meaningful with the traffic.
A couple of tweaks approved by the customer; net.ipv4.eth0.rp_filter = 1 and net.ipv4.eth1.rp_filter = 0 and we were running happily.
That's a lot of multicast streams, typically NICs have a low limit for hardware filtering and when you exceed that they either drop everything (poor implementation on cheap NICs), or forward everything to the operating system for it to filter instead. When the operating system is performing the filtering your processor usage is going to sky rocket.
Aside of investigating different hardware, which you list some, you could extend to 10GigE based too, the only option is to use proxy servers.
By experimentation find a number of multicast streams which can be managed reliably, then forward the streams on via TCP to a central server or set of servers. That central server can then use TCP segmentation acceleration or full ToE to render the incoming network load insignificant to the processor.
I cannot get decent multicast rates with Broadcom hardware at all due to very poor Windows drivers. It would be interesting to see how Linux performs on the same hardware, that should give you a good indication of the hardware and IP stack quality.
You list Windows XP as working fine, the major difference between Windows Server and Windows XP is the quantum time. Windows Server gives longer quantum times, it might be worth investigating forcing a shorter quantum (if you can even set it).
Best Answer
Try iperf
An article that explains the different steps: http://taosecurity.blogspot.com/2006/09/generating-multicast-traffic.html