In our instance, our problem was solved by sysctl parameters, one different from Maciej.
Please note that I do not speak for the OP (buecking), I came on this post due to the problem being related by the basic detail (no multicast traffic in userland).
We have an application that reads data sent to four multicast addresses, and a unique port per multicast address, from an appliance that is (usually) connected directly to an interface on the receiving server.
We were attempting to deploy this software on a customer site when it mysteriously failed with no known reason. Attempts at debugging this software resulted in inspecting every system call, ultimately they all told us the same thing:
Our software asks for data, and the OS never provides any.
The multicast packet counter incremented, tcpdump showed the traffic reaching the box/specific interface, yet we couldn't do anything with it. SELinux was disabled, iptables was running but had no rules in any of the tables.
Stumped, we were.
In randomly poking around, we started thinking about the kernel parameters that sysctl handles, but none of the documented features was either particularly relevant, or if they had to do with multicast traffic, they were enabled. Oh, and ifconfig did list "MULTICAST" in the feature line (up, broadcast, running, multicast). Out of curiosity we looked at /etc/sysctl.conf. 'lo and behold, this customer's base image had a couple of extra lines added to it at the bottom.
In our case, the customer had set net.ipv4.all.rp_filter = 1. rp_filter is the Route Path filter, which (as I understand it) rejects all traffic that could not have possibly reached this box. Network subnet hopping, the thought being that the source IP is being spoofed.
Well, this server was on a 192.168.1/24 subnet and the appliance's source IP address for the multicast traffic was somewhere in the 10.* network. Thus, the filter was preventing the server from doing anything meaningful with the traffic.
A couple of tweaks approved by the customer; net.ipv4.eth0.rp_filter = 1 and net.ipv4.eth1.rp_filter = 0 and we were running happily.
Simply put:
------------------------------------------------------------
| TYPE | ASSOCIATIONS | SCOPE | EXAMPLE |
------------------------------------------------------------
| Unicast | 1 to 1 | Whole network | HTTP |
------------------------------------------------------------
| Broadcast | 1 to Many | Subnet | ARP |
------------------------------------------------------------
| Multicast | One/Many to Many | Defined horizon | SLP |
------------------------------------------------------------
| Anycast | Many to Few | Whole network | 6to4 |
------------------------------------------------------------
Unicast is used when two network nodes need to talk to each other. This is pretty straight forward, so I'm not going to spend much time on it. TCP by definition is a Unicast protocol, except when there is Anycast involved (more on that below).
When you need to have more than two nodes see the traffic, you have options.
If all of the nodes are on the same subnet, then broadcast becomes a viable solution. All nodes on the subnet will see all traffic. There is no TCP-like connection state maintained. Broadcast is a layer 2 feature in the Ethernet protocol, and also a layer 3 feature in IPv4.
Multicast is like a broadcast that can cross subnets, but unlike broadcast does not touch all nodes. Nodes have to subscribe to a multicast group to receive information. Multicast protocols are usually UDP protocols, since by definition no connection-state can be maintained. Nodes transmitting data to a multicast group do not know what nodes are receiving. By default, Internet routers do not pass Multicast traffic. For internal use, though, it is perfectly allowed; thus, "Defined horizon" in the above chart. Multicast is a layer 3 feature of IPv4 & IPv6.
To use anycast you advertise the same network in multiple spots of the Internet, and rely on shortest-path calculations to funnel clients to your multiple locations. As far the network nodes themselves are concerned, they're using a unicast connection to talk to your anycasted nodes. For more on Anycast, try: What is "anycast" and how is it helpful?. Anycast is also a layer 3 feature, but is a function of how route-coalescing happens.
Examples
Some examples of how the non-Unicast methods are used in the real Internet.
Broadcast
ARP is a broadcast protocol, and is used by TCP/IP stacks to determine how to send traffic to other nodes on the network. If the destination is on the same subnet, ARP is used to figure out the MAC address that goes to the stated IP address. This is a Level 2 (Ethernet) broadcast, to the reserved FF:FF:FF:FF:FF:FF MAC address.
Also, Microsoft's machine browsing protocol is famously broadcast based. Work-arounds like WINS were created to allow cross-subnet browsing. This involves a Level 3 (IP) broadcast, which is an IP packet with the Destination address listed as the broadcast address of the subnet (in 192.168.101.0/24, the broadcast address would be 192.168.101.255).
The NTP protocol allows a broadcast method for announcing time sources.
Multicast
Inside a corporate network, Multicast can deliver live video to multiple nodes without having to have massive bandwidth on the part of the server delivering the video feed. This way you can have a video server feeding a 720p stream on only a 100Mb connection, and yet still serve that feed to 3000 clients.
When Novell moved away from IPX and to IP, they had to pick a service-advertising protocol to replace the SAP protocol in IPX. In IPX, the Service Advertising Protocol, did a network-wide announcement every time it announced a service was available. As TCP/IP lacked such a global announcement protocol, Novell chose to use a Multicast based protocol instead: the Service Location Protocol. New servers announce their services on the SLP multicast group. Clients looking for specific types of services announce their need to the multicast group and listen for unicasted replies.
HP printers announce their presence on a multicast group by default. With the right tools, it makes it real easy to learn what printers are available on your network.
The NTP protocol also allows a multicast method (IP 224.0.1.1) for announcing time sources to areas beyond just the one subnet.
Anycast
Anycast is a bit special since Unicast layers on top of it. Anycast is announcing the same network in different parts of the network, in order to decrease the network hops needed to get to that network.
The 6to4 IPv6 transition protocol uses Anycast. 6to4 gateways announce their presence on a specific IP, 192.88.99.1. Clients looking to use a 6to4 gateway send traffic to 192.88.99.1 and trust the network to deliver the connection request to a 6to4 router.
NTP services for especially popular NTP hosts may very well be anycasted, but I don't have proof of this. There is nothing in the protocol to prevent it.
Other services use Anycast to improve data locality to end users. Google does Anycast with its search pages in some places (and geo-IP in others). The Root DNS servers use Anycast for similar reasons. ServerFault itself just might go there, they do have datacenters in New York and Oregon, but hasn't gone there yet.
Network concerns
Excessive broadcast traffic can rob all nodes in that subnet of bandwidth. This is less of a concern these days with full-duplex GigE ports, but back in the half-duplex 10Mb days a broadcast storm could bring a network to a halt real fast. Those half-duplex networks with one big collision domain across all nodes were especially vulnerable to broadcast storms, which is why networking books, especially older ones, say to keep an eye on broadcast traffic. Switched/Full-Duplex networks are a lot harder to bring to a halt with a broadcast storm, but it can still happen. Broadcast is required for correct functioning of IP networks.
Multicast has the same possibility for abuse. If one node on the multicast group starts sending huge amounts of traffic to that group, all subscribed nodes will see all of that traffic. As with broadcast, excessive Mcast traffic can increase the possibilities of collisions on such connections where that is a problem.
Multicast is an optional feature with IPv4, but required for IPv6. The IPv4 broadcast is replaced by multicast in IPv6 (See also: Why can't IPv6 send broadcasts?). It is frequently turned off on IPv4 networks. Not coincidentally, enabling multicast is one of the many reasons network-engineers are leery of moving to IPv6 before they have to do it.
Calculating how much traffic is too much traffic depends on a few things
- Half vs Full Duplex: Half-duplex networks have much lower tolerances for bcast/mcast traffic.
- Speed of network ports: The faster your network, the less of an issue this becomes. In the 10Mb ethernet days 5-10% of traffic on a port could be bcast traffic, if not more, but on GigE less than 1% (probably way less) is more likely.
- Number of nodes on the network: The more nodes you have, the more unavoidable broadcast traffic you'll incur (ARP). If you have broadcast specific protocols in use, Windows browsing or other things like cluster heartbeats, where problems start will change.
- Network technology: Wired Ethernet is fast enough that so long as you have modern gear driving it, bcast/mcast isn't likely to cause you problems. Wireless, on the other hand, can suffer from excessive broadcast traffic as it is a shared medium amongst all nodes and therefore in a single collision domain.
In the end, Bcast and Mcast traffic rob ports of bandwidth off the top. When you start to worry is highly dependent on your individual network and tolerance for variable performance. In general, network-node counts haven't scaled as fast as network speeds so the overall broadcast percentage-as-traffic number has been dropping over time.
Some networks disallow Multicast for specific reasons, and others have never taken the time to set it up. There are some multicast protocols that can reveal interesting information (SLP is one such) to anyone listening for the right things. Personally, I don't mind minor multicast traffic as the biggest annoyance I've seen with it is polluted network captures when I'm doing some network analysis; and for that there are filters.
Best Answer
Check whether the switch is using flow control. It'll issue pause frames from the source port when any linked port isn't able to keep up with the multicast rate, ie. is linked at 100 or 10 Mbit/s.
You might want to disable flow control for the test. I generally disable it since it's rarely useful and may cause head-of-line blocking, as in your case. TCP does a much better job and there are few cases where Ethernet flow control makes sense (e.g. in a properly designed iSCSI SAN).