Linux – Creating more than 20 VXLAN interfaces on linux

interfacelinuxnetworking

How can I create more than 20 vxlan interfaces in Ubuntu/Debian?

  • In Linux kernel 3.16 I found that I can create more than 20 vxlan interfaces but they will not work properly as in sending arp messages does not work.

  • in Linux Kernel 4.4 I get this error message "RTNETLINK answers: No buffer space available" when creating the 21'st interface

I've tested this by creating a small shell script that creates the interfaces on a fresh ubuntu 14.04 & 16.04, also tested this on Debian 8.

The script testvxlan.sh looks like this:

#!/bin/bash

for i in {1..30}
do
    echo "Setting up interface br0.$i"
    ip link add br0.$i type vxlan id $i group 239.0.0.$i dev eth0 dstport 4789
    ip addr add 192.168.$i.1/24 dev br0.$i
    ip link set dev br0.$i up
    #ip link delete br0.$i
done

when running this on a fresh Ubuntu 16.04 it looks like this:

root@ubuntu-xenial:~# ./testvxlan.sh
Setting up interface br0.1
Setting up interface br0.2
Setting up interface br0.3
Setting up interface br0.4
Setting up interface br0.5
Setting up interface br0.6
Setting up interface br0.7
Setting up interface br0.8
Setting up interface br0.9
Setting up interface br0.10
Setting up interface br0.11
Setting up interface br0.12
Setting up interface br0.13
Setting up interface br0.14
Setting up interface br0.15
Setting up interface br0.16
Setting up interface br0.17
Setting up interface br0.18
Setting up interface br0.19
Setting up interface br0.20
Setting up interface br0.21
RTNETLINK answers: No buffer space available
Setting up interface br0.22
RTNETLINK answers: No buffer space available
Setting up interface br0.23
RTNETLINK answers: No buffer space available
Setting up interface br0.24
RTNETLINK answers: No buffer space available
Setting up interface br0.25
RTNETLINK answers: No buffer space available
Setting up interface br0.26
RTNETLINK answers: No buffer space available
Setting up interface br0.27
RTNETLINK answers: No buffer space available
Setting up interface br0.28
RTNETLINK answers: No buffer space available
Setting up interface br0.29
RTNETLINK answers: No buffer space available
Setting up interface br0.30
RTNETLINK answers: No buffer space available

How can I increase this bufferspace or is it even possible?

Best Answer

Since you use multicast vxlan's, the limit is actually the maximum number of IGMP memberships:

[root@cpu1 ~]# cat /proc/sys/net/ipv4/igmp_max_memberships 
20

You can raise this limit and should be able to bring up more than 20 vxlans:

[root@cpu1 ~]# echo 100 >/proc/sys/net/ipv4/igmp_max_memberships

If you want this change to be persistent across reboots, you do need to add the following snippet to your /etc/sysctl.conf or /etc/sysctl.d/:

net.ipv4.igmp_max_memberships = 100
Related Topic