Linux – NetworkManager bridge is not working

bridgelinuxnetworkmanager

I face the same problem on 2 difference machines: A CentOS 8.1 and an openSUSE Leap 15.2
I want to create a bridge on each host that will be used from libvirtfor bridged networking.
On both servers, I use NetworkManager and I created the bridges as:

nmcli connection add type bridge bridge.stp no autoconnect yes con-name suse_virt ifname peter_virt ipv4.addresses 192.168.0.3/24 ipv4.gateway 192.168.0.254 ipv4.dns "192.168.0.1,8.8.8.8"  ipv4.method manual
nmcli connection add type bridge-slave master suse_virt autoconnect yes ifname eth0 con-name suse_virt-slave

The only differences between the CentOS and the openSUSE are the connection and interface names as well as the IPs.

When I start the bridge with nmcli con up suse_virt, I can see this:

ip a 

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master peter_virt state UP group default qlen 1000
    link/ether d0:50:99:17:3f:e6 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0a:f7:09:a2:1d brd ff:ff:ff:ff:ff:ff
7: suse_virt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d0:50:99:17:3f:e6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.3/24 brd 192.168.0.255 scope global noprefixroute peter_virt
       valid_lft forever preferred_lft forever
    inet6 fe80::af0c:23de:5e0c:ded8/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Which looks fine, but, only a few seconds later, my whole network is not working! I was suspecting Spanning Tree, but STP is disabled and my switch is an un-managed one. The only solution I have is to stop the bridge and enable the simple connection on each machine.

Restarting Network manager results to even stranger issues like the following:

ping 192.168.0.6
PING 192.168.0.6 (192.168.0.6) 56(84) bytes of data.
64 bytes from 192.168.0.6: icmp_seq=1 ttl=64 time=1.84 ms
^C
--- 192.168.0.6 ping statistics ---
5 packets transmitted, 1 received, 80% packet loss, time 4039ms
rtt min/avg/max/mdev = 1.846/1.846/1.846/0.000 ms

ping -c4 192.168.0.254
PING 192.168.0.254 (192.168.0.254) 56(84) bytes of data.
From 192.168.0.3 icmp_seq=1 Destination Host Unreachable
From 192.168.0.3 icmp_seq=2 Destination Host Unreachable
From 192.168.0.3 icmp_seq=3 Destination Host Unreachable
From 192.168.0.3 icmp_seq=4 Destination Host Unreachable

--- 192.168.0.254 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3071ms

The configuration as reported from the NM is:

nmcli con show suse_virt

connection.id:                          suse_virt
connection.uuid:                        45bc9ba8-8120-4bc5-93f4-168f28687f88
connection.stable-id:                   --
connection.type:                        bridge
connection.interface-name:              suse_virt
connection.autoconnect:                 yes
connection.autoconnect-priority:        0
connection.autoconnect-retries:         -1 (default)
connection.multi-connect:               0 (default)
connection.auth-retries:                -1
connection.timestamp:                   1600532118
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
connection.mdns:                        -1 (default)
connection.llmnr:                       -1 (default)
connection.wait-device-timeout:         -1
ipv4.method:                            manual
ipv4.dns:                               192.168.0.1,8.8.8.8
ipv4.dns-search:                        --
ipv4.dns-options:                       --
ipv4.dns-priority:                      0
ipv4.addresses:                         192.168.0.3/24
ipv4.gateway:                           192.168.0.254
ipv4.routes:                            --
ipv4.route-metric:                      -1
ipv4.route-table:                       0 (unspec)
ipv4.routing-rules:                     --
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-iaid:                         --
ipv4.dhcp-timeout:                      0 (default)
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.dhcp-fqdn:                         --
ipv4.dhcp-hostname-flags:               0x0 (none)
ipv4.never-default:                     no
ipv4.may-fail:                          yes
ipv4.dad-timeout:                       -1 (default)
ipv6.method:                            auto
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       --
ipv6.dns-priority:                      0
ipv6.addresses:                         --
ipv6.gateway:                           --
ipv6.routes:                            --
ipv6.route-metric:                      -1
ipv6.route-table:                       0 (unspec)
ipv6.routing-rules:                     --
ipv6.ignore-auto-routes:                no
ipv6.ignore-auto-dns:                   no
ipv6.never-default:                     no
ipv6.may-fail:                          yes
ipv6.ip6-privacy:                       -1 (unknown)
ipv6.addr-gen-mode:                     stable-privacy
ipv6.ra-timeout:                        0 (default)
ipv6.dhcp-duid:                         --
ipv6.dhcp-iaid:                         --
ipv6.dhcp-timeout:                      0 (default)
ipv6.dhcp-send-hostname:                yes
ipv6.dhcp-hostname:                     --
ipv6.dhcp-hostname-flags:               0x0 (none)
ipv6.token:                             --
bridge.mac-address:                     D0:50:99:17:3F:E6
bridge.stp:                             no
bridge.priority:                        128
bridge.forward-delay:                   15
bridge.hello-time:                      2
bridge.max-age:                         20
bridge.ageing-time:                     300
bridge.group-forward-mask:              0
bridge.multicast-snooping:              yes
bridge.vlan-filtering:                  no
bridge.vlan-default-pvid:               1
bridge.vlans:                           --
proxy.method:                           none
proxy.browser-only:                     no
proxy.pac-url:                          --
proxy.pac-script:                       --
GENERAL.NAME:                           suse_virt
GENERAL.UUID:                           45bc9ba8-8120-4bc5-93f4-168f28687f88
GENERAL.DEVICES:                        suse_virt
GENERAL.IP-IFACE:                       suse_virt
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        yes
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/3
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/7
bridge.priority:                        128
bridge.forward-delay:                   15
bridge.hello-time:                      2
bridge.max-age:                         20
bridge.ageing-time:                     300
bridge.group-forward-mask:              0
bridge.multicast-snooping:              yes
bridge.vlan-filtering:                  no
bridge.vlan-default-pvid:               1
bridge.vlans:                           --
proxy.method:                           none
proxy.browser-only:                     no
proxy.pac-url:                          --
proxy.pac-script:                       --
GENERAL.NAME:                           suse_virt
GENERAL.UUID:                           45bc9ba8-8120-4bc5-93f4-168f28687f88
GENERAL.DEVICES:                        suse_virt
GENERAL.IP-IFACE:                       suse_virt
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        yes
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/3
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/7
GENERAL.ZONE:                           --
GENERAL.MASTER-PATH:                    --
IP4.ADDRESS[1]:                         192.168.0.3/24
IP4.GATEWAY:                            192.168.0.254
IP4.ROUTE[1]:                           dst = 192.168.0.0/24, nh = 0.0.0.0, mt = 425
IP4.ROUTE[2]:                           dst = 0.0.0.0/0, nh = 192.168.0.254, mt = 20425
IP4.DNS[1]:                             192.168.0.1
IP4.DNS[2]:                             8.8.8.8

nmcli con show suse_virt-slave

connection.id:                          suse_virt-slave
connection.uuid:                        8a6be03b-debc-472d-a44d-eac7145b6ae0
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              eth0
connection.autoconnect:                 yes
connection.autoconnect-priority:        0
connection.autoconnect-retries:         -1 (default)
connection.multi-connect:               0 (default)
connection.auth-retries:                -1
connection.timestamp:                   1600532164
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --
connection.master:                      suse_virt
connection.slave-type:                  bridge
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
connection.mdns:                        -1 (default)
connection.llmnr:                       -1 (default)
connection.wait-device-timeout:         -1
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          no
802-3-ethernet.mac-address:             --
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.generate-mac-address-mask:--
802-3-ethernet.mac-address-blacklist:   --
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        --
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            --
802-3-ethernet.wake-on-lan:             default
802-3-ethernet.wake-on-lan-password:    --
bridge-port.priority:                   32
bridge-port.path-cost:                  100
bridge-port.hairpin-mode:               yes
bridge-port.vlans:                      --
GENERAL.NAME:                           suse_virt-slave
GENERAL.UUID:                           8a6be03b-debc-472d-a44d-eac7145b6ae0
GENERAL.DEVICES:                        eth0
GENERAL.IP-IFACE:                       eth0
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        no
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/4
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/2
GENERAL.ZONE:                           --
GENERAL.MASTER-PATH:                    /org/freedesktop/NetworkManager/Devices/5
IP4.GATEWAY:                            --
IP6.GATEWAY:                            --

ip route:

default via 192.168.0.254 dev peter_virt proto static metric 425 
192.168.0.0/24 dev peter_virt proto kernel scope link src 192.168.0.3 metric 425

ip -ts -4 monitor

[2020-09-20T07:17:55.876194] 192.168.0.1 dev suse_virt lladdr a8:a1:59:00:35:88 STALE
       valid_lft forever preferred_lft forever
[2020-09-20T07:18:01.252200] 192.168.0.1 dev suse_virt lladdr a8:a1:59:00:35:88 PROBE
[2020-09-20T07:18:04.324192] 192.168.0.1 dev suse_virt  FAILED
       valid_lft forever preferred_lft forever
[2020-09-20T07:18:06.180909] 192.168.0.1 dev suse_virt lladdr a8:a1:59:00:35:88 REACHABLE
       valid_lft forever preferred_lft forever
       valid_lft forever preferred_lft forever
       valid_lft forever preferred_lft forever
       valid_lft forever preferred_lft forever
       valid_lft forever preferred_lft forever
[2020-09-20T07:18:45.032169] 192.168.0.1 dev suse_virt lladdr a8:a1:59:00:35:88 STALE
       valid_lft forever preferred_lft forever
       valid_lft forever preferred_lft forever
       valid_lft forever preferred_lft forever
       valid_lft forever preferred_lft forever
[2020-09-20T07:18:56.292191] 192.168.0.1 dev suse_virt lladdr a8:a1:59:00:35:88 PROBE
       valid_lft forever preferred_lft forever
[2020-09-20T07:18:59.364174] 192.168.0.1 dev suse_virt  FAILED
[2020-09-20T07:19:01.224802] 192.168.0.1 dev suse_virt lladdr a8:a1:59:00:35:88 REACHABLE
[2020-09-20T07:19:07.556221] 192.168.0.254 dev suse_virt lladdr 04:bf:6d:7f:35:14 STALE
       valid_lft forever preferred_lft forever

Best Answer

Would you have by any chance two connections on eth0? You can check with nmcli con show to see the list of connections. The only connection on eth0 should be your suse_virt-slave connection.

How to fix all of your crontab related woes/problems (Linux)

This is a community wiki, if you notice anything incorrect with this answer or have additional information then please edit it.

First, basic terminology:

cron(8) is the daemon that executes scheduled commands.
crontab(1) is the program used to modify user crontab(5) files.
crontab(5) is a per user file that contains instructions for cron(8).

Next, education about cron:

Every user on a system may have their own crontab file. The location of the root and user crontab files are system dependant but they are generally below /var/spool/cron.

There is a system-wide /etc/crontab file, the /etc/cron.d directory may contain crontab fragments which are also read and actioned by cron. Some Linux distributions (eg, Red Hat) also have /etc/cron.{hourly,daily,weekly,monthly} which are directories, scripts inside which will be executed every hour/day/week/month, with root privilege.

root can always use the crontab command; regular users may or may not be granted access. When you edit the crontab file with the command crontab -e and save it, crond checks it for basic validity but does not guarantee your crontab file is correctly formed. There is a file called cron.deny which will specify which users cannot use cron. The cron.deny file location is system dependent and can be deleted which will allow all users to use cron.

If the computer is not powered on or crond daemon is not running, and the date/time for a command to run has passed, crond will not catchup and run past queries.

crontab particulars, how to formulate a command:

A crontab command is represented by a single line. You cannot use \ to extend a command over multiple lines. The hash (#) sign represents a comment which means anything on that line is ignored by cron. Leading whitespace and blank lines are ignored.

Be VERY careful when using the percent (%) sign in your command. Unless they are escaped \% they are converted into newlines and everything after the first non-escaped % is passed to your command on stdin.

There are two formats for crontab files:

User crontabs

 # Example of job definition:
 # .---------------- minute (0 - 59)
 # |  .------------- hour (0 - 23)
 # |  |  .---------- day of month (1 - 31)
 # |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
 # |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7)
 # |  |  |  |  |
 # *  *  *  *  *   command to be executed

System wide /etc/crontab and /etc/cron.d fragments

 # Example of job definition:
 # .---------------- minute (0 - 59)
 # |  .------------- hour (0 - 23)
 # |  |  .---------- day of month (1 - 31)
 # |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
 # |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7)
 # |  |  |  |  |
 # *  *  *  *  * user-name  command to be executed

Notice that the latter requires a user-name. The command will be run as the named user.

The first 5 fields of the line represent the time(s) when the command should be run. You can use numbers or where applicable day/month names in the time specification.

The fields are separated by spaces or tabs.
A comma (,) is used to specify a list e.g 1,4,6,8 which means run at 1,4,6,8.
Ranges are specified with a dash (-) and may be combined with lists e.g. 1-3,9-12 which means between 1 and 3 then between 9 and 12.
The / character can be used to introduce a step e.g. 2/5 which means starting at 2 then every 5 (2,7,12,17,22...). They do not wrap past the end.
An asterisk (*) in a field signifies the entire range for that field (e.g. 0-59 for the minute field).
Ranges and steps can be combined e.g. */2 signifies starting at the minimum for the relevant field then every 2 e.g. 0 for minutes( 0,2...58), 1 for months (1,3 ... 11) etc.

Debugging cron commands

Check the mail!

By default cron will mail any output from the command to the user it is running the command as. If there is no output there will be no mail. If you want cron to send mail to a different account then you can set the MAILTO environment variable in the crontab file e.g.

MAILTO=user@somehost.tld
1 2 * * * /path/to/your/command

Capture the output yourself

You can redirect stdout and stderr to a file. The exact syntax for capturing output may vary depending on what shell cron is using. Here are two examples which save all output to a file at /tmp/mycommand.log:

1 2 * * * /path/to/your/command &>/tmp/mycommand.log
1 2 * * * /path/to/your/command >/tmp/mycommand.log 2>&1

Look at the logs

Cron logs its actions via syslog, which (depending on your setup) often go to /var/log/cron or /var/log/syslog.

If required you can filter the cron statements with e.g.

grep CRON /var/log/syslog

Now that we've gone over the basics of cron, where the files are and how to use them let's look at some common problems.

Check that cron is running

If cron isn't running then your commands won't be scheduled ...

ps -ef | grep cron | grep -v grep

should get you something like

root    1224   1  0 Nov16 ?    00:00:03 cron

root    2018   1  0 Nov14 ?    00:00:06 crond

If not restart it

/sbin/service cron start

/sbin/service crond start

There may be other methods; use what your distro provides.

cron runs your command in a restricted environment.

What environment variables are available is likely to be very limited. Typically, you'll only get a few variables defined, such as $LOGNAME, $HOME, and $PATH.

Of particular note is the PATH is restricted to /bin:/usr/bin. The vast majority of "my cron script doesn't work" problems are caused by this restrictive path. If your command is in a different location you can solve this in a couple of ways:

Provide the full path to your command.
```
 1 2 * * * /path/to/your/command
```

Provide a suitable PATH in the crontab file

 PATH=/bin:/usr/bin:/path/to/something/else
 1 2 * * * command

If your command requires other environment variables you can define them in the crontab file too.

cron runs your command with cwd == $HOME

Regardless of where the program you execute resides on the filesystem, the current working directory of the program when cron runs it will be the user's home directory. If you access files in your program, you'll need to take this into account if you use relative paths, or (preferably) just use fully-qualified paths everywhere, and save everyone a whole lot of confusion.

The last command in my crontab doesn't run

Cron generally requires that commands are terminated with a new line. Edit your crontab; go to the end of the line which contains the last command and insert a new line (press enter).

Check the crontab format

You can't use a user crontab formatted crontab for /etc/crontab or the fragments in /etc/cron.d and vice versa. A user formatted crontab does not include a username in the 6th position of a row, while a system formatted crontab includes the username and runs the command as that user.

I put a file in /etc/cron.{hourly,daily,weekly,monthly} and it doesn't run

Check that the filename doesn't have an extension see run-parts
Ensure the file has execute permissions.
Tell the system what to use when executing your script (eg. put #!/bin/sh at top)

Cron date related bugs

If your date is recently changed by a user or system update, timezone or other, then crontab will start behaving erratically and exhibit bizarre bugs, sometimes working, sometimes not. This is crontab's attempt to try to "do what you want" when the time changes out from underneath it. The "minute" field will become ineffective after the hour is changed. In this scenario, only asterisks would be accepted. Restart cron and try it again without connecting to the internet (so the date doesn't have a chance to reset to one of the time servers).

Percent signs, again

To emphasise the advice about percent signs, here's an example of what cron does with them:

# cron entry
* * * * * cat >$HOME/cron.out%foo%bar%baz

will create the ~/cron.out file containing the 3 lines

foo
bar
baz

This is particularly intrusive when using the date command. Be sure to escape the percent signs

* * * * * /path/to/command --day "$(date "+\%Y\%m\%d")"

How to use `sudo` in cron jobs

when running as a non-root user,

crontab -e

will open the user's crontab, while

sudo crontab -e

will open the root user's crontab. It's not recommended to run sudo commands in a cron job, so if you're trying to run a sudo command in a user's cron, try moving that command to root's cron and remove sudo from the command.