Linux – Fixing umount Bind Source and Disk Space Issues

binddisk-space-utilizationlinuxmount

I think I may have shot myself in the foot. Long ago I tried to backup a old linux machine to a nas using the mount bind command:

mount –bind / /mnt/src
tar -C /mnt/src -c . > /mnt/backup_to_nas/full-backup-date '+%d-%B-%Y'.tar.gz –exclude=tmp –exclude=mnt

Then I realized I never umounted /mnt/src

My question is, is this taking up double the space on the / that I have? I am woefully out of space and not sure if I am chasing my tail trying to delete files to recover space.

df -h shows:

[root@web-server mnt]# df -h
Filesystem                        Size  Used Avail Use% Mounted on
devtmpfs                          1.9G     0  1.9G   0% /dev
tmpfs                             1.9G  4.0K  1.9G   1% /dev/shm
tmpfs                             1.9G  194M  1.7G  11% /run
tmpfs                             1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vda2                         7.6G  7.6G     0 100% /
/dev/vda1                         190M  171M  5.3M  98% /boot
/dev/vdb                          230G  152G   67G  70% /usr/local
tmpfs                             379M     0  379M   0% /run/user/0
tmpfs                             379M     0  379M   0% /run/user/2527
tmpfs                             379M     0  379M   0% /run/user/2543
10.50.1.104:/data                 9.1T  8.0T  610G  94% /mnt/backup
tmpfs                             379M     0  379M   0% /run/user/2539
10.75.0.199://volume1/ICCBackups   32T  4.2T   28T  14% /mnt/backup_to_nas
tmpfs                             379M     0  379M   0% /run/user/500

lsblk shows:

[root@web-server mnt]# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
vda    253:0    0     8G  0 disk 
├─vda1 253:1    0   200M  0 part /boot
└─vda2 253:2    0   7.8G  0 part /mnt/src
vdb    253:16   0 232.8G  0 disk /usr/local

did a space check in root:

du -xhs * | sort -rh

134G    home
15G data
6.9G    mnt
186M    run
170M    boot
52M etc
848K    ARC-History.pdf
16K lost+found
8.0K    export
8.0K    backup
4.0K    media
4.0K    check_permissions.py
0   sys
0   proc
0   lib64
0   lib
0   dev
0   bin

I don't understand why some large folders like /home are bigger than what df -h reports so I did a mount | grep home and got:

[root@web-server mnt]# mount | grep home
/dev/vdb on /home type ext4 (rw,relatime,data=ordered)
/dev/vda2 on /home/weather/public_html/weather_rrd type ext4 (rw,relatime,data=ordered)
/dev/vda2 on /usr/local/home/weather/public_html/weather_rrd type ext4 (rw,relatime,data=ordered)
/dev/vdb on /home/workers/public_html/VM-SYSTEMS type ext4 (rw,relatime,data=ordered)
/dev/vdb on /usr/local/home/workers/public_html/VM-SYSTEMS type ext4 (rw,relatime,data=ordered)

It looks like if I could figure out what these are and how to relocate I might buy myself some breathing room:

/dev/vda2 on /home/weather/public_html/weather_rrd type ext4 (rw,relatime,data=ordered)
/dev/vda2 on /usr/local/home/weather/public_html/weather_rrd type ext4 (rw,relatime,data=ordered)

Then again all this to say, did my mount –bind command of / into /mnt/src take up double space? Is there something I didn't understand (probably) when doing this? I did a lsof /mnt/src and it seems that everything is using it.

but did start with this to report not sure if its relevant:

[root@web-server mnt]# lsof /mnt/src
lsof: WARNING: can't stat() ext4 file system /var/www/html/net-status/bw-mon (deleted)
      Output information may be incomplete.
lsof: WARNING: can't stat() ext4 file system /usr/local/www/net-status/bw-mon (deleted)
      Output information may be incomplete.

So not sure where to start deleting large files as I find them in /? (even though it seems /home is somewhere else?). ls -lh doesn't show it as a symlink.

[root@web-server /]# ls -lh
total 1.2M
-rw-------    1 root root 847K Jul 27  2020 ARC-History.pdf
drwxr-xr-x    3 root root 4.0K Sep 13 17:21 backup
lrwxrwxrwx    1 root root    7 May 11  2018 bin -> usr/bin
dr-xr-xr-x.   5 root root 3.0K Aug 16 18:06 boot
-rw-r--r--    1 root root 3.3K May 12  2019 check_permissions.py
-rw-------    1 root root    0 Sep 13 17:33 core.20448
-rw-------    1 root root    0 Sep 13 17:46 core.28055
drwxr-xr-x    7 root root 4.0K Sep 13 17:16 data
drwxr-xr-x   18 root root 3.1K Sep 12 18:08 dev
drwxr-xr-x. 148 root root  12K Aug 31 17:26 etc
drwxr-xr-x    3 root root 4.0K Apr  2  2015 export
drwxr-xr-x   22 root root 4.0K Aug 27 18:06 home
lrwxrwxrwx    1 root root    7 May 11  2018 lib -> usr/lib
lrwxrwxrwx    1 root root    9 May 11  2018 lib64 -> usr/lib64
drwx------.   2 root root  16K Mar 31  2015 lost+found
drwxr-xr-x.   2 root root 4.0K Apr 12  2018 media
drwxr-xr-x.   6 root root 4.0K Feb  9  2023 mnt
drwxr-xr-x.   7 root root 4.0K Aug 25 15:13 opt
dr-xr-xr-x  225 root root    0 Nov  8  2021 proc
dr-xr-x---.  26 root root  12K Aug 25 15:19 root
drwxr-xr-x   48 root root 1.5K Sep 13 17:01 run
lrwxrwxrwx    1 root root    8 May 11  2018 sbin -> usr/sbin
-rw-r--r--    1 root root    0 May 15  2019 searchresults.txt
drwxr-xr-x.   2 root root 4.0K Apr 12  2018 srv
dr-xr-xr-x   13 root root    0 Nov 29  2021 sys
drwxrwxrwt.  16 root root 244K Sep 13 17:49 tmp
drwxr-xr-x.  14 root root 4.0K May 11  2018 usr
drwxr-xr-x.  25 root root 4.0K Aug  9 20:21 var
drwxr-xr-x    2 root root 4.0K Aug  9  2015 zaphod-data

EDIT:
Looked at the fstab file that shed some light onto where stuff is:

UUID=c9d6c99f-d7a5-4117-93ba-029cc34d8b61 /                       ext4    defaults        1 1
UUID=19fcad32-0fcb-423a-87e9-586d03d2e406 /boot                   ext4    defaults        1 2
#LABEL=/home    /home       ext4 defaults 1 2
#192.41.211.105:/export/images      /export/images          nfs     rsize=32768,wsize=32768,actimeo=0,bg,intr
LABEL=local-web-server  /usr/local  ext4    defaults    1 2
/usr/local/home     /home       none    bind        0 0
/usr/local/www      /var/www/html   none    bind        0 0
/usr/local/data     /data       none    bind        0 0
/tmp/rrdweather     /home/weather/public_html/weather_rrd   none    bind    0 0
/usr/local/data     /data       none    bind        0 0
/home/workers/Site/VM-SYSTEMS /home/workers/public_html/VM-SYSTEMS none bind 0 0
#/home/workers/public_html/WebCalendar-1.2.3 /home/workers/public_html/WebCalendar none bind 0 0
#/home/workers/public_html/WebCalendar-1.2.0 /home/workers/public_html/WebCalendar~ none bind 0 0
/home/workers/public_html/net-status /usr/local/www/net-status none bind 0 0
/tmp/bw-mon     /var/www/html/net-status/bw-mon     none    bind        0 0
/var/lib/smokeping/images /var/www/html/smokeping/images none   bind        0 0

#mounting for our cheezy backup of web-server
10.50.1.104:/data /mnt/backup nfs

Edit 2: So I realized if I did the sort command in /mnt/src I get more accurate information….

[root@web-server src]# du -xhs * | sort -rh
4.8G    usr
707M    var
421M    opt
397M    backup
168M    root
150M    tmp
52M etc
848K    ARC-History.pdf
16K lost+found
12K mnt
8.0K    export
4.0K    zaphod-data
4.0K    sys
4.0K    srv
4.0K    run
4.0K    proc
4.0K    media
4.0K    home
4.0K    dev
4.0K    data
4.0K    check_permissions.py
4.0K    boot
0   searchresults.txt
0   sbin
0   lib64
0   lib
0   core.28055
0   core.20448
0   bin

Shows me maybe space to clear in usr (no idea what I am not a linux guru), var had some nice stuff I was able to clear out (old logs). Still working on it but the crux I guess of what I am asking is should I really get that /mnt/src unmounted? or is it okay letting it ride like this since everytime I try to issue the command it says its busy.

Best Answer

The problem

/mnt/src is not taking space but you should unmount it to not confuse some tools, including tools doing backup. For example lsblk chooses to display the first matching mount and displays /mnt/src instead of / for /dev/vda2 which is not optimal.

Most Linux systems run systemd so I assume this system is running systemd.

systemd remounts / as shared mount instead of the Linux kernel's default of private. See the kernel documentation about mount propagation here: Shared Subtrees. That means that when later additional mounts are added (or removed and readded) over / they get also automatically mounted over /mnt/src by the propagation mechanism, and since it's shared (rather than slave) attempting to unmount them from /mnt/src will also unmount them from /, either failing (because in use by processes) or causing possible harm if succeeeding (they will also disappear from their original mountpoint).

What should have been done?

Always do such bind mounts using the private propagation option. While at it, when it's for backup purpose, also have the bind mount be (re)mounted read-only:

mount --bind --make-private -o ro / /mnt/src

That way no further interaction with other mounts will happen and this can be unmounted at any time with a simple umount /mnt/src without any side effect.

What can be done to fix the current problem?

So to do this properly without affecting the actual original mountpoints, one should set every problematic mounts as private so they can be unmounted without affecting the original mountpoint.

I make an example here (all as root user) mimicking OP's case where after having done:

# mkdir -p /mnt/src
# mount --bind -o ro / /mnt/src

And then later (in a place that can't be harmful):

# mkdir -p /root/intheway
# mount -t tmpfs tmpfs /root/intheway

leading then to:

# umount /mnt/src
umount: /mnt/src: target is busy.

  • identify problematic mounts using findmnt:

    # findmnt --kernel --submounts --list -o +PROPAGATION /mnt/src
TARGET                  SOURCE     FSTYPE  OPTIONS                        PROPAGATION
/mnt/src                /dev/vda2  ext4    ro,relatime,errors=remount-ro  shared
/mnt/src/root/intheway  tmpfs      tmpfs   rw,relatime,inode64            shared

  • Set all such found shared mounts as private, including /mnt/src:

    # mount --make-private /mnt/src/root/intheway
# mount --make-private /mnt/src

  • unmount them (in inverse order of the order they are mounted if there are multiple depths), finishing with /mnt/src

    # umount /mnt/src/root/intheway
# umount /mnt/src

This didn't disrupt the (shared) mount that was in the way:

# findmnt --kernel --list -o +PROPAGATION /root/intheway
TARGET         SOURCE FSTYPE OPTIONS             PROPAGATION
/root/intheway tmpfs  tmpfs  rw,relatime,inode64 shared
Related Topic