Linux – different uuid from blkid and tune2fs

I have solved my problem. The reason for not being able to access files over 2GB in size is due to my antivirus software (McAfee for Linux - Endpoint Security for Linux Threat Prevention). It was not due to apparmor, funky filesystem options, permissions, disk space or older filesystems.

Here is how I came to my conclusion. The troubleshooting steps might be interesting. Immediately following a reboot, I can quickly create a 2G file and probe it like so:

root@asci /tmp
# dd if=/dev/zero of=/tmp/file2049.iso bs=1M count=2049
2049+0 records in
2049+0 records out
2148532224 bytes (2.1 GB, 2.0 GiB) copied, 2.05888 s, 1.0 GB/s

root@asci /tmp
# md5sum file2049.iso
4555da35a7064cc499ba1e3f6fa1993a  file2049.iso

Within a minute, the md5sum no longer works.

To rule out crontabs, I disabled crontab, and also wrote a 1-liner script to output 0 upon successful read and 1 if unsuccessful. Notice that it breaks at the 40'th second (indicating most likely not a cron job):

# while [ 1 == 1 ]; do echo -n "`date` - ";dd if=asm8.iso bs=1M count=30 2>&1|grep -c "Operation not permi"; sleep 1;date >> /var/log/ps.log; ps -efww >> /var/log/ps.log; done
Fri Jan 19 18:57:28 CET 2018 - 0
Fri Jan 19 18:57:30 CET 2018 - 0
Fri Jan 19 18:57:31 CET 2018 - 0
Fri Jan 19 18:57:32 CET 2018 - 0
Fri Jan 19 18:57:33 CET 2018 - 0
Fri Jan 19 18:57:34 CET 2018 - 0
Fri Jan 19 18:57:35 CET 2018 - 0
Fri Jan 19 18:57:36 CET 2018 - 0
Fri Jan 19 18:57:37 CET 2018 - 0
Fri Jan 19 18:57:38 CET 2018 - 0
Fri Jan 19 18:57:39 CET 2018 - 0
Fri Jan 19 18:57:40 CET 2018 - 1
Fri Jan 19 18:57:41 CET 2018 - 1
Fri Jan 19 18:57:42 CET 2018 - 1
Fri Jan 19 18:57:43 CET 2018 - 1
Fri Jan 19 18:57:44 CET 2018 - 1

I only dd the start of the file, as that's all I need to test it, as it takes 48 seconds to read the entire thing. I logged the ps output to /var/log/ps.log. To do a diff on the ps output by comparing what changed between the 39th second and 40th second, I did this:

# cat ps.log |grep "Fri Jan 19 18:57:39 CET 2018" -A10000 |grep "Fri Jan 19 18:57:40 CET 2018" -B10000 > /tmp/ps39
# cat ps.log |grep "Fri Jan 19 18:57:40 CET 2018" -A10000 |grep "Fri Jan 19 18:57:41 CET 2018" -B10000 > /tmp/ps40
# cd /tmp

Comparing the ps outputs between the 39th and 40th second:

# diff ps39 ps40
 Fri Jan 19 18:57:40 CET 2018
266c266
 root      2412  1276 97 18:57 ?        00:00:19 /opt/isec/ens/threatprevention/bin/isectpd
275,276c275,278
 root      2632  2412  1 18:57 ?        00:00:00 /opt/isec/ens/threatprevention/bin/isectpd
> root      2635  2412 11 18:57 ?        00:00:00 /opt/isec/ens/threatprevention/bin/isectpd
> root      2663  2518  0 18:57 pts/1    00:00:00 ps -efww

So, that's it! Once the isectpd process started up at the 40th second, it disabled access to my 2GB file.

Once I did this:

# systemctl stop isectpd

It started working. Obviously, this is a bandaid until I find out how to allow > 2GB files from McAfee. If anyone has any experience with this, feel free to chime in.

Cheers.

In summary from the discussion in the comments:

Your problem was essentially that you used symlinks in your mount point paths and at boot the system was not able to follow those properly to recognise the result as "nested mounts". Therefor systemd did not mount your file-systems in a proper sequential order to handle that dependancy.

• You have a mount point /apps_home

• You have a symlink /apps --> /apps_home/apps

• And you have also try to mount volumes on /apps/var/progress /apps/var/custom and /apps/var/custom

The problem is that the mount points /apps/var/[custom|progress|standard] don't exist until /apps_home is mounted.

Solution:

Leave the symlink but mount your file-systems on the actual directory paths of the symlink target: i.e. convert your fstab entries to:

UUID=5717b613-a9f4-43c9-95d2-cfbbb891bd19   /apps_home                    ext4 defaults 1 2
UUID=e24df090-2dda-404c-8944-a28bd37d6c5e   /apps_home/apps/var/progress  ext4 defaults 1 2
UUID=5f254c77-a91d-4255-8315-9325ddb7a9d8   /apps_home/apps/var/standard  ext4 defaults 1 2
UUID=746c70c1-002a-4249-a06f-df393a99252c   /apps_home/apps/var/custom    ext4 defaults 1 2

systemd-fstab-generato will generate the needed mount unit files and systemd.mount will implicitly add the correct dependancies:

If a mount unit is beneath another mount unit in the file system hierarchy, both a requirement dependency and an ordering dependency between both units are created automatically.

Alternative: remove the entries from /etc/fstab and create your own mount unit files and manually configure the requirement and ordering dependancies to ensure that /apps/var/progress /apps/var/custom and /apps/var/custom don't get mounted before /apps_home.