Linux – Root Cause of ‘A Start Job is Running for Create Volatile Files and Directories’

Stuck at "A start job is running for Create Volatile Files and Directories" after reboot a server(Debian 9.5, 64bit), and solve by this "boot-stuck-at-a-start-job-is-running-for-create-volatile-files-and-directories".

I can't figure out what is the root cause of this issue, although search from many questions which are not refer the root cause but just the varied solutions that not meet me.

We have not reach the limit of file or (sub) directory, and set the dir_nlink for ext4.

# sudo tune2fs -l /dev/debian-vg/root | grep dir_nlink
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent
 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum

And the are more than 50% capacity of inode and disk.

The original /tmp directory only little file and directory, total disk space usage only 1G.

Some info:

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.9.0-7-amd64 root=/dev/mapper/debian--vg-root ro net.ifnames=0 biosdevname=0 console0=tty0 console=ttyS0,115200n8 quiet

$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=4077900k,nr_inodes=1019475,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=817924k,mode=755)
/dev/mapper/debian--vg-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9039)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=817920k,mode=700,uid=1000,gid=1000)

$ lsblk
NAME                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                 254:0    0 1000G  0 disk 
└─vda1              254:1    0 1000G  0 part 
  └─debian--vg-root 253:0    0    3T  0 lvm  /
vdb                 254:16   0    4T  0 disk 
vdc                 254:32   0    2T  0 disk 
└─debian--vg-root   253:0    0    3T  0 lvm  /

$ blkid
/dev/vda1: UUID="ijfyeQ-***" TYPE="LVM2_member" PARTUUID="d6***"
/dev/mapper/debian--vg-root: UUID="2d2294a9-***" TYPE="ext4"
/dev/vdc: UUID="PXrGC9-***" TYPE="LVM2_member"

$ sudo find /tmp/ | wc -l
28905144