Cgroup fork rejected by pids controller

cgrouppamsystemdubuntu-20.04

Have a small server program written in C/C++ that uses nginx and Postgres, currently all are hosted on the same ubuntu system. I usually run the server program from the bash command line.

Recently on the newer versions of Ubuntu, when the server program is running, after about half a minute I am not able to execute any other commands; if the screen saves, then I cannot log back in. Terminating my server program resumes usual behaviour.

In bash: on ENTER for any command:

bash: fork: retry: Resource temporarily unavailable

And the following is written to /var/log/syslog:

Sep  5 09:46:08 ubuntu kernel: [  145.614883] cgroup: fork rejected by pids controller in /user.slice/user-1000.slice/user@1000.service

The experience is different when the server program is started after a system reboot as apposed to after the system has been operational for some time. Specifically after the reboot the sever program will run without issue. If the server program is restarted it will run at 60% utilisation for say half a minute, jump to 80% for about 5 seconds and then drop down to 20% utilisation thereafter. It almost seems like something is directly throttling the application. CGroups?

PAM Configuration:
Within /etc/pam.d/common-session has the lines:

session required    pam_unix.so 
session optional    pam_systemd.so 

I don't believe it makes a difference but in /etc/security/limits.d/91-nofile.conf I have set the following PAM settings:

*                soft    nofile          350000
*                hard    nofile          350000
*                soft    nproc           100000
*                hard    nproc           100000
*                soft    sigpending      100000
*                hard    sigpending      100000

CGroups / Systemd Configuration:

myk@ubuntu:/etc/systemd/system$ systemctl status user.slice 
● user.slice - User and Session Slice
     Loaded: loaded (/lib/systemd/system/user.slice; static; vendor preset: ena>
     Active: active since Sat 2020-09-05 10:47:19 +08; 38min ago
       Docs: man:systemd.special(7)
      Tasks: 1396
     Memory: 1.0G
     CGroup: /user.slice
             └─user-1000.slice
               ├─session-2.scope

myk@ubuntu:~$ systemctl status user-1000.slice 
user-1000.slice - User Slice of UID 1000
     Loaded: loaded
    Drop-In: /usr/lib/systemd/system/user-.slice.d
             └─10-defaults.conf
     Active: active since Sat 2020-09-05 09:44:50 +08; 21min ago
       Docs: man:user@.service(5)
      Tasks: 340 (limit: 15479)
     Memory: 1.6G

cat /proc/sys/kernel/threads-max
46907

cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max
15479

In /etc/systemd/system.conf tried adding:

DefaultMemoryAccounting=no
DefaultTasksAccounting=no

without success. This did result in 'systemctl status user-1000.slice' no longer displaying a Task limit

In /etc/systemd/logind.conf tried adding:

UserTasksMax=infinity

Without success

In /etc/systemd/system.conf
Changed:

#DefaultTasksMax=

To:

DefaultTasksMax=infinity

Without success

Ubuntu is running under VM-Ware hosted by MacOs on a MBP. Pmstat shows thermals on the MBC to be ok. Ubuntu 20.04; vmware 11.6.5; macOs 10.15.6

Question:
Is there some way to configure cgroups / pam / systemd / etc to be able to continue to use the command line when the server program is running / be able to log back in on screen save?

Best Answer

Solved this issue by changing /usr/lib/systemd/system/user-.slice.d/10-defaults.conf

Changed:

TasksMax=33%

to read:

TasksMax=infinity

Related Topic