Ubuntu – How to change stack depth limits in /etc/security/limits.d/ and have the changes apply to services at boot

ansiblepostgresqlUbuntuulimit

My system

  • Series of Ubuntu 14.04.5 (x86_64) Servers, kept updated
  • My application required I increase stack depth for postgres
  • I created a file in /etc/security/limits.d/myapplication.conf
  • The myapplication.conf file has the line: * - stack 131072
  • Note that 131072KB == 128MB
  • Upon making this myapplication.conf file, my ulimit -s returns: 131072
  • I then edited my /etc/postgresql/9.3/main/postgresql.conf file and appended the line: max_stack_depth = 126MB

My problem

During boot, the following message appears:

 * The PostgreSQL server failed to start. Please check the log output:
2018-01-24 09:27:53 MST LOG:  invalid value for parameter "max_stack_depth": 129024
2018-01-24 09:27:53 MST DETAIL:  "max_stack_depth" must not exceed 7680kB.
2018-01-24 09:27:53 MST HINT:  Increase the platform's stack depth limit via "ulimit -s" or local equivalent.
2018-01-24 09:27:53 MST FATAL:  configuration file "/etc/postgresql/9.3/main/postgresql.conf" contains errors
 * Starting Mount network filesystems                                    [ OK ]
 * Starting PostgreSQL 9.3 database server        * Stopping Mount networ[ OK ]systems
                                                                                             [fail]

This in turn causes my application service to then fail, as I depend on my databases. After boot, if I start the postgres service, it's fine:

dev@wipn:~$ sudo service postgresql start
 * Starting PostgreSQL 9.3 database server                                                                                                                                                                                                                                                                             [ OK ] 
dev@wipn:~$

My guess is that the effects of /etc/security/limits.d/myapplication.conf are only applied at a stage during boot that's after when my system tries to start postgres. So, perhaps an obvious solution is to just change when I start postgres, that's fine and I can handle that.

My Question

What's a way to change the kernel's stack depth, such that I need only make minimal alterations to my servers?

I want something as clean as possible. I want it to withstand upgrades and preferably be applicable to other distros. I manage my stuff through Ansible plays, so I'd rather write one clean play for this.

It may simply be the case that changing start order of my services is the best resolution. Anyone out there know of other suitable options?

Things I've tried

Here's a list of some of the things I've tried, without success.

In /etc/security/limits.d/myapplication.conf:

  • postgres - stack 131072
  • * - stack 131072
    root - stack 131072

Best Answer

Until someone can give me a clean solution, this is what I've come up with, and it sucks. I won't accept it as THE answer to my question, but here it is (gags). At least it works.


Background

Seems that changes to /etc/security/limits.* never impact services, but rather stuff being executed from the shell. So, that kind of makes my changes in /etc/security/limits.* quite meaningless. (insert cussing here). I've now deleted my /etc/security/limits.d/myapplication.conf.


Changing Stack Size limits for postgres

This is a garbage solution. I hate it.

I've edited my "/usr/share/postgresql-common/init.d-functions", specifically the start() function, to appear as:

...
# start all clusters of version $1
# output according to Debian Policy for init scripts
start() {
    ulimit -s 131072    #JTS: To avoid Issue #XYZ 

    # create socket directory
    if [ -d /var/run/postgresql ]; then
        chmod 2775 /var/run/postgresql
    else
    ...

Obviously I've added the ulimit line. It's disgusting to me to modify this file because I expect it to be perpetually changed by updates. At least I have an Ansible rule to enforce it exists.


My Ansible solution

Here's the Ansible task I've created to enforce this config change for me:

- blockinfile:
    dest: /usr/share/postgresql-common/init.d-functions
    block: |
          ulimit -s 131072
    backup: yes
    insertafter: '^start\(\) \{'
    state: present

This Ansible task results in the function looking like this:

...
# start all clusters of version $1
# output according to Debian Policy for init scripts
start() {
# BEGIN ANSIBLE MANAGED BLOCK
ulimit -s 131072
# END ANSIBLE MANAGED BLOCK
    # create socket directory
    if [ -d /var/run/postgresql ]; then
       ...


Of Note: Upstart Services Ignore /etc/security/limits

Seems that /etc/security/limits.* is ignorred by Upstart, which Ubuntu 14.04 uses. My application service actually uses upstart and one can insert a line for upstart that looks like:

limit stack <softlimit> <hardlimit>

Ubuntu switched to systemd after 14.04, so this upstart tid-bit will fade in to irrelevance.

This isn't relevant to my question because on 14.04, postgresql is not managed by upstart.

Related Topic