Linux – script run through jenkins user gets killed

Short answer: you can't. Ports below 1024 can be opened only by root. As per comment - well, you can, using CAP_NET_BIND_SERVICE, but that approach, applied to java bin will make any java program to be run with this setting, which is undesirable, if not a security risk.

The long answer: you can redirect connections on port 80 to some other port you can open as normal user.

Run as root:

# iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

As loopback devices (like localhost) do not use the prerouting rules, if you need to use localhost, etc., add this rule as well (thanks @Francesco):

# iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-ports 8080

NOTE: The above solution is not well suited for multi-user systems, as any user can open port 8080 (or any other high port you decide to use), thus intercepting the traffic. (Credits to CesarB).

EDIT: as per comment question - to delete the above rule:

# iptables -t nat --line-numbers -n -L

This will output something like:

Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:8080 redir ports 8088
2    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80 redir ports 8080

The rule you are interested in is nr. 2, so to delete it:

# iptables -t nat -D PREROUTING 2

Orien is correct, it is the fork() system call triggered by ProcessBuilder or Runtime.exec or other means of the JVM executing an external process (e.g. another JVM running ant, a git command, etc.).

There have been some posts on the Jenkins mailing lists about this: Cannot run program "git" ... error=12, Cannot allocate memory

There is a nice description of the issue on the SCons dev list: fork()+exec() vs posix_spawn()

There is a long standing JVM bug report with solutions: Use posix_spawn, not fork, on S10 to avoid swap exhaustion. But I'm not sure if this actually made it into JDK7 as the comments suggest was the plan.

In summary, on Unix-like systems, when one process (e.g. the JVM) needs to launch another process (e.g. git) a system call is made to fork() which effectively duplicates the current process and all its memory (Linux and others optimize this with copy-on-write so the memory isn't actually copied until the child attempts to write to it). The duplicate process then makes another system call, exec() to launch the other process (e.g. git) at which point all that copied memory from the parent process may be discarded by the operating system. If the parent process is using large amounts of memory (as JVM processes tend to do), the call to fork() may fail if the operating system determines it does not have enough memory+swap to hold two copies, even if the child process will never actually use that copied memory.

There are several solutions:

• Add more physical memory/RAM to the machine.

• Add more swap space to trick the fork() into working, even though the swap space is not strictly needed for anything. This is the solution I chose because it's fairly easy to add a swapfile, and I did not want to live with the potential for processes being killed due to overcommit.

• On Linux, enable overcommit_memory option of the vm system (/proc/sys/vm/overcommit_memory). With overcommit, the call to fork() would always succeed, and since the child process isn't actually going to use that copy of the memory, all is well. Of course, it's possible that with overcommit, your processes will actually attempt to use more memory than is available and will be killed by the kernel. Whether this is appropriate depends on the other uses of the machine. Mission critical machines should probably not risk the out-of-memory killer running amok. But an internal development server that can afford some downtime would be a good place to enable overcommit.

• Change the JVM to not use fork()+exec() but to use posix_spawn() when available. This is the solution requested in the JVM bug report above and mentioned on the SCons mailing list. It is also implemented in java_posix_spawn.

  I'm trying to find out if that fix made it into JDK7. If not, I wonder if the Jenkins people would be interested in a work around such as java_posix_spawn. There seem to have been attempts to integrate that into Apache commons-exec.

  Programmieraffe, I'm not 100% sure, but your link does suggest that the fix is in JDK7 and JDK6 1.6.0_23 and later. For the record, I was running OpenJDK 1.6.0_18.

See https://stackoverflow.com/questions/1124771/how-to-solve-java-io-ioexception-error-12-cannot-allocate-memory-calling-run