Java – Jenkins CI – Cannot allocate memory

javaJenkins

I tested jenkins-ci successfully on a ubuntu 10.4 (with vmware fusion) on my local computer. Now I want to install and use it on my virtual server at hosteurope. The basic installation was no problem, but now I have problems with my build project.

After pulling an mercurial update from a repository, ant is invoked and throws the following error in my build project:

"Buildfile: /var/lib/jenkins/workspace/concrete5-seed-clean/build.xml
[property] java.io.IOException: Cannot run program "/usr/bin/env": java.io.IOException: error=12, Cannot allocate memory"

There is a known problem with heap size at virtual servers at hosteurope (http://faq.hosteurope.de/index.php?cpid=13918), so I tried to set the heap size manually:

# for ant
export ANT_OPTS="-Xms512m -Xmx512m"

# jenkins
# edited /etc/default/jenkins, added line 
JAVA_ARGS="-Xms512m -Xmx512m"
# restarted jenkins via /etc/init.d/jenkins restart

After setting this for ant, the command "ant -diagnostics" runs through and does not cause an error, but the error still occurs when I try to build the project.

Server-Details:
– http://www.hosteurope.de/produkt/Virtual-Server-Linux-L

  • Ubuntu 10.4 LTS
  • RAM: 1GB / Dynamic 2GB

My questions:
– Is 1GB enough for Jenkins or do I have to upgrade the server?
– Is this error caused by ant or jenkins?

Update: I got it running with ant options -Xmx128m -Xms128m, but sometimes the error occurs again. (this freaks me out, cause i can not reproduce it by now :/ )

Help much appreciated!

Cheers,
Matthias

Best Answer

Orien is correct, it is the fork() system call triggered by ProcessBuilder or Runtime.exec or other means of the JVM executing an external process (e.g. another JVM running ant, a git command, etc.).

There have been some posts on the Jenkins mailing lists about this: Cannot run program "git" ... error=12, Cannot allocate memory

There is a nice description of the issue on the SCons dev list: fork()+exec() vs posix_spawn()

There is a long standing JVM bug report with solutions: Use posix_spawn, not fork, on S10 to avoid swap exhaustion. But I'm not sure if this actually made it into JDK7 as the comments suggest was the plan.

In summary, on Unix-like systems, when one process (e.g. the JVM) needs to launch another process (e.g. git) a system call is made to fork() which effectively duplicates the current process and all its memory (Linux and others optimize this with copy-on-write so the memory isn't actually copied until the child attempts to write to it). The duplicate process then makes another system call, exec() to launch the other process (e.g. git) at which point all that copied memory from the parent process may be discarded by the operating system. If the parent process is using large amounts of memory (as JVM processes tend to do), the call to fork() may fail if the operating system determines it does not have enough memory+swap to hold two copies, even if the child process will never actually use that copied memory.

There are several solutions:

  • Add more physical memory/RAM to the machine.

  • Add more swap space to trick the fork() into working, even though the swap space is not strictly needed for anything. This is the solution I chose because it's fairly easy to add a swapfile, and I did not want to live with the potential for processes being killed due to overcommit.

  • On Linux, enable overcommit_memory option of the vm system (/proc/sys/vm/overcommit_memory). With overcommit, the call to fork() would always succeed, and since the child process isn't actually going to use that copy of the memory, all is well. Of course, it's possible that with overcommit, your processes will actually attempt to use more memory than is available and will be killed by the kernel. Whether this is appropriate depends on the other uses of the machine. Mission critical machines should probably not risk the out-of-memory killer running amok. But an internal development server that can afford some downtime would be a good place to enable overcommit.

  • Change the JVM to not use fork()+exec() but to use posix_spawn() when available. This is the solution requested in the JVM bug report above and mentioned on the SCons mailing list. It is also implemented in java_posix_spawn.

    I'm trying to find out if that fix made it into JDK7. If not, I wonder if the Jenkins people would be interested in a work around such as java_posix_spawn. There seem to have been attempts to integrate that into Apache commons-exec.

    Programmieraffe, I'm not 100% sure, but your link does suggest that the fix is in JDK7 and JDK6 1.6.0_23 and later. For the record, I was running OpenJDK 1.6.0_18.

See https://stackoverflow.com/questions/1124771/how-to-solve-java-io-ioexception-error-12-cannot-allocate-memory-calling-run

Related Topic