1. Cause of Error

Growth in virtual address space causes this error. Leak in native code causes the virtual address space growth. The JVM throws native out of memory (native OOM) if it is not able to get any more native memory. This usually happens when the process reaches the process size limitation on that OS or the machine runs out of RAM and swap space. When this happens, the JVM handles the native OOM condition, logs a message saying that it ran out of native memory or unable to acquire memory and exits. If the JVM or any other loaded module (like libc or a third party module) doesn’t handle this native OOM situation, then the OS will send a sigabort signal to the JVM which will make the JVM exit. Usually, the JVMs will generate a core file when it gets a sigabort signal.

In this case the cause of the problem is the spring framework uses Apache Common's FileUpload library to handle multipart post requests. Version 1.0 of this library uses File.deleteOnExit() to delete temporary files. This method, however, leaks substantial amounts of memory, causing the server process to grow with each handled multipart post request. File Upload uses DiskFileUpload which in turn uses DefaultFileItem whose getTempFile() method calls File.deleteOnExit() which is known to leak memory in the native heap.

2. Way to identify this error

Record the process virtual memory size periodically from the time the application was started until the JVM runs out of native memory. This will help to understand whether the process really hits the size limitation on that OS.

The virtual memory size can be found using these commands

prstat –L –p <PID>

pmap <PID>

ps –p <PID> -o
vsz

where PID is the process Id of the weblogic managed instance. And if the growth of RSS value is significant then it means we have a native memory leak.

3. Suggested Resolution /Recommendations

Apply the patch as suggested in the following bug report http://issues.apache.org/bugzilla/show_bug.cgi?id=27477 or migrate the version of commons file upload jar file from version 1.0 to 1.2.