java.lang.OutOfMemoryError: unable to create new native thread

If you’ve been spending too much time dealing with higher level business logic, it’s easy to start losing touch with JVM performance tuning, and the OS in general. A recent bout of java.lang.OutOfMemoryError: unable to create new native thread forced me to spend some time initially barking up the wrong tree, and then fixing it by brute force.

Brute Force Fix

To get onto the fix first. All I needed to do was to configure the nproc settings for the user in question (it happened to be jenkins) upwards in /etc/security/limits.conf

Playing

Before getting onto the most obvious fix, I went off on a tangent and refreshed my memory on the effects of the heap size, permanent generation size, thread stack  size and other related topics (the bash based thread count script is a work in progress). A simple program to help you determine how many threads you can create can be found below if you are also keen on experimentation.

public class RunOutOfMemory {

/**
* Results
*
* ==== Environment: 32bit Win 7, 2GB RAM, Java 7 ====
*
* Reducing the default maximum thread stack size allows more of the process' virtual
* memory address space to be used by the Java heap and vice-versa.
*
* Note, when it comes to virtual memory usage consumed by the jvm process, we are
* not including the heap and permanent generation allocation.
*
* Test: 1. ALL VM arguments at default values: OutOfMemory with 3974 threads =
*                                                       1238 MB (320k / thread)
* 	2. -Xss1m, rest VM arguments at default values: OutOfMemory with 1211 threads
*                                                                           = 1211 MB
* 	3. -Xss2m, rest VM arguments at default values: OutOfMemory with 533 threads
*                                                                           = 1066 MB
*       4. -Xss128k, rest VM arguments at default values: OutOfMemory with 10358 threads
*                                                                           = 1294 MB
*
* "On Windows, the default thread stack size is read from the binary (java.exe). As of
* Java SE 6, this value is 320k in the 32-bit VM and 1024k in the 64-bit VM." oracle.com
*
* ==== Linux ====
*
* NOTE: Pay close attention to your limits configured in /etc/security/limits.conf You
* may want to adjust the nproc setting for your process upwards if you are seeing a
* "cannot create native thread" message
*/
		public static void main(String[] pArgs) {

			try {
				// keep spawning new threads forever
				while (true) {
					// thread begins execution, JVM calls run method of this new thread
					new TestThread().start();
				}
			} catch (java.lang.OutOfMemoryError e ) {

				// when out of memory error is reached, print out the number of
				// successful threads spawned and exit
				System.out.println(TestThread.CREATE_COUNT);
				System.exit(-1);
			}
		}

		static class TestThread extends Thread {

			private static int CREATE_COUNT = 0;

			public TestThread() {
				CREATE_COUNT++;
			}

			// make the thread wait for eternity after being spawned
			public void run() {
				this.suspend();
			}
		}
}
About these ads

One thought on “java.lang.OutOfMemoryError: unable to create new native thread

  1. nice post. I’ve used your class to check my limit.
    Turns out that my problem was in the host machine that run the VM I tries to test. Your class helped me prove the ops guy what I’m talking about
    Thanks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s