Let's talk performance! A 4-Part Series on Heap Performance

• Part 1: Heap Memory Test Engine - custom tool used to bring a system under load and introduce a memory leak.
• Part 2: jmap - Java Memory Map. This tool is used to view and dump the JVM heap
• Part 3: EclipseMAT - (personal) Preferred tool for viewing and analyzing heap dumps (hprof)
• Part 4: jvisualvm - Build in tool for monitoring Heap, Memory, threads and many more JVM internal features

Introduction:As companies try to do more with less, Java developers and architects are beginning to find themselves needing to better manage the heap space - something my generation never considered. The better you manage your heap space, and assuming you have the horsepower, the more efficiency you can get from your servers; aka: more for less.

Heap Memory Test Engine (HMTE)


Quick word of warning, if you plan on following along with these posts, check your JDK version. If you are running JDK 1.5.0_11-b03, please upgrade to the next version of 1.5 (or up to the latest 1.6). This version of the JDK (http://www.performanceengineer.com/blog/java-memory-leaks/) has a bug in jmap that will throw the following error:
Exception in thread "main" java.lang.OutOfMemoryError: requested 8192 bytes for byte in /BUILD_AREA/jdk1.5.0_11/hotspot/src/share/vm/prims/jni.cpp. Out of swap space?


Over the past two months I have led an effort to understand how the JVM manages heap and threads under different loads with and without a memory leak.  The first objective was to write a tool that would allow me to bring my servers under load and then introduce a memory leak.  This article will cover that 'Heap Memory Test Engine'.  This tool is the root of the article and is used throughout all the parts.
What is the "Heap Memory Test Engine"?  HMTE is a stand-alone Java application used to simulate a memory leak in order to generate profile signatures. The application has the ability to generate system (CPU) load in order to observe profile signature changes while under load.  The software is released under GPL 3 and can be freely downloaded from SourceForge (https://sourceforge.net/projects/hmte/files/) or pulled from CVS (svn co https://hmte.svn.sourceforge.net/svnroot/hmte heapMemoryTestEngine-2.7).

Part 1: Heap Memory Test Engine


HTME uses a number of arguments to control the dynamic features available at a very granular level. Here are the basics to get you started.
Before starting, please be warned! This application can hard-crash big systems if you set your Heap Max [-Xmx12m] too high.
I have brought down test server by setting it to 3 gigs, please be careful.

First off we need to Compile the source using the maven pom file
from the root directory, execute:
mvn clean package

The project has no external dependencies for compiling or executing

Once the package has completed, run the application by executing
java -jar target\heapMemoryTestEngine-<version>.jar [ARG_LIST]

[ARG_LIST]
-chunk <int_value> Size of Heap memory to persist, default is 100000
-sleep <int_value> Time in seconds to pause before consuming the next chunk of heap memory, default is 3 seconds
-delay <int_value> Time in seconds to pause before starting the test, default is 10 (this is used to give developers a chance to connect a memory leaking tool to the application)
-noleak Diabled the memory leak (this is best used if you want to observe normal garbage collection and heap management
-loadthread <int_value> How many Threads to initialize to apply CPU Load (Default is 100)
-loadthreadsleep <int_value> Delay between starting the CPU Load Threads (default is 3 seconds)
-silent Disable the thread output, JVM and Garbage Collection messages will continue to display
-noload Disable the CPU load test (default is enabled)
-nosort Disable the CPU sorting threads (default is enabled)
-nocrypt Disable the CPU encryption threads (default is enabled)
-sortsize <int_value> Number of entries in the list to sort (default is 2500)
-sortbytes <int_value> Number of bytes to sort (default is 500)
-sortsleep <int_value> Time in seconds to sleep when sorting (default is 2000)
-cryptbytes <int_value> Number of bytes to encrypt (default is 500)
-cryptsleep <int_value> Time in seconds to sleep when sorting (default is 2000)
-test set the system to run in test mode only. The engine will not start, but the variables will be initialized and objects created

[EXAMPLE]
java -Xms8m -Xmx12m -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -jar target\heapMemoryTestEngine-2.7.jar -chunk 250000 -sleep 1 -threads 10 -delay 5 -loadthread 100

In this example, the JVM settings are set to:
[-Xms8m] set the initial Heap Memory space to 8 megs
[-Xmx12m] set the max Heap space to 12 megs
[-XX:+HeapDumpOnOutOfMemoryError] We tell the JVM to dump out a binary heap map when it crashes
[-XX:+PrintGCDetails] Write to the console every time GC is executed. This is great for understanding how the JVM panics as it suffocates for more memory

and the Engine settings are set to:
[-chunk 250000] set the memory leak to consume 250,000 bytes every iteration (-sleep <time>)
[-sleep 1] Execute the memory leak every 1 second
[-threads 10] Start 10 memory leak threads
[-delay 5] wait 5 seconds before starting, so the engineer can attach a monitoring tool to test engine
[-loadthread 100] how many CPU load threads to start (these alternate between Encryption and Array Sorting, you can control each of these individually)

Running the Program


Let's run the application with the default settings in the example, with one change. Let's increase the memory leak to half-a-meg at a time as opposed to a quarter: -chunk 500000

Here is the bootstrap, notice the change to the chunk size: java -Xms8m -Xmx12m -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -jar target\heapMemoryTestEngine-2.7.jar -chunk 500000 -sleep 1 -threads 10 -delay 5 -loadthread 100

The following screenshots will show you the stages in which the application runs:. 1) Initialization 2) Running (leaking), 3) Dying and 4) Death

Notice in the "Dying" screenshot, the threads are no longer able to write to the command line due to the thrashing of the GC. What you do see is the last attempt by the GC to recover any heap space available.  This is what big application servers do when they run out of heap space, they thrash around like a landed fish and gulp for air, only to inevitably die out.

Once you have completed this part of the series, make sure that in your working directory you have the dump file. It can vary depending on your OS and Java version.  On my machine, the file is name: java_pid1232.hprof.  Make sure you are able to get to this point, we will come back to use this file in Phase III when we discuss the 'Eclipse Memory Analyzer' (EclipseMAT).

If you want to start testing how to fine tune your JVM memory settings, begin by changing certain parameters and getting an understanding for what each of the JVM arguments will do, and what impact, both positive and negative, they will have on your system.  A good place to start is Oracle's 'Virtual Machine Performance' page found (here).

Once you become comfortable with the JVM settings, begin tweaking the HTME settings. Increase and decrease your leak thread count.  Punish the CPU by increasing the sorting size and encryption blocks. Create long delays between threads to see how the JVM responds to high leaks with minimal CPU load or low leaks with high CPU load.

The final goal of this tool is allow an engineer to create a load similar to the production environment and mimic "best case scenarios". Best of luck, I hope this tool assists in your analysis. The next topic will cover how to generate a heap-dump using jmap while the server (my test case will be Weblogic) is still online and under load.

Initialization

Running (leaking)Dying

Dying

Death