Diagnostic Tools and Detailed Descriptions

Diagnostic Tools - Overview

Most of the command-line utilities described in this section are either included in the JDK or native operating system tools and utilities. Although the JDK command-line utilities are included in the JDK download, it is important to note that they can be used to diagnose issues and monitor applications that are deployed with the Java Runtime Environment (JRE).

In general, the diagnostic tools and options use various mechanisms to obtain the information they report. The mechanisms are specific to the Virtual Machine (VM) implementation, operating systems, and release. Frequently, only a subset of the tools is applicable to a given issue at a particular time. Command-line options that are prefixed with -XX are specific to Java HotSpot VM. For more information about command-line options used by Java HotSpot VM, see "Java HotSpot VM Command-Line Options."

Note:

The -XX options are not part of the Java API and can vary from one release to the next.

The tools and options are divided into several categories, depending on the type of problem that you are troubleshooting. Certain tools and options might fall into more than one category.

Postmortem diagnostics. These tools and options can be used to diagnose a problem after an application has crashes. See "Postmortem Diagnostics Tools."
Hung processes. These tools can be used to investigate a hung or deadlocked process. See "Hung Processes Tools."
Monitoring. These tools can be used to monitor a running application. See "Monitoring Tools."
Other. These tools and options can be used to help diagnose other issues. See "Other Tools, Options, Variables, and Properties."

Note:

Some command-line utilities described in this section are experimental. The jstack, jinfo, and jmap utilities are examples of utilities that are experimental. The jcmd is a new JDK profiling utility in JDK 8. It is suggested to use the latest diagnostic utility, jcmd instead of the earlier jstack, jinfo, and jmap utilities.

Postmortem Diagnostics Tools

Table 1-1 summarizes the options and tools that are designed for postmortem diagnostics. If an application crashes, these options and tools can be used to obtain additional information, either at the time of the crash or later using information from the crash dump.

Table 1-1 Postmortem Diagnostics Tools

Tool or Option	Description and Usage
Fatal Error Log	When an irrecoverable (fatal) error occurs, an error log is created. This file contains much information obtained at the time of the fatal error. In many cases it is the first item to examine when a crash occurs. See
`-XX:+HeapDumpOnOutOfMemoryError` option	This command-line option specifies the generation of a heap dump when the VM detects a native out-of-memory error. See "The -XX:HeapDumpOnOutOfMemoryError Option."
`-XX:OnError` option	This command-line option specifies a sequence of user-supplied scripts or commands to be executed when a fatal error occurs. For example, on Windows, this option can execute a command to force a crash dump. This option is very useful on systems where a post-mortem debugger is not configured. See "The -XX:OnError Option."
`-XX:+ShowMessageBoxOnError` option	This command-line option suspends a process when a fatal error occurs. Depending on the user response, the option can launch the native debugger (for example, `dbx`, `gdb`, `msdev`) to attach to the VM. See "The -XX:ShowMessageBoxOnError Option."
Other `-XX` options	Several other `-XX` command-line options can be useful in troubleshooting. See "Other -XX Options."
Java VisualVM (postmortem use on Oracle Solaris and Linux operating systems only)	This utility can analyze a core dump by providing a readable display of the core dump in the form of a heap dump and a thread dump, as well as overview information (for example, JVM arguments and system properties. See "Java VisualVM."
`jdb` utility	Debugger support includes an `AttachingConnector`, which allows `jdb` and other Java language debuggers to attach to a core file. This can be useful when trying to understand what each thread was doing at the time of a crash. See "The jdb Utility."
`jhat` utility	This utility provides a convenient means to browse the object topology in a heap dump. See "The jhat Utility."
`jinfo` utility (postmortem use on Oracle Solaris and Linux operating systems only)	This utility can obtain configuration information from a core file obtained from a crash or from a core file obtained using the `gcore` utility. See "The jinfo Utility."
`jmap` utility (postmortem use on Oracle Solaris and Linux operating systems only)	This utility can obtain memory map information, including a heap histogram, from a core file obtained from a crash or from a core file obtained using the `gcore` utility. See "The jmap Utility."
`jsadebugd` daemon (Oracle Solaris and Linux operating systems only)	The Serviceability Agent Debug Daemon (`jsadebugd`) attaches to a Java process or to a core file and acts as a debug server. See "The jsadebugd Daemon."
`jstack` utility	This utility can obtain Java and native stack information from a Java process. On Oracle Solaris and Linux operating systems the utility can also get the information from a core file or a remote debug server. See "The jstack Utility."
Native tools	Each operating system has native tools and utilities that can be used for postmortem diagnosis. See "Native Operating System Tools."

Hung Processes Tools

Table 1-2 summarizes the options and tools that can help in scenarios involving a hung or deadlocked process. These tools do not require any special options to start the application.

The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jstack, jinfo, and jmap utilities for enhanced diagnostics and reduced performance overhead.

Table 1-2 Hung Processes Tools

Tool or Option	Description and Usage
Ctrl-Break handler (Control+\ or `kill -QUIT` pid on Oracle Solaris and Linux operating systems, and Control+Break on Windows)	This key combination performs a thread dump as well as deadlock detection. The Ctrl-Break handler can optionally print a list of concurrent locks and their owners, as well as a heap histogram. See "Control+Break Handler."
`jcmd` utility	The jcmd utility is used to send diagnostic command requests to the JVM, where these requests are useful for controlling Java Flight Recordings. The JFRs are used to troubleshoot and diagnose flight recording events. See "The jcmd Utility."
`jdb` utility	Debugger support includes attaching connectors, which allow `jdb` and other Java language debuggers to attach to a process. This can help show what each thread is doing at the time of a hang or deadlock. See "The jdb Utility."
`jhat` utility	This utility provides a convenient means to browse the object topology in a heap dump. See "The jhat Utility."
`jinfo` utility	This utility can obtain configuration information from a Java process. See "The jinfo Utility."
`jmap` utility	This utility can obtain memory map information, including a heap histogram, from a Java process. On Oracle Solaris and Linux operating systems, the `-F` option can be used if the process is hung. See "The jmap Utility."
`jsadebugd` daemon (Oracle Solaris and Linux operating systems only)	The Serviceability Agent Debug Daemon (`jsadebugd`) attaches to a Java process or to a core file and acts as a debug server. See "The jsadebugd Daemon."
`jstack` utility	This utility can obtain Java and native stack information from a Java process. On Oracle Solaris and Linux operating systems the `-F` option can be used if the process is hung. See "The jstack Utility."
Native tools	Each operating system has native tools and utilities that can be useful in hang or deadlock situations. See "Native Operating System Tools."

Monitoring Tools

The tools listed in the Table 1-3 are designed for monitoring applications that are running.

Table 1-3 Monitoring Tools

Tool or Option	Description and Usage
Java Mission Control	Java Mission Control (JMC) is a new JDK profiling and diagnostic tools platform for HotSpot JVM. It s a tool suite basic monitoring, managing, and production time profiling and diagnostics with high performance. Java Mission Control minimizes the performance overhead that's usually an issue with profiling tools. See "Java Mission Control."
jcmd utility	The jcmd utility is used to send diagnostic command requests to the JVM, where these requests are useful for controlling Java Flight Recordings. The JFRs are used to troubleshoot and diagnose JVM and Java Applications with flight recording events. See "The jcmd Utility."
Java VisualVM	This utility provides a visual interface for viewing detailed information about Java applications while they are running on a Java Virtual Machine. This information can be used in troubleshooting local and remote applications, as well as for profiling local applications. See "Java VisualVM."
JConsole utility	This utility is a monitoring tool that is based on Java Management Extensions (JMX). The tool uses the built-in JMX instrumentation in the Java Virtual Machine to provide information about performance and resource consumption of running applications. See "JConsole."
`jmap` utility	This utility can obtain memory map information, including a heap histogram, from a Java process, a core file, or a remote debug server. See "The jmap Utility."
`jps` utility	This utility lists the instrumented Java HotSpot VMs on the target system. The utility is very useful in environments where the VM is embedded, that is, it is started using the JNI Invocation API rather than the `java` launcher. See "The jps Utility."
`jstack` utility	This utility can obtain Java and native stack information from a Java process. On Oracle Solaris and Linux operating systems the utility can alos get the information from a core file or a remote debug server. See "The jstack Utility."
`jstat` utility	This utility uses the built-in instrumentation in Java to provide information about performance and resource consumption of running applications. The tool can be used when diagnosing performance issues, especially those related to heap sizing and garbage collection. See "The jstat Utility."
`jstatd` daemon	This tool is a Remote Method Invocation (RMI) server application that monitors the creation and termination of instrumented Java Virtual Machines and provides an interface to allow remote monitoring tools to attach to VMs running on the local host. See "The jstatd Daemon."
`visualgc` utility	This utility provides a graphical view of the garbage collection system. As with `jstat`, it uses the built-in instrumentation of Java HotSpot VM. See "The visualgc Tool."
Native tools	Each operating system has native tools and utilities that can be useful for monitoring purposes. For example, the dynamic tracing (DTrace) capability introduced in Oracle Solaris 10 operating system performs advanced monitoring. See "Native Operating System Tools."

Other Tools, Options, Variables, and Properties

In addition to the tools that are designed for specific types of problems, the tools, options, variables, and properties listed in Table 1-4 can help in diagnosing other issues.

Table 1-4 General Troubleshooting Tools and Options

Tool or Option	Description and Usage
Java Mission Control	Java Mission Control (JMC) is a new JDK profiling and diagnostic tools platform for HotSpot JVM. It s a tool suite basic monitoring, managing, and production time profiling and diagnostics with high performance. Java Mission Control minimizes the performance overhead that's usually an issue with profiling tools. See "Java Mission Control."
jcmd utility	The jcmd utility is used to send diagnostic command requests to the JVM, where these requests are useful for controlling Java Flight Recordings (JFR). The JFRs are used to troubleshoot and diagnose JVM and Java Applications with flight recording events. See "The jcmd Utility."
HPROF profiler	This simple profiler can present CPU usage, heap allocation statistics, contention profiles, heap dumps, and states of all the monitors and threads in the Java Virtual Machine. HPROF is useful in analyzing performance, lock contention, memory leaks, and other issues. See "HPROF."
`jhat` utility	This utility is useful in diagnosing unnecessary object retention (or memory leaks). It can be used to browse an object dump, view all reachable objects in the heap, and show which references are keeping an object alive. See "The jhat Utility."
`jinfo` utility	This utility can dynamically set, unset, and change the values of certain JVM flags for a specified Java process. On Oracle Solaris and Linux operating systems, it can also print configuration information. See "The jinfo Utility."
`jrunscript` utility	This utility is a command-line script shell, which supports both interactive and batch-mode script execution. See "The jrunscript Utility."
Oracle Solaris Studio `dbx` debugger	This is an interactive, command-line debugging tool, which allows you to have complete control of the dynamic execution of a program, including stopping the program and inspecting its state. For details, see the latest `dbx` documentation, located at the Oracle Solaris Studio Program Debugging site here `http://docs.oracle.com/cd/E24457_01/html/E21993/index.html`.
Oracle Solaris Studio Performance Analyzer	This tool can help you assess the performance of your code, identify potential performance problems, and locate the part of the code where the problems occur. The Performance Analyzer can be used from the command line or from a graphical user interface. For details, see the Oracle Solaris Studio Performance Analyzer site here `http://docs.oracle.com/cd/E18659_01/html/821-1379/`.
Sun's Dataspace Profiling: DProfile	This tool provides insight into the flow of data within Sun computing systems, helping you identify bottlenecks in both software and hardware. DProfile is supported in the Sun Studio 11 compiler suite through the Performance Analyzer GUI. For more details, see `http://www.oracle.com/technetwork/server-storage/solaris/dtrace-tutorial-142317.html`.
`-Xcheck:jni` option	This option is useful in diagnosing problems with applications that use the Java Native Interface (JNI) or that employ third-party libraries (some JDBC drivers, for example). See "The -Xcheck:jni Option."
`-verbose:class` option	This option enables logging of class loading and unloading. See "The -verbose:class Option."
`-verbose:gc` option	This option enables logging of garbage collection information. See "The -verbose:gc Option."
`-verbose:jni` option	This option enables logging of JNI. See "The -verbose:jni Option."
`JAVA_TOOL_OPTIONS` environment variable	This environment variable allows you to specify the initialization of tools, specifically the launching of native or Java programming language agents using the `-agentlib` or `-javaagent` options. See Appendix C, "Environment Variables and System Properties."
`java.security.debug` system property	This system property controls whether the security checks in the JRE of the Java print trace messages during execution. See "The `java.security.debug` System Property."

Java Mission Control

The Java Mission Control (JMC) is a new JDK profiling and diagnostics tools platform for HotSpot JVM. It is a tool suite for basic monitoring, managing, and production time profiling and diagnostics with high performance. Java Mission Control minimizes the performance overhead that's usually an issue with profiling tools. This tool is a commercial feature built into the JVM and available at runtime.

The Java Mission Control (JMC) consists of Java Management Console (JMX), Java Flight Recorder (JFR), and several other plug-ins downloadable from the tool. The JMX is a tool for monitoring and managing Java applications and the JFR is a profiling tool. Java Mission Control is also available as a set of plug-ins for the Eclipse IDE.

The Java Flight Recorder (JFR) is a commercial feature. You can use it for free on developer desktops/laptops, and for evaluation purposes in test, development, and production environments. However, to enable JFR on a production server, you require a commercial license. Using JMC UI for other purposes on the JDK does not require a commercial license.

For more details about JMC, see the JMC documentation at

http://docs.oracle.com/javase/8/docs/technotes/guides/jmc/index.html

Java Mission Control allows you to perform the following troubleshooting activities:

Java Management console (JMX) connects to a running JVM, collects and displays key characteristics in real time.
Triggers user provided custom actions and rules for JVM, which requires JDK 8.
Experimental plug-ins like - WLS, DTrace, JOverflow and others from JMC tool provide troubleshooting activities.
- DTrace plug-in is an extended DScript language to produce self describing events. It provides visualization similar to Flight Recorder.
- JOverflow is another plug-in tool for analyzing heap waste (empty/sparse collections), which uses hprof dumps. It is recommended to use JDK 8 release for optimal use of JOverflow plug-in.
The Java Flight Recordings (JFR) in Java Mission Control is available to analyze events. The pre-configured tabs allow easy drill down in various areas of common interest, such as, code, memory and gc, threads and IO. The General Events tab and Operative Events tab together allow drilling down further and rapidly homing in on a set of events with certain properties. The Events tab usually have check boxes to only show events in the Operative set.
- JFR when used as a plug-in for JMC client presents diagnostic information in logically grouped tables, charts, and dials. It enables you to select the range of time and level of detail necessary to focus on the problem. For more details about JFR, see Java Flight Recorder at
  
  http://docs.oracle.com/javase/8/docs/technotes/guides/jfr/
The Java Mission Control plug-ins connect to JVM using Java Management Extensions (JMX) agent. The JMX is a standard API for management and monitoring of resources such as applications, devices, services, and the Java Virtual Machine. For more details about JMX, see Java Management Extensions at

http://docs.oracle.com/javase/8/docs/technotes/guides/jmx/index.html

How to Produce a Flight Recording

The Java Flight Recorder (JFR) is a commercial feature. You can use it for free on developer desktops/laptops, and for evaluation purposes in test, development, and production environments. However, to enable JFR on a production server, you need a commercial license. Using JMC UI for other purposes on the JDK does not require a commercial license.

For more details about commercial features and availability, see the product documentation at

http://www.oracle.com/technetwork/java/javase/terms/products/index.html

For more details about JFR commercial license, see the license agreement at

http://www.oracle.com/technetwork/java/javase/terms/license/index.html

The Java Flight Recorder (JFR) can run multiple recordings concurrently and configure each recording with concurrent settings. The following are the advanced troubleshooting and managing techniques for JFR recordings:

Using JMC's Flight Recorder nodes

In the JVM browser double click the Flight Recorder node under the JVM. Follow the wizard for ongoing/continuous recordings that are listed as nodes under the flight recorder node. To dump a node, simply drag and drop the ongoing recording to the editor area or double click it. Continuous recordings must be explicitly dumped.
Using JVM options
- You can collect a significant amount of diagnostic information from Java Flight Recorder by starting the JVM with one of the following options:
  
  -XX:FlightRecorderOptions=loglevel=debug
  
  -XX:FlightRecorderOptions=loglevel=trace
Using startup flags at the command line

You can configure a recording from the command line for continuous recording or fixed time recording at the start of the application using the -XX:StartFlightRecording option. The JFR is a commercial feature, you also have to specify the -XX:+UnlockCommercialFeatures option. The following example illustrates how to run the MyApp application and start a 60-second recording 20seconds after starting the JVM, which will be saved to a file named myrecording.jfr:

java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:StartFlightRecording=delay=20s,duration=60s,name=myrecording,filename=C:\TEMP\myrecording.jfr,settings=profile MyApp

Fixed time recordings are called profile recordings. The settings parameter takes either the path to or the name of a template. Default templates are located in jre/lib/jfr folder.

Note:

Using the settings parameter will require either a JRockit or a HotSpot JDK 8.
Using Diagnostic Commands

You can also control recordings by using Java command-line diagnostic commands. The simplest way to execute a diagnostic command is to use the jcmd tool (located in the Java installation directory). For more details see "The jcmd Utility."
Configuring recordings

You can configure an explicit recording in a number of other ways. These techniques work the same regardless of how you start a recording (that is, either by using the command-line approach or by using diagnostic commands).
Automatic recording using triggers

You can use the Console in Java Mission Control to set triggers. A trigger is a rule that executes an action whenever a condition specified by the rule is true.

For more details about configuring and managing Java Flight Recordings, see the following documentation.

http://docs.oracle.com/javase/8/docs/technotes/guides/jfr/run.html

Java VisualVM

The Java VisualVM is one of the latest tools included in the JDK download. This tool is useful for Java application developers to troubleshoot applications and to monitor and improve the applications' performance. With Java VisualVM you can generate and analyze heap dumps, track down memory leaks, perform and monitor garbage collection, and perform lightweight memory and CPU profiling. The tool is also useful for tuning, heap sizing, offline analysis, and postmortem diagnosis.

In addition, you can use existing plug-ins that expand the functionality of Java VisualVM. For example, most of the functionality of the JConsole tool is available through the MBeans tab and the JConsole plug-in wrapper tab. You can choose from a catalog of standard Java VisualVM plug-ins by choosing Plugins from the Tools menu in the main Java VisualVM window.

For more information, see the Java VisualVM documentation at

http://docs.oracle.com/javase/8/docs/technotes/guides/visualvm/index.html

Java VisualVM allows you to perform the following troubleshooting activities:

View a list of local and remote Java applications.
View application configuration and runtime environment. For each application, the tool shows basic runtime information: PID, host, main class, arguments passed to the process, JVM version, JDK home, JVM flags, JVM arguments, and system properties.
Enable and disable the creation of a heap dump when a specified application encounters an OutOfMemoryError exception.
Monitor application memory consumption, running threads, and loaded classes.
Trigger a garbage collection immediately.
Create a heap dump immediately. You can then view the heap dump in several views: summary, by class, by instance. You can also save the heap dump to your local file system.
Profile application performance or analyze memory allocation (for local applications only). You can also save the profiling data.
Create a thread dump (stack trace of the application's active threads) immediately. You can then view the thread dump.
Analyze core dumps (with Oracle Solaris and Linux operating systems).
Analyze applications offline, by taking application snapshots.
Get additional plug-ins contributed by the community.
Write and share your own plug-ins.
Display and interact with MBeans (after installing the MBeans tab plug-in).

When you start Java VisualVM, the main Application window opens, displaying a list of Java applications running on the local machine, a list of Java applications running on any connected remote machines, a list of any JVM core dumps that were taken and saved (with Oracle Solaris and Linux operating systems), and a list of any application snapshots that were taken and saved.

Th Java VisualVM automatically detects and connects to the Java Management Extensions (JMX) agents for Java applications that are running on JDK or that have been started with the correct system properties. For the tool to detect and connect to the agents on a remote machine, the jstatd daemon must be running on the remote machine (see "The jstatd Daemon.")If Java VisualVM cannot automatically discover and connect to JMX agents that are running in a target application, then the tool provides a means for you to explicitly create these connections.

The jdb Utility

The jdb utility is included in the JDK as an example command-line debugger. The jdb utility uses the Java Debug Interface (JDI) to launch or connect to the target JVM. The source code for jdb is included in $JAVA_HOME/demo/jpda/examples.jar.

The JDI is a high-level Java API that provides information useful for debuggers and similar systems that need access to the running state of a (usually remote) virtual machine. JDI is a component of the Java Platform Debugger Architecture (JPDA). See "Java Platform Debugger Architecture."

In JDI a connector is the means by which the debugger connects to the target JVM. The JDK traditionally ships with connectors that launch and establish a debugging session with a target JVM, as well as connectors that are used for remote debugging (using TCP/IP or shared memory transports).

The JDK also ships with several Serviceability Agent (SA) connectors that allow a Java language debugger to attach to a crash dump or hung process. This can be useful in determining what the application was doing at the time of the crash or hang.

These connectors are SACoreAttachingConnector, SADebugServerAttachingConnector, and SAPIDAttachingConnector.

These connectors are generally used with enterprise debuggers, such as the NetBeans integrated development environment (IDE) or commercial IDEs. The following subsections demonstrate how these connectors can be used with the jdb command-line debugger.

For detailed information about the connectors, see the "Connection and Invocation Details" section of the JPDA documentation at

http://docs.oracle.com/javase/8/docs/technotes/guides/jpda/index.html

The command jdb -listconnectors prints a list of the available connectors. The command jdb -help prints the command usage help.

For more information about the jdb utility, refer to the man pages:

Oracle Solaris and Linux operating systems:

http://docs.oracle.com/javase/8/docs/technotes/tools/unix/jdb.html
Windows:

http://docs.oracle.com/javase/8/docs/technotes/tools/windows/jdb.html

Attaching to a Process

This example uses the SA PID Attaching Connector to attach to a process. The target process is not started with any special options; that is, the -agentlib:jdwp option is not required. When this connector attaches to a process it does so in read-only mode: the debugger can examine threads and the running application, but it cannot change anything. The process is frozen while the debugger is attached.

The command in the following example instructs jdb to use a connector named sun.jvm.hotspot.jdi.SAPIDAttachingConnector. This is a connector name rather than a class name. The connector takes one argument named pid, whose value is the process ID of the target process (9302 in this example).

$ jdb -connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid=9302

Initializing jdb ...
> threads
Group system:
  (java.lang.ref.Reference$ReferenceHandler)0xa Reference Handler unknown
  (java.lang.ref.Finalizer$FinalizerThread)0x9  Finalizer         unknown
  (java.lang.Thread)0x8                         Signal Dispatcher running
  (java.lang.Thread)0x7                         Java2D Disposer   unknown
  (java.lang.Thread)0x2                         TimerQueue        unknown
Group main:
  (java.lang.Thread)0x6                         AWT-XAWT          running
  (java.lang.Thread)0x5                         AWT-Shutdown      unknown
  (java.awt.EventDispatchThread)0x4             AWT-EventQueue-0  unknown
  (java.lang.Thread)0x3                         DestroyJavaVM     running
  (sun.awt.image.ImageFetcher)0x1               Image Animator 0  sleeping
  (java.lang.Thread)0x0                         Intro             running
> thread 0x7
Java2D Disposer[1] where
  [1] java.lang.Object.wait (native method)
  [2] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:116)
  [3] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:132)
  [4] sun.java2d.Disposer.run (Disposer.java:125)
  [5] java.lang.Thread.run (Thread.java:619)
Java2D Disposer[1] up 1
Java2D Disposer[2] where
  [2] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:116)
  [3] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:132)
  [4] sun.java2d.Disposer.run (Disposer.java:125)
  [5] java.lang.Thread.run (Thread.java:619)

In this example, the threads command is used to get a list of all threads. Then a specific thread is selected with the thread 0x7 command, and the where command is used to get a thread dump. Next, the up 1 command is used to move up one frame in the stack, and the where command is used again to get a thread dump.

Attaching to a Core File on the Same Machine

The SA Core Attaching Connector is used to attach the debugger to a core file. The core file might have been created after a crash (see "Troubleshooting System Crashes.") The core file can also be obtained by using the gcore command on Oracle Solaris operating system or the gcore command in gdb on Linux. Because the core file is a snapshot of the process at the time the core file was created, the connector attaches in read-only mode: the debugger can examine threads and the running application at the time of the crash.

The following command is an example of using this connector:

$ jdb -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable=$JAVA_HOME/bin/java,core=core.20441

This command instructs jdb to use a connector named sun.jvm.hotspot.jdi.SACoreAttachingConnector. The connector takes two arguments: javaExecutable and core. The javaExecutable argument indicates the name of the Java binary. The core argument is the core file name (the core from the process with PID 20441 in this example).

Attaching to a Core File or a Hung Process from a Different Machine

To debug a core file that has been transported from another machine, the operating system versions and libraries must match. In this case you can first run a proxy server called the SA Debug Server. Then, on the machine where the debugger is installed, you can use the SA Debug Server Attaching Connector to connect to the debug server.

In the following example, there are two machines: machine1 and machine2. A core file is available on machine1 and the debugger is available on machine2. The SA Debug Server is started on machine1 with:

$ jsadebugd $JAVA_HOME/bin/java core.20441

The jsadebugd command takes two arguments. The first argument is the name of the executable. Usually, this is java, but it can be another name (in embedded VMs, for example). The second argument is the name of the core file. In this example, the core file was obtained for a process with PID 20441 using the gcore utility.

On machine2, the debugger connects to the remote SA Debug Server using the SA Debug Server Attaching Connector, as with the following command:

$ jdb -connect sun.jvm.hotspot.jdi.SADebugServerAttachingConnector:debugServerName=machine1

This command instructs jdb to use a connector named sun.jvm.hotspot.jdi.SADebugServerAttachingConnector. The connector has one argument, debugServerName, which is the host name or IP address of the machine where the SA Debug Server is running.

Note:

The SA Debug Server can also be used to remotely debug a hung process. In that case, it takes a single argument, which is the PID of the process. In addition, if it is required to run multiple debug servers on the same machine, each one must be provided with a unique ID. With the SA Debug Server Attaching Connector, this ID is provided as an additional connector argument. For more information on these and other related details, see the JPDA documentation at

http://docs.oracle.com/javase/8/docs/technotes/guides/jpda/index.html

The jhat Utility

The jhat tool provides a convenient means to browse the object topology in a heap snapshot. This tool replaces the Heap Analysis Tool (HAT).

For more information about the jhat utility, see the jhat man page at

http://docs.oracle.com/javase/8/docs/technotes/tools/unix/jhat.html

The tool parses a heap dump in binary format (for example, a heap dump produced by jmap -dump).

This utility can help debug unintentional object retention. This term is used to describe an object that is no longer needed but is kept alive due to references through some path from the rootset. This can happen, for example, if an unintentional static reference to an object remains after the object is no longer needed, if an Observer or Listener fails to unregister itself from its subject when it is no longer needed, or if a Thread that refers to an object does not terminate when it should. Unintentional object retention is the Java language equivalent of a memory leak.

The tool provides a number of standard queries. For example, the Roots query displays all reference paths from the rootset to a specified object and is particularly useful for finding unnecessary object retention.

In addition to the standard queries, you can develop your own custom queries with the Object Query Language (OQL) interface.

When you issue the jhat command, the utility starts an HTTP server on a specified TCP port. You can then use any browser to connect to the server and execute queries on the specified heap dump.

The following example shows how to execute jhat to analyze a heap dump file named snapshot.hprof:

$ jhat snapshot.hprof
Started HTTP server on port 7000
Reading from java_pid2278.hprof...
Dump file created Fri May 19 17:18:38 BST 2006
Snapshot read, resolving...
Resolving 6162194 objects...
Chasing references, expect 12324 dots................................
Eliminating duplicate references.....................................
Snapshot resolved.
Server is ready.

At this point, jhat has started an HTTP server on port 7000. Point your browser to http://localhost:7000 to connect to the jhat server.

When you are connected to the server, you can execute a standard query (see "Standard Queries") or create an OQL query (see "Custom Queries.") The All Classes query is displayed by default.

Standard Queries

When you have connected to the jhat server, you can execute the following standard queries:

All Classes Query

The default page is the All Classes query, which displays all of the classes present in the heap, excluding platform classes. This list is sorted by fully qualified class name, and broken out by package. Click the name of a class to go to the Class query.

The second variant of this query includes the platform classes. Platform classes include classes whose fully qualified names start with prefixes such as java, sun., javax.swing., or char[. The list of prefixes is in a system resource file called /resources/platform_names.txt. You can override this list by replacing it in the JAR file, or by arranging for your replacement to occur first on the classpath when jhat is invoked.

Class Query

The Class query displays information about a class. This includes its superclass, any subclasses, instance data members, and static data members. From this page you can navigate to any of the classes that are referenced, or you can navigate to an Instances query.

Object Query

The Object query provides information about an object that was on the heap. From here, you can navigate to the class of the object and to the value of any object members. You can also navigate to objects that refer to the current object. Perhaps the most valuable query is at the end: the Roots query reference chains from the rootset.

Note that the object query also provides a stack backtrace of the point of allocation of the object.

Instances Query

The Instances query displays all instances of a given class. The allInstances variant includes instances of subclasses of the given class as well. From here, you can navigate back to the source class, or you can navigate to an Object query on one of the instances.

Roots Query

The Roots query displays reference chains from the rootset to a given object. It provides one chain for each member of the rootset from which the given object is reachable. When calculating these chains, the tool does a depth-first search, so that it will provide reference chains of minimal length.

There are two kinds of the Roots query: one that excludes weak references (Roots), and one that includes them (All Roots). A weak reference is a reference object that does not prevent its referent from being made finalizable, finalized, and then reclaimed. If an object is only referred to by a weak reference, then it is usually not considered to be retained, because the garbage collector can collect it as soon as it needs the space.

Note:

This is probably the most valuable query in jhat for debugging unintentional object retention. When you find an object that is being retained, this query tells you why it is being retained.

Reachable Objects Query

This query is accessible from the Object query and shows the transitive closure of all objects reachable from a given object. This list is sorted in decreasing size, and alphabetically within each size. At the end, the total size of all of the reachable objects is given. This can be useful for determining the total runtime footprint of an object in memory, at least in systems with simple object topologies.

This query is most valuable when used in conjunction with the -exclude command-line option. This is useful, for example, if the object being analyzed is an Observable. By default, all of its Observers would be reachable, which would count against the total size. The -exclude option allows you to exclude the data members java.util.Observable.obs and java.util.Observable.arr.

Instance Counts for All Classes Query

This query shows the count of instances for every class in the system, excluding platform classes. It is sorted in descending order, by instance count. A good way to spot a problem with unintentional object retention is to run a program for a long time with a variety of input, and then request a heap dump. Looking at the instance counts for all classes, you may recognize a number of classes because there are more instances than you expect. Then you can analyze them to determine why they are being retained (possibly using the Roots query). A variant of this query includes platform classes.

For more information about platform classes, see "All Classes Query."
All Roots Query

This query shows all members of the rootset, including weak references.

For more information about weak references, see "All Roots Query."
New Instances Query

The New Instances query is available only if you invoke the jhat server with two heap dumps. This query is similar to the Instances query, except that it shows only new instances. An instance is considered new if it is in the second heap dump and there is no object of the same type with the same ID in the baseline heap dump. An object's ID is a 32-bit or 64-bit integer that uniquely identifies the object.
Histogram Queries

The built-in histogram and finalizer histogram queries also provide useful information.

Custom Queries

You can develop your own custom queries with the built-in Object Query Language (OQL) interface. Click the Execute OQL Query button on the first page to display the OQL query page, where you can create and execute your custom queries. The OQL Help facility describes the built-in functions, with examples.

The syntax of the select statement is as follows:

select JavaScript-expression-to-select [from [instanceof] classname identifier [where JavaScript-boolean-expression-to-filter]]

The following is an example of the select statement:

select s from java.lang.String s where s.count >= 100

Heap Analysis Hints

To get useful information from jhat often requires some knowledge of the application and the libraries and APIs that it uses. You can use jhat to answer two important questions:

What is keeping an object alive?

When you view an object instance, you can check the objects listed in the section entitled "References to this object" to see which objects directly reference this object. More importantly, you can use the Roots query to determine the reference chains from the root set to the given object. These reference chains show a path from a root object to this object. With these chains, you can quickly see how an object is reachable from the root set.

As noted earlier, the Roots query excludes weak references, whereas All Roots query includes them. A weak reference is a reference object that does not prevent its referent from being made finalizable, finalized, and then reclaimed. If an object is only referred to by a weak reference, then the garbage collector can collect it as soon as it needs the space.

The jhat tool sorts the rootset reference chains by the type of the root, in the following order:

Static data members of Java classes.
Java local variables. For these roots, the thread responsible for them is shown. Because a thread is a Java object, this link is clickable. This allows you, for example, to easily navigate to the name of the thread.
Native static values.
Native local variables. Again, such roots are identified by their thread.

Where was this object allocated?

When an object instance is being displayed, the section entitled "Objects allocated from" shows the allocation site in the form of a stack trace. In this way, you can see where the object was created.

Note that this allocation site information is available only if the heap dump was created with HPROF using the heap=all option. This HPROF option includes both the heap=dump option and the heap=sites option. For more information about HPROF and its options, see "HPROF."

If the leak cannot be identified using a single object dump, then another approach is to collect a series of dumps and to focus on the objects created in the interval between each dump. The jhat tool provides this capability using the -baseline option.

The -baseline option allows two dumps to be compared if they were produced by HPROF and from the same VM instance. If the same object appears in both dumps, then it will be excluded from the list of new objects reported. One dump is specified as a baseline, and the analysis can focus on the objects that are created in the second dump since the baseline was obtained.

The following example shows how to specify the baseline:

$ jhat -baseline snapshot.hprof#1 snapshot.hprof#2

In the preceding example, the two dumps are in the file snapshot.hprof, and they are distinguished by appending #1 and #2 to the file name.

When jhat is started with two heap dumps, the Instance Counts for All Classes query includes an additional column that is the count of the number of new objects for that type. An instance is considered new if it is in the second heap dump and there is no object of the same type with the same ID in the baseline. If you click a new count, then jhat lists the new objects of that type. Then for each instance, you can view where it was allocated, which objects these new objects reference, and which other objects reference the new object.

In general, the -baseline option can be very useful if the objects that need to be identified are created in the interval between successive dumps.

The jinfo Utility

The jinfo command-line utility gets configuration information from a running Java process or crash dump and prints the system properties or the command-line flags that were used to start the JVM.

The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jinfo utility for enhanced diagnostics and reduced performance overhead.

The utility can also use the jsadebugd daemon to query a process or core file on a remote machine. Note that the output takes longer to print in this case.

With the -flag option, the utility can dynamically set, unset, or change the value of certain JVM flags for the specified Java process. See "Java HotSpot VM Command-Line Options."

For more information about the jinfo utility, refer to the man page at

http://docs.oracle.com/javase/8/docs/technotes/tools/unix/jinfo.html

The following is an example of the output from a Java process with PID number 29620:

$ jinfo 29620
Attaching to process ID 29620, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.6.0-rc-b100
Java System Properties:

java.runtime.name = Java(TM) SE Runtime Environment
sun.boot.library.path = /usr/jdk/instances/jdk1.6.0/jre/lib/sparc
java.vm.version = 1.6.0-rc-b100
java.vm.vendor = Sun Microsystems Inc.
java.vendor.url = http://java.sun.com/
path.separator = :
java.vm.name = Java HotSpot(TM) Client VM
file.encoding.pkg = sun.io
sun.java.launcher = SUN_STANDARD
sun.os.patch.level = unknown
java.vm.specification.name = Java Virtual Machine Specification
user.dir = /home/js159705
java.runtime.version = 1.6.0-rc-b100
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
java.endorsed.dirs = /usr/jdk/instances/jdk1.6.0/jre/lib/endorsed
os.arch = sparc
java.io.tmpdir = /var/tmp/
line.separator =

java.vm.specification.vendor = Sun Microsystems Inc.
os.name = SunOS
sun.jnu.encoding = ISO646-US
java.library.path = /usr/jdk/instances/jdk1.6.0/jre/lib/sparc/client:/usr/jdk/instances/jdk1.6.0/jre/lib/sparc:
/usr/jdk/instances/jdk1.6.0/jre/../lib/sparc:/net/gtee.sfbay/usr/sge/sge6/lib/sol-sparc64:
/usr/jdk/packages/lib/sparc:/lib:/usr/lib
java.specification.name = Java Platform API Specification
java.class.version = 50.0
sun.management.compiler = HotSpot Client Compiler
os.version = 5.10
user.home = /home/js159705
user.timezone = US/Pacific
java.awt.printerjob = sun.print.PSPrinterJob
file.encoding = ISO646-US
java.specification.version = 1.6
java.class.path = /usr/jdk/jdk1.6.0/demo/jfc/Java2D/Java2Demo.jar
user.name = js159705
java.vm.specification.version = 1.0
java.home = /usr/jdk/instances/jdk1.6.0/jre
sun.arch.data.model = 32
user.language = en
java.specification.vendor = Sun Microsystems Inc.
java.vm.info = mixed mode, sharing
java.version = 1.6.0-rc
java.ext.dirs = /usr/jdk/instances/jdk1.6.0/jre/lib/ext:/usr/jdk/packages/lib/ext
sun.boot.class.path = /usr/jdk/instances/jdk1.6.0/jre/lib/resources.jar:
/usr/jdk/instances/jdk1.6.0/jre/lib/rt.jar:/usr/jdk/instances/jdk1.6.0/jre/lib/sunrsasign.jar:
/usr/jdk/instances/jdk1.6.0/jre/lib/jsse.jar:
/usr/jdk/instances/jdk1.6.0/jre/lib/jce.jar:/usr/jdk/instances/jdk1.6.0/jre/lib/charsets.jar:
/usr/jdk/instances/jdk1.6.0/jre/classes
java.vendor = Sun Microsystems Inc.
file.separator = /
java.vendor.url.bug = http://java.sun.com/cgi-bin/bugreport.cgi
sun.io.unicode.encoding = UnicodeBig
sun.cpu.endian = big
sun.cpu.isalist =

VM Flags:

If you start the target JVM with the -classpath and -Xbootclasspath arguments, the output from jinfo provides the settings for java.class.path and sun.boot.class.path. This information might be needed when investigating class loader issues.

In addition to obtaining information from a process, the jinfo tool can use a core file as input. On Oracle Solaris operating system, for example, the gcore utility can be used to get a core file of the process in the preceding example. The core file will be named core.29620 and will be generated in the working directory of the process. The path to the Java executable and the core file must be specified as arguments to the jinfo utility, as in the following example:

$ jinfo $JAVA_HOME/bin/java core.29620

Sometimes the binary name will not be java. This occurs when the VM is created using the JNI invocation API. The jinfo tool requires the binary from which the core file was generated.

The jmap Utility

The jmap command-line utility prints memory-related statistics for a running VM or core file.

The utility can also use the jsadebugd daemon to query a process or core file on a remote machine. Note that the output takes longer to print in this case.

The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jmap utility for enhanced diagnostics and reduced performance overhead.

If jmap is used with a process or core file without any command-line options, then it prints the list of shared objects loaded (the output is similar to the pmap utility on Oracle Solaris operating system). For more specific information, you can use the options -heap, -histo, or -permstat. These options are described in the subsections that follow.

In addition, the JDK 7 release introduced the -dump:format=b,file=filename option, which causes jmap to dump the Java heap in binary HPROF format to a specified file. This file can then be analyzed with the jhat tool.

If the jmap pid command does not respond because of a hung process, then the -F option can be used (on Oracle Solaris and Linux operating systems only) to force the use of the Serviceability Agent.

For more information about the jmap utility, see the man page at

http://docs.oracle.com/javase/8/docs/technotes/tools/unix/jmap.html

Heap Configuration and Usage

The -heap option is used to obtain the following Java heap information:

Information specific to the garbage collection (GC) algorithm, including the name of the GC algorithm (for example, parallel GC) and algorithm-specific details (such as number of threads for parallel GC).
Heap configuration that might have been specified as command-line options or selected by the VM based on the machine configuration.
Heap usage summary. For each generation (area of the heap), the tool prints the total heap capacity, in-use memory, and available free memory. If a generation is organized as a collection of spaces (for example, the new generation), then a space specific memory size summary is included.

The following example shows output from the jmap -heap command.

$ jmap -heap 29620
Attaching to process ID 29620, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.6.0-rc-b100

using thread-local object allocation.
Mark Sweep Compact GC

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 67108864 (64.0MB)
   NewSize          = 2228224 (2.125MB)
   MaxNewSize       = 4294901760 (4095.9375MB)
   OldSize          = 4194304 (4.0MB)
   NewRatio         = 8
   SurvivorRatio    = 8
   PermSize         = 12582912 (12.0MB)
   MaxPermSize      = 67108864 (64.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 2031616 (1.9375MB)
   used     = 70984 (0.06769561767578125MB)
   free     = 1960632 (1.8698043823242188MB)
   3.4939673639112905% used
Eden Space:
   capacity = 1835008 (1.75MB)
   used     = 36152 (0.03447723388671875MB)
   free     = 1798856 (1.7155227661132812MB)
   1.9701276506696428% used
From Space:
   capacity = 196608 (0.1875MB)
   used     = 34832 (0.0332183837890625MB)
   free     = 161776 (0.1542816162109375MB)
   17.716471354166668% used
To Space:
   capacity = 196608 (0.1875MB)
   used     = 0 (0.0MB)
   free     = 196608 (0.1875MB)
   0.0% used
tenured generation:
   capacity = 15966208 (15.2265625MB)
   used     = 9577760 (9.134063720703125MB)
   free     = 6388448 (6.092498779296875MB)
   59.98769400974859% used
Perm Generation:
   capacity = 12582912 (12.0MB)
   used     = 1469408 (1.401336669921875MB)
   free     = 11113504 (10.598663330078125MB)
   11.677805582682291% used

Heap Histogram

The jmap command with the -histo option can be used to obtain a class specific histogram of the heap. Depending on the parameter specified, the jmap -histo command can print out the heap histogram for a running process or a core file.

When the command is executed on a running process, the tool prints the number of objects, memory size in bytes, and fully qualified class name for each class. Internal classes in the Java HotSpot VM are enclosed in angle brackets. The histogram is useful in understanding how the heap is used. To get the size of an object, you must divide the total size by the count of that object type.

The following example shows output from the jmap -histo command when it is executed on a process with PID number 29620:

$ jmap -histo 29620
num   #instances    #bytes  class name
--------------------------------------
  1:      1414     6013016  [I
  2:       793      482888  [B
  3:      2502      334928  <constMethodKlass>
  4:       280      274976  <instanceKlassKlass>
  5:       324      227152  [D
  6:      2502      200896  <methodKlass>
  7:      2094      187496  [C
  8:       280      172248  <constantPoolKlass>
  9:      3767      139000  [Ljava.lang.Object;
 10:       260      122416  <constantPoolCacheKlass>
 11:      3304      112864  <symbolKlass>
 12:       160       72960  java2d.Tools$3
 13:       192       61440  <objArrayKlassKlass>
 14:       219       55640  [F
 15:      2114       50736  java.lang.String
 16:      2079       49896  java.util.HashMap$Entry
 17:       528       48344  [S
 18:      1940       46560  java.util.Hashtable$Entry
 19:       481       46176  java.lang.Class
 20:        92       43424  javax.swing.plaf.metal.MetalScrollButton
... more lines removed here to reduce output...
1118:         1           8  java.util.Hashtable$EmptyIterator
1119:         1           8  sun.java2d.pipe.SolidTextRenderer
Total    61297    10152040

When the jmap -histo command is executed on a core file, the tool prints the size, count, and class name for each class. Internal classes in the Java HotSpot VM are prefixed with an asterisk (*).

The following example shows output of the jmap -histo command when it is executed on a core file:

& jmap -histo /net/koori.sfbay/onestop/jdk/6.0/promoted/all/b100/binaries/solaris-sparcv9/bin/java core
Attaching to core core from executable /net/koori.sfbay/onestop/jdk/6.0/
promoted/all/b100/binaries/solaris-sparcv9/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.6.0-rc-b100
Iterating over heap. This may take a while...
Heap traversal took 8.902 seconds.

Object Histogram:

Size    Count    Class description
-------------------------------------------------------
4151816    2941    int[]
2997816    26403    * ConstMethodKlass
2118728    26403    * MethodKlass
1613184    39750    * SymbolKlass
1268896    2011    * ConstantPoolKlass
1097040    2011    * InstanceKlassKlass
882048    1906    * ConstantPoolCacheKlass
758424    7572    char[]
733776    2518    byte[]
252240    3260    short[]
214944    2239    java.lang.Class
177448    3341    * System ObjArray
176832    7368    java.lang.String
137792    3756    java.lang.Object[]
121744    74    long[]
72960    160    java2d.Tools$3
63680    199    * ObjArrayKlassKlass
53264    158    float[]
... more lines removed here to reduce output...

Permanent Generation Statistics

The permanent generation is the area of the heap that holds all the reflective data of the virtual machine itself, such as class and method objects. This area is also called "method area" in The Java Virtual Machine Specification.

Configuring the size of the permanent generation can be important for applications that dynamically generate and load a very large number of classes (for example, Java Server Pages or web containers). If an application loads too many classes, then it is possible it will terminate with the following error:

Exception in thread thread_name java.lang.OutOfMemoryError: PermGen space

For a description of this and other variants of the OutOfMemoryError exception, see "Understanding the OutofMemoryError Exception."

To get further information about the permanent generation, you can use the -permstat option of the jmap command to print statistics for the objects in the permanent generation. The following example shows output from the jmap -permstat command executed on a process with PID number 29620:

$ jmap -permstat 29620
Attaching to process ID 29620, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.6.0-rc-b100
12674 intern Strings occupying 1082616 bytes.
finding class loader instances ..Unknown oop at 0xd0400900
Oop's klass is 0xd0bf8408
Unknown oop at 0xd0401100
Oop's klass is null
done.
computing per loader stat ..done.
please wait.. computing liveness.........................................done.
class_loader    classes bytes   parent_loader   alive?  type

<bootstrap>     1846 5321080  null        live   <internal>
0xd0bf3828  0      0      null         live    sun/misc/Launcher$ExtClassLoader@0xd8c98c78
0xd0d2f370  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0c99280  1   1440      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0b71d90  0      0   0xd0b5b9c0    live java/util/ResourceBundle$RBClassLoader@0xd8d042e8
0xd0d2f4c0  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0b5bf98  1    920   0xd0b5bf38      dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0c99248  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0d2f488  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0b5bf38  6   11832  0xd0b5b9c0      dead    sun/reflect/misc/MethodUtil@0xd8e8e560
0xd0d2f338  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0d2f418  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0d2f3a8  1    904     null          dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0b5b9c0  317 1397448 0xd0bf3828     live    sun/misc/Launcher$AppClassLoader@0xd8cb83d8
0xd0d2f300  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0d2f3e0  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0ec3968  1   1440      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0e0a248  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0c99210  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0d2f450  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0d2f4f8  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50
0xd0e0a280  1    904      null         dead    sun/reflect/DelegatingClassLoader@0xd8c22f50

total = 22      2186    6746816   N/A   alive=4, dead=18       N/A

For each class loader object, the following details are printed:

The address of the class loader object at the snapshot when the utility was run
The number of classes loaded
The approximate number of bytes consumed by metadata for all classes loaded by this class loader
The address of the parent class loader (if any)
A "live" or "dead" indication of whether the loader object will be garbage collected in the future
The class name of this class loader

The jstack Utility

The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jstack utility for enhanced diagnostics and reduced performance overhead.

The jstack command-line utility attaches to the specified process or core file and prints the stack traces of all threads that are attached to the virtual machine, including Java threads and VM internal threads, and optionally native stack frames. The utility also performs deadlock detection.

The utility can also use the jsadebugd daemon to query a process or core file on a remote machine. Note that the output takes longer to print in this case.

A stack trace of all threads can be useful in diagnosing a number of issues, such as deadlocks or hangs.

The -l option, which instructs the utility to look for ownable synchronizers in the heap and print information about java.util.concurrent.locks. Without this option, the thread dump includes information only on monitors.

The output from the jstack pid option is the same as that obtained by pressing Ctrl+\ at the application console (standard input) or by sending the process a QUIT signal. See "Control+Break Handler" for an output example.

Thread dumps can also be obtained programmatically using the Thread.getAllStackTraces method, or in the debugger using the debugger option to print all thread stacks (the where command in the case of the jdb sample debugger).

For more information about the jstack utility, see the man page at

http://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstack.html

Forcing a Stack Dump

If the jstack pid command does not respond because of a hung process, then the -F option can be used (on Oracle Solaris and Linux operating systems only) to force a stack dump, as in the following example:

$ jstack -F 8321
Attaching to process ID 8321, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.6.0-rc-b100
Deadlock Detection:

Found one Java-level deadlock:
=============================

"Thread2":
  waiting to lock Monitor@0x000af398 (Object@0xf819aa10, a java/lang/String),
  which is held by "Thread1"
"Thread1":
  waiting to lock Monitor@0x000af400 (Object@0xf819aa48, a java/lang/String),
  which is held by "Thread2"

Found a total of 1 deadlock.

Thread t@2: (state = BLOCKED)

Thread t@11: (state = BLOCKED)
 - Deadlock$DeadlockMakerThread.run() @bci=108, line=32 (Interpreted frame)

Thread t@10: (state = BLOCKED)
 - Deadlock$DeadlockMakerThread.run() @bci=108, line=32 (Interpreted frame)

Thread t@6: (state = BLOCKED)

Thread t@5: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=-1107318896 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)

Thread t@4: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Interpreted frame)

Printing Stack Trace from a Core Dump

To obtain stack traces from a core dump, execute the jstack command on a core file, as follows:

$ jstack $JAVA_HOME/bin/java core

Printing a Mixed Stack

The jstack utility can also be used to print a mixed stack; that is, it can print native stack frames in addition to the Java stack. Native frames are the C/C++ frames associated with VM code and JNI/native code.

To print a mixed stack, use the -m option, as in the following example:

$ jstack -m 21177
Attaching to process ID 21177, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.6.0-rc-b100
Deadlock Detection:

Found one Java-level deadlock:
=============================

"Thread1":
  waiting to lock Monitor@0x0005c750 (Object@0xd4405938, a java/lang/String),
  which is held by "Thread2"
"Thread2":
  waiting to lock Monitor@0x0005c6e8 (Object@0xd4405900, a java/lang/String),
  which is held by "Thread1"

Found a total of 1 deadlock.

----------------- t@1 -----------------
0xff2c0fbc    __lwp_wait + 0x4
0xff2bc9bc    _thrp_join + 0x34
0xff2bcb28    thr_join + 0x10
0x00018a04    ContinueInNewThread + 0x30
0x00012480    main + 0xeb0
0x000111a0    _start + 0x108
----------------- t@2 -----------------
0xff2c1070    ___lwp_cond_wait + 0x4
0xfec03638    bool Monitor::wait(bool,long) + 0x420
0xfec9e2c8    bool Threads::destroy_vm() + 0xa4
0xfe93ad5c    jni_DestroyJavaVM + 0x1bc
0x00013ac0    JavaMain + 0x1600
0xff2bfd9c    _lwp_start
----------------- t@3 -----------------
0xff2c1070    ___lwp_cond_wait + 0x4
0xff2ac104    _lwp_cond_timedwait + 0x1c
0xfec034f4    bool Monitor::wait(bool,long) + 0x2dc
0xfece60bc    void VMThread::loop() + 0x1b8
0xfe8b66a4    void VMThread::run() + 0x98
0xfec139f4    java_start + 0x118
0xff2bfd9c    _lwp_start
----------------- t@4 -----------------
0xff2c1070    ___lwp_cond_wait + 0x4
0xfec195e8    void os::PlatformEvent::park() + 0xf0
0xfec88464    void ObjectMonitor::wait(long long,bool,Thread*) + 0x548
0xfe8cb974    void ObjectSynchronizer::wait(Handle,long long,Thread*) + 0x148
0xfe8cb508    JVM_MonitorWait + 0x29c
0xfc40e548    * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xfc40e4f4    * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xfc405a10    * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
... more lines removed here to reduce output...
----------------- t@12 -----------------
0xff2bfe3c    __lwp_park + 0x10
0xfe9925e4    AttachOperation*AttachListener::dequeue() + 0x148
0xfe99115c    void attach_listener_thread_entry(JavaThread*,Thread*) + 0x1fc
0xfec99ad8    void JavaThread::thread_main_inner() + 0x48
0xfec139f4    java_start + 0x118
0xff2bfd9c    _lwp_start
----------------- t@13 -----------------
0xff2c1500    _door_return + 0xc
----------------- t@14 -----------------
0xff2c1500    _door_return + 0xc

Frames that are prefixed with an asterisk (*) are Java frames, whereas frames that are not prefixed with an asterisk are native C/C++ frames.

The output of the utility can be piped through c++filt to demangle C++ mangled symbol names. Because the Java HotSpot VM is developed in the C++ language, the jstack utility prints C++ mangled symbol names for the Java HotSpot internal functions.

The c++filt utility is delivered with the native C++ compiler suite: SUNWspro on Oracle Solaris operating system and gnu on Linux.

The jstat Utility

The jstat utility uses the built-in instrumentation in the Java HotSpot VM to provide information about performance and resource consumption of running applications. The tool can be used when diagnosing performance issues, and in particular issues related to heap sizing and garbage collection. The jstat utility does not require the VM to be started with any special options. The built-in instrumentation in the Java HotSpot VM is enabled by default. This utility is included in the JDK download for all operating system platforms supported by Oracle.

Note:

The instrumentation is not accessible on a FAT32 file system.

The jstat Utility Options

-class: Specify this option to print statistics of the behavior of the class loader.
-compiler: Specify this option to print statistics of the behavior of the Java HotSpot compiler.
-gc: Specify this option to print statistics of the behavior of the garbage collected heap.
-gccapacity: Specify this option to print statistics of the capacities of the generations and their corresponding spaces.
-gccause: Specify this option to print the summary of garbage collection statistics (same as -gcutil), with the cause of the last and current (if applicable) garbage collection events.
-gcnew: Specify this option to print statistics of the behavior of the new generation.
-gcnewcapacity: Specify this option to print statistics of the sizes of the new generations and their corresponding spaces.
-gcold: Specify this option to print statistics of the behavior of the old and permanent generations.
-gcoldcapacity: Specify this option to print statistics of the sizes of the old generation.
-gcmetacapacity: Specify this option to print statistics of the sizes of the metaspace.
-gcutil: Specify this option to print a summary of garbage collection statistics.
-printcompilation: Specify this option to print Java HotSpot compilation method statistics.

For more information about the jstat utility, see the man page at

http://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html

The documentation includes a number of examples, and a few of those examples are repeated here in this document.

The jstat utility uses the virtual machine identifier (VMID) to identify the target process. The documentation describes the syntax of the VMID, but its only required component is the local virtual machine identifier (LVMID). The LVMID is typically (but not always) the operating system's PID for the target JVM process.

The jstat tool provides data similar to the data provided by the tools vmstat and iostat on Oracle Solaris and Linux operating systems.

For a graphical representation of the data, you can use the visualgc tool. See "The visualgc Tool."

The jstat Utility Examples

Example 1Using the gcutil option: The following example illustrates the use of -gcutil option where the jstat utility attaches to LVMID number 2834, takes seven samples at 250-millisecond intervals, and displays the output.

$ jstat -gcutil 2834 250 7 S0 S1 E O M YGC YGCT FGC FGCT GCT 0.00 99.74 13.49 7.86 95.82 3 0.124 0 0.000 0.124 0.00 99.74 13.49 7.86 95.82 3 0.124 0 0.000 0.124 0.00 99.74 13.49 7.86 95.82 3 0.124 0 0.000 0.124 0.00 99.74 13.49 7.86 95.82 3 0.124 0 0.000 0.124 0.00 99.74 13.49 7.86 95.82 3 0.124 0 0.000 0.124 0.00 99.74 13.49 7.86 95.82 3 0.124 0 0.000 0.124 0.00 99.74 13.49 7.86 95.82 3 0.124 0 0.000 0.124

The output of this example shows that a young generation collection occurred between the third and fourth samples. The collection took 0.017 seconds and promoted objects from the eden space (E) to the old space (O), resulting in an increase of old space utilization from 46.56% to 54.60%.
Example 2Using the gcnew option: The following example illustrates the use of the -gcnew option where the jstat utility attaches to LVMID number 2834, takes samples at 250-millisecond intervals, and displays the output. In addition, it uses the -h3 option to display the column headers after every three lines of data.

$ jstat -gcnew -h3 2834 250 S0C S1C S0U S1U TT MTT DSS EC EU YGC YGCT 192.0 192.0 0.0 0.0 15 15 96.0 1984.0 942.0 218 1.999 192.0 192.0 0.0 0.0 15 15 96.0 1984.0 1024.8 218 1.999 192.0 192.0 0.0 0.0 15 15 96.0 1984.0 1068.1 218 1.999 S0C S1C S0U S1U TT MTT DSS EC EU YGC YGCT 192.0 192.0 0.0 0.0 15 15 96.0 1984.0 1109.0 218 1.999 192.0 192.0 0.0 103.2 1 15 96.0 1984.0 0.0 219 2.019 192.0 192.0 0.0 103.2 1 15 96.0 1984.0 71.6 219 2.019 S0C S1C S0U S1U TT MTT DSS EC EU YGC YGCT 192.0 192.0 0.0 103.2 1 15 96.0 1984.0 73.7 219 2.019 192.0 192.0 0.0 103.2 1 15 96.0 1984.0 78.0 219 2.019 192.0 192.0 0.0 103.2 1 15 96.0 1984.0 116.1 219 2.019

In addition to showing the repeating header string, this example shows that between the fourth and fifth samples, a young generation collection occurred, whose duration was 0.02 seconds. The collection found enough live data that the survivor space 1 utilization (S1U) would have exceeded the desired survivor size (DSS). As a result, objects were promoted to the old generation (not visible in this output), and the tenuring threshold (TT) was lowered from 15 to 1.
Example 3Using the gcoldcapacity option: The following example illustrates the use of the -gcoldcapacity option where the jstat utility attaches to LVMID number 21891 and takes three samples at 250-millisecond intervals. The -t option is used to generate a time stamp for each sample in the first column.

$ jstat -gcoldcapacity -t 21891 250 3 Timestamp OGCMN OGCMX OGC OC YGC FGC FGCT GCT 150.1 1408.0 60544.0 11696.0 11696.0 194 80 2.874 3.799 150.4 1408.0 60544.0 13820.0 13820.0 194 81 2.938 3.863 150.7 1408.0 60544.0 13820.0 13820.0 194 81 2.938 3.863

The Timestamp column reports the elapsed time in seconds since the start of the target JVM. In addition, the -gcoldcapacity output shows the old generation capacity (OGC) and the old space capacity (OC) increasing as the heap expands to meet allocation or promotion demands. The OGC has grown from 11696 KB to 13820 KB after the 81^st full generation capacity (FGC). The maximum capacity of the generation (and space) is 60544 KB (OGCMX), so it still has room to expand.

Control+Break Handler

On Oracle Solaris or Linux operating systems, the combination of pressing the Control key and the backslash (\) key at the application console (standard input) causes the Java HotSpot VM to print a thread dump to the application's standard output. On Windows, the equivalent key sequence is the Control and Break keys. The general term for these key combinations is the Control+Break handler.

On Oracle Solaris and Linux operating systems, a thread dump is printed if the Java process receives a QUIT signal. Therefore, the kill -QUIT pid command causes the process with the ID pid to print a thread dump to standard output.

The following sections describe the data printed by the Control+Break handler:

Thread Dump
Detected Deadlocks
Heap Summary

Thread Dump

The thread dump consists of the thread stack, including thread state, for all Java threads in the virtual machine. The thread dump does not terminate the application: it continues after the thread information is printed.

The following example illustrates a thread dump.

Full thread dump Java HotSpot(TM) Client VM (1.6.0-rc-b100 mixed mode):

"DestroyJavaVM" prio=10 tid=0x00030400 nid=0x2 waiting on condition [0x00000000..0xfe77fbf0]
   java.lang.Thread.State: RUNNABLE

"Thread2" prio=10 tid=0x000d7c00 nid=0xb waiting for monitor entry [0xf36ff000..0xf36ff8c0]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at Deadlock$DeadlockMakerThread.run(Deadlock.java:32)
        - waiting to lock <0xf819a938> (a java.lang.String)
        - locked <0xf819a970> (a java.lang.String)

"Thread1" prio=10 tid=0x000d6c00 nid=0xa waiting for monitor entry [0xf37ff000..0xf37ffbc0]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at Deadlock$DeadlockMakerThread.run(Deadlock.java:32)
        - waiting to lock <0xf819a970> (a java.lang.String)
        - locked <0xf819a938> (a java.lang.String)

"Low Memory Detector" daemon prio=10 tid=0x000c7800 nid=0x8 runnable [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x000c5400 nid=0x7 waiting on condition [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x000c4400 nid=0x6 waiting on condition [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x000b2800 nid=0x5 in Object.wait() [0xf3f7f000..0xf3f7f9c0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xf4000b40> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0xf4000b40> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x000ae000 nid=0x4 in Object.wait() [0xfe57f000..0xfe57f940]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xf4000a40> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0xf4000a40> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x000ab000 nid=0x3 runnable 

"VM Periodic Task Thread" prio=10 tid=0x000c8c00 nid=0x9 waiting on condition

The output consists of a number of thread entries separated by an empty line. The Java Threads (threads that are capable of executing Java language code) are printed first, and these are followed by information about VM internal threads. Each thread entry consists of a header line followed by the thread stack trace.

The header line contains the following information about the thread:

Thread name
Indication if the thread is a daemon thread
Thread priority (prio)
Thread ID (tid), which is the address of a thread structure in memory
ID of the native thread (nid)
Thread state, which indicates what the thread was doing at the time of the thread dump (see Table 1-8)
Address range, which gives an estimate of the valid stack region for the thread

Table 1-8 lists the possible thread states that can be printed.

Table 1-5 Thread States

Thread State	Description
NEW	The thread has not yet started.
RUNNABLE	The thread is executing in the JVM.
BLOCKED	The thread is blocked waiting for a monitor lock.
WAITING	The thread is waiting indefinitely for another thread to perform a particular action.
TIMED_WAITING	The thread is waiting for another thread to perform an action for up to a specified waiting time.
TERMINATED	The thread has exited.

Detected Deadlocks

In addition to the thread stacks, the Control+Break handler executes a deadlock detection algorithm. If any deadlocks are detected, the Control+Break handler prints additional information after the thread dump on each deadlocked thread.

Found one Java-level deadlock:
=============================
"Thread2":
  waiting to lock monitor 0x000af330 (object 0xf819a938, a java.lang.String),
  which is held by "Thread1"
"Thread1":
  waiting to lock monitor 0x000af398 (object 0xf819a970, a java.lang.String),
  which is held by "Thread2"

Java stack information for the threads listed above:
===================================================
"Thread2":
        at Deadlock$DeadlockMakerThread.run(Deadlock.java:32)
        - waiting to lock <0xf819a938> (a java.lang.String)
        - locked <0xf819a970> (a java.lang.String)
"Thread1":
        at Deadlock$DeadlockMakerThread.run(Deadlock.java:32)
        - waiting to lock <0xf819a970> (a java.lang.String)
        - locked <0xf819a938> (a java.lang.String)

Found 1 deadlock.

If the JVM flag -XX:+PrintConcurrentLocks is set, then the Control+Break handler will also print the list of concurrent locks owned by each thread.

Heap Summary

The Control+Break handler also prints a heap summary. This output shows the different generations (areas of the heap), with the size, the amount used, and the address range. The address range is especially useful if you are also examining the process with tools such as pmap.

Heap
 def new generation   total 1152K, used 435K [0x22960000, 0x22a90000, 0x22e40000
)
  eden space 1088K,  40% used [0x22960000, 0x229ccd40, 0x22a70000)
  from space 64K,   0% used [0x22a70000, 0x22a70000, 0x22a80000)
  to   space 64K,   0% used [0x22a80000, 0x22a80000, 0x22a90000)
 tenured generation   total 13728K, used 6971K [0x22e40000, 0x23ba8000, 0x269600
00)
   the space 13728K,  50% used [0x22e40000, 0x2350ecb0, 0x2350ee00, 0x23ba8000)
 compacting perm gen  total 12288K, used 1417K [0x26960000, 0x27560000, 0x2a9600
00)
   the space 12288K,  11% used [0x26960000, 0x26ac24f8, 0x26ac2600, 0x27560000)
    ro space 8192K,  62% used [0x2a960000, 0x2ae5ba98, 0x2ae5bc00, 0x2b160000)
    rw space 12288K,  52% used [0x2b160000, 0x2b79e410, 0x2b79e600, 0x2bd60000)

If the JVM flag -XX:+PrintClassHistogram is set, then the Control+Break handler will produce a heap histogram.

Native Operating System Tools

This section lists a number of native tools available on Windows, Linux, and Oracle Solaris operating systems that are useful for troubleshooting or monitoring purposes. A brief description is provided for each tool. For further details, refer to the operating system documentation (or man pages for Oracle Solaris and Linux operating systems).

The format of log files and output from command-line utilities depends on the release. For example, if you develop a script that relies on the format of the fatal error log, then the same script may not work if the format of the log file changes in a future release.

Table 1-6 Native Troubleshooting Tools on Windows

Tool	Description
`dumpchk`	Command-line utility to verify that a memory dump file has been created correctly. This tool is included in the Debugging Tools for Windows download available from the Microsoft website (see "Collecting Crash Dumps on Windows."
`msdev` debugger	Command-line utility that can be used to launch Visual C++ and the Win32 debugger
`userdump`	The User Mode Process Dumper is included in the OEM Support Tools download available from the Microsoft website (see "Collecting Crash Dumps on Windows.")
`windbg`	Windows debugger can be used to debug Windows applications or crash dumps. This tool is included in the Debugging Tools for Windows download available from the Microsoft website (see "Collecting Crash Dumps on Windows.")
`/Md` and `/Mdd` compiler options	Compiler options that automatically include extra support for tracking memory allocations

You can also search for Windows-specific debug support on MSDN at

http://msdn.microsoft.com

Table 1-7 describes some troubleshooting tools introduced or improved in Linux operating system version 10.

Table 1-7 Native Troubleshooting Tools on Linux

Tool	Description
`c++filt`	Demangle C++ mangled symbol names. This utility is delivered with the native C++ compiler suite: `gcc` on Linux.
`gdb`	GNU debugger
`libnjamd`	Memory allocation tracking
`lsstack`	Print thread stack (similar to `pstack` in Oracle Solaris operating system) Not all distributions provide this tool by default; therefore, you might have to download it from `http://sourceforge.net`
`ltrace`	Library call tracer (equivalent to `truss -u` in Oracle Solaris operating system) Not all distributions provide this tool by default; therefore, you might have to download it from `http://sourceforge.net`
`mtrace` and `muntrace`	GNU `malloc` tracer
`proc` tools such as `pmap` and `pstack`	Some, but not all, of the `proc` tools on Oracle Solaris operating system have equivalent tools on Linux. Core file support is not as good for Linux as for Oracle Solaris operating system; for example, `pstack` does not work for core dumps
`strace`	System call tracer (equivalent to `truss -t` in Oracle Solaris operating system)
`top`	Display most CPU-intensive processes.
`vmstat`	Report information about processes, memory, paging, block I/O, traps, and CPU activity.

Table 1-8 lists troubleshooting tools available on Oracle Solaris operating system.

Table 1-8 Native Troubleshooting Tools on Oracle Solaris operating system

Tool	Description
`coreadm`	Specify name and location of core files produced by the JVM.
`cpustat`	Monitor system behavior using CPU performance counters.
`cputrack`	Monitor process and LWP behavior using CPU performance counters.
`c++filt`	Demangle C++ mangled symbol names. This utility is delivered with the native C++ compiler suite: `SUNWspro` on Oracle Solaris operating system.
`dtrace`	Introduced in Oracle Solaris 10 operating system, DTrace is a dynamic tracing compiler and tracing utility. It can perform dynamic tracing of kernel functions, system calls, and user functions. This tool allows arbitrary, safe scripting to be executed at entry, exit, and other probe points. The script is written in C-like but safe pointer semantics language called the D programming language. See also "Using the DTrace Tool."
`gcore`	Force a core dump of a process. The process continues after the core dump is written.
`intrstat`	Report statistics on CPU consumed by interrupt threads.
`iostat`	Report I/O statistics.
`libumem`	Introduced in Oracle Solaris 9 operating system update 3, this library provides fast, scalable object-caching memory allocation and extensive debugging support. The tool can be used to find and fix memory management bugs (see "Using the libumem Tool to Find Leaks.")
`mdb`	Modular debugger for kernel and user applications and crash dumps
`netstat`	Display the contents of various network-related data structures.
`pargs`	Print process arguments, environment variables, or the auxiliary vector. Long output is not truncated as it would be by other commands, such as `ps`.
`pfiles`	Print information on process file descriptors. Starting with Oracle Solaris 10 operating system, the tool prints the file name also.
`pldd`	Print shared objects loaded by a process.
`pmap`	Print memory layout of a process or core file, including heap, data, and text sections. Starting with Oracle Solaris 10, stack segments are clearly identified with the text `[stack]` along with the thread ID. See also "Improvements in the pmap Tool."
`prstat`	Report statistics for active Oracle Solaris operating system processes. (Similar to `top`)
`prun`	Set the process to running mode (reverse of `pstop`).
`ps`	List all processes.
`psig`	List the signal handlers of a process.
`pstack`	Print stack of threads of a given process or core file. Starting with Oracle Solaris 10 operating system, Java method names can be printed for Java frames. See also "Improvements in the pstack Tool."
`pstop`	Stop the process (suspend).
`ptree`	Print process tree containing the given PID.
`sar`	System activity reporter
`sdtprocess`	Display most CPU-intensive processes. (Similar to `top`)
`sdtperfmeter`	Display graphs showing system performance (for example, CPU, disks, and network).
`top`	Display most CPU-intensive processes. This tool is available as freeware for Oracle Solaris operating system but is not installed by default.
`trapstat`	Display runtime trap statistics. (SPARC only)
`truss`	Trace entry and exit events for system calls, user-mode functions, and signals; optionally stop the process at one of these events. This tool also prints the arguments of system calls and user functions.
`vmstat`	Report system virtual memory statistics.
`watchmalloc`	Track memory allocations.

Using the DTrace Tool

Oracle Solaris 10 operating system includes the DTrace tool, which allows dynamic tracing of the operating system kernel and user-level programs. This tool supports scripting at system-call entry and exit, at user-mode function entry and exit, and at many other probe points. The scripts are written in the D programming language, which is a C-like language with safe pointer semantics. These scripts can help you in troubleshooting problems or solving performance issues.

The dtrace command is a generic front end to the DTrace tool. This command provides a simple interface to invoke the D language, to retrieve buffered trace data, and to access a set of basic routines to format and print traced data.

You can write your own customized DTrace scripts, using the D language, or download and use one or more of the many scripts that are already available on various sites.

The probes are delivered and instrumented by kernel modules called providers. The types of tracing offered by the probe providers include user instruction tracing, function boundary tracing, kernel lock instrumentation, profile interrupt, system call tracing, and many more. If you write your own scripts, you use the D language to enable the probes; this language also allows conditional tracing and output formatting.

You can use the dtrace -l command to explore the set of providers and probes that are available on your Oracle Solaris operating system.

The DTraceToolkit is a collection of useful documented scripts developed by the Open Oracle Solaris DTrace community. For more information about the DTraceToolkit, see the DTraceToolkit web page at

http://www.brendangregg.com/dtracetoolkit.html

For more information about dynamic tracing in general, see the Solaris Dynamic Tracing Guide at

http://docs.oracle.com/cd/E19253-01/817-6223/

Probe Providers in Java HotSpot VM

The Java HotSpot VM contains two built-in probe providers: hotspot and hotspot_jni. These providers deliver probes that can be used to monitor the internal state and activities of the VM, as well as the Java application that is running.

The JVM probe providers can be categorized as follows:

VM lifecycle: VM initialization begin and end, and VM shutdown
Thread lifecycle: thread start and stop, thread name, thread ID, and so on
Class-loading: Java class loading and unloading
Garbage collection: start and stop of garbage collection, systemwide or by memory pool
Method compilation: method compilation begin and end, and method loading and unloading
Monitor probes: wait events, notification events, contended monitor entry and exit
Application tracking: method entry and return, allocation of a Java object

In order to call from native code to Java code, the native code must make a call through the JNI interface. The hotspot_jni provider manages DTrace probes at the entry point and return point for each of the methods that the JNI interface provides for invoking Java code and examining the state of the VM.

Example 1-1 Example of Using DTrace

At probe points, you can print the stack trace current thread using the ustack built-in function. This function prints Java method names in addition to C/C++ native function names. Example 1-1 is a simple D script that prints a full stack trace whenever a thread calls the read system call.

#!/usr/sbin/dtrace -s
syscall::read:entry 
/pid == $1 && tid == 1/ {    
   ustack(50, 0x2000);
}

The script in example 1-1 is stored in a file named read.d and is run by specifying the PID of the Java process that is traced.

read.d pid

If your Java application generated a lot of I/O or had some unexpected latency, then the DTrace tool and its ustack() action can help you diagnose the problem.

Improvements in the pmap Tool

The pmap utility was improved in Oracle Solaris 10 operating system to print stack segments with the text [stack]. This text helps you to locate the stack easily.

The following example shows the output from this tool:

19846:    /net/myserver/export1/user/j2sdk6/bin/java -Djava.endorsed.d
00010000      72K r-x--  /export/disk09/jdk/6/rc/b63/binaries/solsparc/bin/java
00030000      16K rwx--  /export/disk09/jdk/6/rc/b63/binaries/solsparc/bin/java
00034000   32544K rwx--    [ heap ]
D1378000      32K rwx-R    [ stack tid=44 ]
D1478000      32K rwx-R    [ stack tid=43 ]
D1578000      32K rwx-R    [ stack tid=42 ]
D1678000      32K rwx-R    [ stack tid=41 ]
D1778000      32K rwx-R    [ stack tid=40 ]
D1878000      32K rwx-R    [ stack tid=39 ]
D1974000      48K rwx-R    [ stack tid=38 ]
D1A78000      32K rwx-R    [ stack tid=37 ]
D1B78000      32K rwx-R    [ stack tid=36 ]
[.. more lines removed here to reduce output ..]
FF370000       8K r-x--  /usr/lib/libsched.so.1
FF380000       8K r-x--  /platform/sun4u-us3/lib/libc_psr.so.1
FF390000      16K r-x--  /lib/libthread.so.1
FF3A4000       8K rwx--  /lib/libthread.so.1
FF3B0000       8K r-x--  /lib/libdl.so.1
FF3C0000     168K r-x--  /lib/ld.so.1
FF3F8000       8K rwx--  /lib/ld.so.1
FF3FA000       8K rwx--  /lib/ld.so.1
FFB80000      24K -----    [ anon ]
FFBF0000      64K rwx--    [ stack ]
 total    167224K

Improvements in the pstack Tool

Prior to Oracle Solaris 10 operating system, the pstack utility did not support Java. It printed hexadecimal addresses for both interpreted and compiled Java methods.

Starting with Oracle Solaris 10 operating system, the pstack command-line tool prints mixed mode stack traces (Java and C/C++ frames) from a core file or a live process. The tool prints Java method names for interpreted, compiled, and inlined Java methods.

Custom Diagnostic Tools

The JDK has extensive APIs for developing custom tools to observe, monitor, profile, debug, and diagnose issues in applications that are deployed in the JRE. The development of new tools is beyond the scope of this document. Instead this section provides a brief overview of the APIs available.

All the packages mentioned in this section are fully described in the Java SE API specification at

http://docs.oracle.com/javase/8/docs/api/index.html

Refer also to example and demonstration code that is included in the JDK download.

The java.lang.management Package

The java.lang.management package provides the management interface for monitoring and management of the JVM and the operating system. Specifically it covers interfaces for the following systems:

Class loading
Compilation
Garbage collection
Memory manager
Runtime
Threads

The JDK includes example code that demonstrates the usage of the java.lang.management package. These examples can be found in the $JAVA_HOME/demo/management directory. Some of these examples are as follows:

MemoryMonitor demonstrates the use of the java.lang.management API to observe the memory usage of all memory pools consumed by the application.
FullThreadDump demonstrates the use of the java.lang.management API to get a full thread dump and detect deadlocks programmatically.
VerboseGC demonstrates the use of the java.lang.management API to print the garbage collection statistics and memory usage of an application.

In addition to the java.lang.management package, the JDK release includes platform extensions in the com.sun.management package. The platform extensions include a management interface to obtain detailed statistics from garbage collectors that perform collections in cycles. These extensions also include a management interface to obtain additional memory statistics from the operating system.

For more information about the platform extensions, see the Java SE Monitoring and Management documentation at

http://docs.oracle.com/javase/8/docs/technotes/guides/management/index.html

The java.lang.instrument Package

The java.lang.instrument package provides services that allow Java programming language agents to instrument programs running on the JVM. Instrumentation is used by tools such as profilers, tools for tracing method calls, and many others. The package facilitates both load-time and dynamic instrumentation. It also includes methods to obtain information about the loaded classes and information about the amount of storage consumed by a given object.

The java.lang.Thread Class

The java.lang.Thread class has a static method called getAllStackTraces, which returns a map of stack traces for all live threads. The Thread class also has a method called getState, which returns the thread state; states are defined by the java.lang.Thread.State enumeration. These methods can be useful when you add diagnostic or monitoring capabilities to an application.

Java Virtual Machine Tool Interface

The JVM Tool Interface (JVM TI) is a native (C/C++) programming interface that can be used by a wide range of development and monitoring tools. JVM TI provides an interface for the full breadth of tools that need access to VM state, including but not limited to profiling, debugging, monitoring, thread analysis, and coverage analysis tools.

Some examples of agents that rely on JVM TI are the following:

HPROF
Java Debug Wire Protocol (JDWP)
The java.lang.instrument package

The specification for JVM TI can be found in the JVM Tool Interface documentation at

http://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html

The JDK includes example code that demonstrates the usage of JVM TI. These examples can be found in the $JAVA_HOME/demo/jvmti directory. Some of the examples are as follows:

mtrace is an agent library that tracks method call and return counts. It uses bytecode instrumentation to instrument all classes loaded into the virtual machine and prints a sorted list of the frequently used methods.
heapTracker is an agent library that tracks object allocation. It uses bytecode instrumentation to instrument constructor methods.
heapViewer is an agent library that prints heap statistics when the Control+Break handler is invoked (for more information, see "Control+Break Handler.") For each loaded class it prints an instance count of that class and the space used.

Java Platform Debugger Architecture

The Java Platform Debugger Architecture (JPDA) is the architecture designed for use by debuggers and debugger-like tools. It consists of two programming interfaces and a wire protocol.

The Java Virtual Machine Tool Interface (JVM TI) is the interface to the virtual machine (see "Java Virtual Machine Tool Interface.")
The Java Debug Interface (JDI) defines information and requests at the user code level. It is a pure Java programming language interface for debugging Java programming language applications. In JPDA, the JDI is a remote view in the debugger process of a virtual machine in the process being debugged. It is implemented by the front end, where as a debugger-like application (for example, IDE, debugger, tracer, or monitoring tool) is the client.
The Java Debug Wire Protocol (JDWP) defines the format of information and requests transferred between the process being debugged and the debugger front end, which implements the JDI.

For a complete description of Java Platform Debugger Architecture (JDPA) connection and invocation details, see the document at

http://docs.oracle.com/javase/8/docs/technotes/guides/jpda/conninv.html

The jdb utility is included in the JDK as an example command-line debugger. The jdb utility uses the JDI to launch or connect to the target VM. For more information about the jdb utility, see "The jdb Utility."

In addition to traditional debugger-type tools, the JDI can also be used to develop tools that help in postmortem diagnostics and scenarios where the tool needs to attach to a process in a noncooperative manner (for example, a hung process).

1 Diagnostic Tools and Detailed Descriptions

Diagnostic Tools - Overview

Postmortem Diagnostics Tools

Hung Processes Tools

Monitoring Tools

Other Tools, Options, Variables, and Properties

Java Mission Control

How to Produce a Flight Recording

The jcmd Utility

Useful Commands for jcmd Utility

Troubleshooting Options for jcmd Utility

HPROF

Heap Allocation Profile heap=sites

Heap Dump Profile heap=dump

CPU Usage Sampling Profile cpu=samples

CPU Usage Times Profile cpu=times

JConsole

Java VisualVM

The jdb Utility

Attaching to a Process

Attaching to a Core File on the Same Machine

Attaching to a Core File or a Hung Process from a Different Machine

The jhat Utility

Standard Queries

Custom Queries

Heap Analysis Hints

The jinfo Utility

The jmap Utility

Heap Configuration and Usage

Heap Histogram

Permanent Generation Statistics

The jps Utility

The jrunscript Utility

The jsadebugd Daemon

The jstack Utility

Forcing a Stack Dump

Printing Stack Trace from a Core Dump

Printing a Mixed Stack

The jstat Utility

The jstatd Daemon

The visualgc Tool

Control+Break Handler

Thread Dump

Detected Deadlocks

Heap Summary

Native Operating System Tools

Using the DTrace Tool

Improvements in the pmap Tool

Improvements in the pstack Tool

Custom Diagnostic Tools

The java.lang.management Package

The java.lang.instrument Package

The java.lang.Thread Class

Java Virtual Machine Tool Interface

Java Platform Debugger Architecture