Java Commands useful for troubleshooting

1. Jinfo -flags < process id >  to print all the JVM options explicitly specified when the process was started.

It will be useful to examine the JVM GC settings.

example:-

Ranger admin process:

jinfo -flags 57374

Attaching to process ID 57374, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15
Non-default VM flags: -XX:CICompilerCount=15 -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=357564416 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=357564416 -XX:OldSize=716177408 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseParallelGC
Command line:  -Dproc_rangeradmin -XX:MaxPermSize=256m -Xmx1024m -Xms1024m -Duser.timezone=UTC -Dservername=rangeradmin -Dlogdir=/var/log/ranger/admin -Dcatalina.base=/usr/hdp/2.6.5.0-292/ranger-admin/ews

2.  To print Java System properties where the JVM is running,

jinfo -sysprops 57374

Attaching to process ID 57374, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15
java.runtime.name = Java(TM) SE Runtime Environment
java.vm.version = 25.112-b15
sun.boot.library.path = /usr/jdk64/jdk1.8.0_112/jre/lib/amd64
java.vendor.url = http://java.oracle.com/
java.vm.vendor = Oracle Corporation
path.separator = :
file.encoding.pkg = sun.io
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
sun.os.patch.level = unknown
sun.java.launcher = SUN_STANDARD
user.country = US
user.dir = /usr/hdp/2.6.5.0-292/ranger-admin/ews
java.vm.specification.name = Java Virtual Machine Specification
java.runtime.version = 1.8.0_112-b15
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
os.arch = amd64
java.endorsed.dirs = /usr/jdk64/jdk1.8.0_112/jre/lib/endorsed
line.separator =

java.io.tmpdir = /tmp
proc_rangeradmin =
webapp.root = /usr/hdp/2.6.5.0-292/ranger-admin/ews/webapp/
java.vm.specification.vendor = Oracle Corporation
os.name = Linux
servername = rangeradmin
sun.jnu.encoding = UTF-8
javax.net.ssl.keyStorePassword =
java.library.path = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
javax.net.ssl.trustStore = /etc/ranger/admin/conf/ranger-admin-keystore.jks
java.class.version = 52.0
java.specification.name = Java Platform API Specification
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
os.version = 3.10.0-862.9.1.el7.x86_64
user.home = /home/ranger
user.timezone = UTC
catalina.useNaming = false
java.awt.printerjob = sun.print.PSPrinterJob
file.encoding = UTF-8
logdir = /var/log/ranger/admin
java.specification.version = 1.8
javax.net.ssl.trustStoreType = jks
catalina.home = /usr/hdp/2.6.5.0-292/ranger-admin/ews
user.name = ranger
java.class.path = /usr/hdp/2.6.5.0-292/ranger-admin/ews/webapp/WEB-INF/classes/conf:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/commons-configuration-1.10.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/commons-logging-1.2.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/ecj-P20140317-1600.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/embeddedwebserver-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/guava-17.0.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/hadoop-auth-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/hadoop-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/ranger-plugins-common-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/tomcat-annotations-api-7.0.82.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/tomcat-embed-core-7.0.82.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/tomcat-embed-el-7.0.82.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/tomcat-embed-jasper-7.0.82.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/tomcat-embed-logging-juli-7.0.82.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/tomcat-embed-logging-log4j-7.0.82.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/tomcat-embed-websocket-7.0.82.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/ranger_jaas/unixauthclient-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/ranger-admin/ews/webapp/WEB-INF/classes/conf/ranger_jaas:/usr/jdk64/jdk1.8.0_112/lib/javafx-mx.jar:/usr/jdk64/jdk1.8.0_112/lib/packager.jar:/usr/jdk64/jdk1.8.0_112/lib/tools.jar:/usr/jdk64/jdk1.8.0_112/lib/ant-javafx.jar:/usr/jdk64/jdk1.8.0_112/lib/sa-jdi.jar:/usr/jdk64/jdk1.8.0_112/lib/dt.jar:/usr/jdk64/jdk1.8.0_112/lib/jconsole.jar:/*:
java.vm.specification.version = 1.8
sun.arch.data.model = 64
sun.java.command = org.apache.ranger.server.tomcat.EmbeddedServer
java.home = /usr/jdk64/jdk1.8.0_112/jre
user.language = en
java.specification.vendor = Oracle Corporation
awt.toolkit = sun.awt.X11.XToolkit
java.vm.info = mixed mode
java.version = 1.8.0_112
java.ext.dirs = /usr/jdk64/jdk1.8.0_112/jre/lib/ext:/usr/java/packages/lib/ext
sun.boot.class.path = /usr/jdk64/jdk1.8.0_112/jre/lib/resources.jar:/usr/jdk64/jdk1.8.0_112/jre/lib/rt.jar:/usr/jdk64/jdk1.8.0_112/jre/lib/sunrsasign.jar:/usr/jdk64/jdk1.8.0_112/jre/lib/jsse.jar:/usr/jdk64/jdk1.8.0_112/jre/lib/jce.jar:/usr/jdk64/jdk1.8.0_112/jre/lib/charsets.jar:/usr/jdk64/jdk1.8.0_112/jre/lib/jfr.jar:/usr/jdk64/jdk1.8.0_112/jre/classes
java.vendor = Oracle Corporation
catalina.base = /usr/hdp/2.6.5.0-292/ranger-admin/ews
java.security.auth.login.config = /dev/null
file.separator = /
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
sun.io.unicode.encoding = UnicodeLittle
sun.cpu.endian = little
javax.net.ssl.trustStorePassword = _
javax.security.auth.useSubjectCredsOnly = false
sun.cpu.isalist =


To Analyze GC Performance stats of a JVM:

jcmd  < process id >  PerfCounter.print |grep -i 'sun.gc'

example:
jcmd 230530 PerfCounter.print |grep -i 'sun.gc'

sun.gc.cause="No GC"
sun.gc.collector.0.invocations=730
sun.gc.collector.0.lastEntryTime=207628337064585
sun.gc.collector.0.lastExitTime=207628347690432
sun.gc.collector.0.name="PSScavenge"
sun.gc.collector.0.time=12846480737
sun.gc.collector.1.invocations=6
sun.gc.collector.1.lastEntryTime=207473351317925
sun.gc.collector.1.lastExitTime=207474340258374
sun.gc.collector.1.name="PSParallelCompact"
sun.gc.collector.1.time=3276755423
sun.gc.compressedclassspace.capacity=9306112
sun.gc.compressedclassspace.maxCapacity=1073741824
sun.gc.compressedclassspace.minCapacity=0
sun.gc.compressedclassspace.used=8960720
sun.gc.generation.0.capacity=198180864
sun.gc.generation.0.maxCapacity=2684354560
sun.gc.generation.0.minCapacity=111673344
sun.gc.generation.0.name="new"
sun.gc.generation.0.space.0.capacity=166723584
sun.gc.generation.0.space.0.initCapacity=0
sun.gc.generation.0.space.0.maxCapacity=2683305984
sun.gc.generation.0.space.0.name="eden"
sun.gc.generation.0.space.0.used=13092376
sun.gc.generation.0.space.1.capacity=14155776
sun.gc.generation.0.space.1.initCapacity=0
sun.gc.generation.0.space.1.maxCapacity=894435328
sun.gc.generation.0.space.1.name="s0"
sun.gc.generation.0.space.1.used=4941728
sun.gc.generation.0.space.2.capacity=15728640
sun.gc.generation.0.space.2.initCapacity=0
sun.gc.generation.0.space.2.maxCapacity=894435328
sun.gc.generation.0.space.2.name="s1"
sun.gc.generation.0.space.2.used=0
sun.gc.generation.0.spaces=3
sun.gc.generation.1.capacity=970981376
sun.gc.generation.1.maxCapacity=5368709120
sun.gc.generation.1.minCapacity=223870976
sun.gc.generation.1.name="old"
sun.gc.generation.1.space.0.capacity=970981376
sun.gc.generation.1.space.0.initCapacity=223870976
sun.gc.generation.1.space.0.maxCapacity=5368709120
sun.gc.generation.1.space.0.name="old"
sun.gc.generation.1.space.0.used=352206136
sun.gc.generation.1.spaces=1
sun.gc.lastCause="Allocation Failure"
sun.gc.metaspace.capacity=89292800
sun.gc.metaspace.maxCapacity=1155530752
sun.gc.metaspace.minCapacity=0
sun.gc.metaspace.used=87968584
sun.gc.policy.avgBaseFootprint=268435456
sun.gc.policy.avgMajorIntervalTime=32211109
sun.gc.policy.avgMajorPauseTime=491
sun.gc.policy.avgMinorIntervalTime=172683
sun.gc.policy.avgMinorPauseTime=11
sun.gc.policy.avgOldLive=218736016
sun.gc.policy.avgPretenuredPaddedAvg=0
sun.gc.policy.avgPromotedAvg=633612
sun.gc.policy.avgPromotedDev=171678
sun.gc.policy.avgPromotedPaddedAvg=1148648
sun.gc.policy.avgSurvivedAvg=11853649
sun.gc.policy.avgSurvivedDev=1154625
sun.gc.policy.avgSurvivedPaddedAvg=15317526
sun.gc.policy.avgYoungLive=11723719
sun.gc.policy.boundaryMoved=0
sun.gc.policy.changeOldGenForMajPauses=0
sun.gc.policy.changeOldGenForMinPauses=0
sun.gc.policy.changeYoungGenForMajPauses=0
sun.gc.policy.changeYoungGenForMinPauses=0
sun.gc.policy.collectors=2
sun.gc.policy.decideAtFullGc=0
sun.gc.policy.decreaseForFootprint=6
sun.gc.policy.decrementTenuringThresholdForGcCost=0
sun.gc.policy.decrementTenuringThresholdForSurvivorLimit=0
sun.gc.policy.desiredSurvivorSize=15728640
sun.gc.policy.edenSize=166723584
sun.gc.policy.freeSpace=713555968
sun.gc.policy.fullFollowsScavenge=0
sun.gc.policy.gcTimeLimitExceeded=0
sun.gc.policy.generations=3
sun.gc.policy.increaseOldGenForThroughput=0
sun.gc.policy.increaseYoungGenForThroughput=0
sun.gc.policy.incrementTenuringThresholdForGcCost=1
sun.gc.policy.liveAtLastFullGc=352206136
sun.gc.policy.liveSpace=498895168
sun.gc.policy.majorCollectionSlope=0
sun.gc.policy.majorGcCost=0
sun.gc.policy.majorPauseOldSlope=3578
sun.gc.policy.majorPauseYoungSlope=88
sun.gc.policy.maxTenuringThreshold=15
sun.gc.policy.minorCollectionSlope=0
sun.gc.policy.minorGcCost=0
sun.gc.policy.minorPauseOldSlope=-117
sun.gc.policy.minorPauseTime=10
sun.gc.policy.minorPauseYoungSlope=45
sun.gc.policy.mutatorCost=99
sun.gc.policy.name="ParScav:MSC"
sun.gc.policy.oldCapacity=970981376
sun.gc.policy.oldEdenSize=168296448
sun.gc.policy.oldPromoSize=546832384
sun.gc.policy.promoSize=546832384
sun.gc.policy.promoted=0
sun.gc.policy.scavengeSkipped=0
sun.gc.policy.survived=4941728
sun.gc.policy.survivorOverflowed=0
sun.gc.policy.tenuringThreshold=15
sun.gc.policy.youngCapacity=182452224
sun.gc.tlab.alloc=20637522
sun.gc.tlab.allocThreads=23
sun.gc.tlab.fastWaste=0
sun.gc.tlab.fills=452
sun.gc.tlab.gcWaste=58401
sun.gc.tlab.maxFastWaste=0
sun.gc.tlab.maxFills=59
sun.gc.tlab.maxGcWaste=20031
sun.gc.tlab.maxSlowAlloc=46
sun.gc.tlab.maxSlowWaste=5741
sun.gc.tlab.slowAlloc=144
sun.gc.tlab.slowWaste=8771

To get most used classes in your JVM :-

For example in hiveserver2 pid, It gives the memory space it takes in bytes

jcmd 230530 GC.class_histogram

 num     #instances         #bytes  class name
----------------------------------------------
   1:        615170       43494440  [Ljava.lang.Object;
   2:        660200       37493616  [C
   3:        889649       21351576  java.util.ArrayList
   4:        556042       17793344  java.util.HashMap$Node
   5:        659400       15825600  java.lang.String
   6:        156852       13022176  [Ljava.util.HashMap$Node;
   7:        193420        9284160  java.util.HashMap
   8:        110787        5317776  org.apache.ranger.plugin.resourcematcher.RangerDefaultResourceMatcher
   9:        143140        4580480  java.util.Hashtable$Entry
  10:        184702        4432848  org.apache.ranger.plugin.model.RangerPolicy$RangerPolicyItemAccess
  11:         36931        3840824  org.apache.ranger.plugin.model.RangerPolicy
  12:         36931        3545376  org.apache.ranger.plugin.policyevaluator.RangerOptimizedPolicyEvaluator