Sometimes there is an urgent need popping up in the middle of an investigation of what is happening with your application server – e.g., a process is showing up, but refuses to serve network connections. There could be thousands of reasons why it is stuck, but one of the most common and a classic problem is deadlocked threads, i.e. threads that didn't share some of the resources on start-up correctly.
There are actually four explicit methods to solve this puzzle, or at least help us shed some light on it:
1) Sending SIGQUIT signal to process (kill -3 on Linux), using jstack utility or visualvm (+ its thread monitoring plugin) to make a thread dump and then analyze it manually.
2) If a remote machine has jmx enabled, then we can simply do the following:
JMXServiceURL serviceURL = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:15001/jmxrmi");
ThreadMXBean threadMXBean = ManagementFactory.newPlatformMXBeanProxy(JMXConnectorFactory.connect(serviceURL).getMBeanServerConnection(), ManagementFactory.THREAD_MXBEAN_NAME, ThreadMXBean.class);
threadMXBean.findDeadlockedThreads(); // synchronized blocks or ownable synchronizer
threadMXBean.findMonitorDeadlockedThreads(); // synchronized blocks
…or, even simpler, use Jconsole.
3) If jmx is not enabled on the remote server in question, then we can do the following trick of using java Attach API to export platform MBeanServer remotely, and then return to option 2:
try {
VirtualMachine virtualMachine = VirtualMachine.attach("PID");
String home = virtualMachine.getSystemProperties().getProperty("java.home");
String agent = home + File.separator + "lib" + File.separator + "management-agent.jar";
virtualMachine.loadAgent(agent, "com.sun.management.jmxremote.port=15001," +"com.sun.management.jmxremote.authenticate=false,com.sun.management.jmxremote.ssl=false");
virtualMachine.detach();
} catch (AttachNotSupportedException | AgentLoadException | AgentInitializationException e) {
e.printStackTrace();
}
4) But what if -XX:+DisableAttachMechanism flag option was applied, and remote access with Attach API was forbidden, or what if we need some more refined information of what’s happening inside? Seems we are stuck at the moment, but not today, because we have a solution based on java Serviceability API. Serviceability API is very similar to Reflection API, but it differs in that scanning of objects is happening from outside world, just like if you were using an X-ray, and all the objects we can iterate are actually mirrored to native C++ JVM model. Sounds cool, isn't? First of all, we have to imitate a classic situation where deadlock is taking place:
final Object resource1 = "resource1";
final Object resource2 = "resource2";
final Phaser phaser = new Phaser(2);Thread thread1 = new Thread(new Runnable() {
@Override
public void run() {
synchronized (resource1) {
phaser.arriveAndAwaitAdvance();
synchronized(resource2) {
System.out.println("Thread 1: locked resource 2");
}
}
}
});
thread1.setName("Thread1");Thread thread2 = new Thread(new Runnable() {
@Override
public void run() {
synchronized (resource2) {
phaser.arriveAndAwaitAdvance();
synchronized(resource1) {
System.out.println("Thread 2: locked resource 1");
}
}}
});
thread2.setName("Thread2");thread1.start();
thread2.start();
Then we have to extend sun.jvm.hotspot.tools.Tool (which comes from sa-jdi.jar packaging with JDK installation), like this:
public class DeadlockFinder extends Tool {
public void run() {
// our code goes here
}
And start using it this way:
DeadlockFinder deadlockFinder = new DeadlockFinder();
deadlockFinder.setDebugeeType(Tool.DEBUGEE_PID);
deadlockFinder.start(new String[] {"PID"});
Let's also involve a couple of helper classes, which can simplify our later traversals:
static class Monitor {
String className;
Object object;
ThreadStruct owningThread;
List pendingThreads = new ArrayList<>();
void setOwningThread(JavaThread owningThread) {
this.owningThread = new ThreadStruct(owningThread);
}
void setPendingThreadNames(List pendingThreads) {
for (Object pendingThread : pendingThreads) {
JavaThread javaThread = (JavaThread) pendingThread;
this.pendingThreads.add(new ThreadStruct(javaThread));
}
}
}
static class ThreadStruct {
String threadName;
long threadId;
long nativeThreadId;
String stackTrace;
ThreadStruct(JavaThread javaThread) {
this.threadName = javaThread.getThreadName();
this.threadId = getTid(javaThread);
this.nativeThreadId = Long.parseLong(javaThread.getThreadProxy().toString());
JavaVFrame lastJavaVFrameDbg = javaThread.getLastJavaVFrameDbg();
StringBuilder stringBuilder = new StringBuilder();
for (;lastJavaVFrameDbg != null; lastJavaVFrameDbg = lastJavaVFrameDbg.javaSender()) {
printJavaFrame(stringBuilder, lastJavaVFrameDbg);
}
this.stackTrace = stringBuilder.toString();
}
void printJavaFrame(StringBuilder stringBuilder, JavaVFrame javaFrame) {
Method method = javaFrame.getMethod();
stringBuilder.append("\tat ");
InstanceKlass klass = (InstanceKlass) method.getMethodHolder();
String className = klass.getName().asString();
stringBuilder.append(className.replace('/','.'));
stringBuilder.append(".");
stringBuilder.append(method.getName().asString());
int bci = javaFrame.getBCI();
stringBuilder.append("(");
if (method.isNative()) {
stringBuilder.append("Native Method");
} else {
int lineNumber = method.getLineNumberFromBCI(bci);
Symbol sourceName = klass.getSourceFileName();
if (lineNumber !=-1 && sourceName != null) {
stringBuilder.append(sourceName.asString());
stringBuilder.append(":");
stringBuilder.append(lineNumber);
} else {
stringBuilder.append("bci=");
stringBuilder.append(bci);
}
}
stringBuilder.append(")");
}
long getTid(JavaThread thread) {
Oop threadObj = thread.getThreadObj();
Klass klass = threadObj.getKlass();
if (!(klass instanceof InstanceKlass)) return -1L;
InstanceKlass instanceKlass = (InstanceKlass) klass;
Field tidField = instanceKlass.findField("tid", "J");
if (!(tidField instanceof LongField)) return -1L;
long tid = ((LongField) tidField).getValue(threadObj);
return tid;
}
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + (int) (threadId ^ (threadId >>> 32));
return result;
}
@Override
public boolean equals(Object obj) {
if (this == obj)
return true;
if (obj == null)
return false;
if (getClass() != obj.getClass())
return false;
ThreadStruct other = (ThreadStruct) obj;
if (threadId != other.threadId)
return false;
return true;
}
}
And the main magic goes under the run() method:
try {
ObjectHeap heap = VM.getVM().getObjectHeap();
final ObjectReader objectReader = new ObjectReader();
Threads threads = VM.getVM().getThreads();
List monitors = new ArrayList();
Iterator objectMonitorIterator = ObjectSynchronizer.objectMonitorIterator();
while (objectMonitorIterator.hasNext()) {
ObjectMonitor objectMonitor = (ObjectMonitor) objectMonitorIterator.next();
List pendingThreads = threads.getPendingThreads(objectMonitor);
if (!pendingThreads.isEmpty()) {
Oop object = heap.newOop(objectMonitor.object());
Object unmrashalledObject = objectReader.readObject(object); // class of this object must reside at the local classpath (internally Class.forName(className) is invoked)!!!
JavaThread owningThread = threads.owningThreadFromMonitor(objectMonitor);
Monitor monitor = new Monitor();
monitor.object = unmrashalledObject;
monitor.className = object.getKlass().getName().asString();
monitor.setOwningThread(owningThread);
monitor.setPendingThreadNames(pendingThreads);
for (Monitor registeredMonitor : monitors) {
if (registeredMonitor.pendingThreads.contains(monitor.owningThread) &&
monitor.pendingThreads.contains(registeredMonitor.owningThread)) {
throw new IllegalStateException(
String.format(
"Deadlock detected! Thread (%s blocking on (%s)) owning resource (%s) is waiting for resource (%s), while Thread (%s blocking on (%s)) owning resource (%s) is waiting for resource (%s)",
registeredMonitor.owningThread.threadName, registeredMonitor.owningThread.stackTrace,
registeredMonitor.object, monitor.object, monitor.owningThread.threadName, monitor.owningThread.stackTrace, monitor.object, registeredMonitor.object));
// prints something like this Exception in thread "main" java.lang.IllegalStateException: Deadlock detected! Thread (Thread2 blocking on ( at com.ipr.heap.dump.Project$2.run(Project.java:44) at java.lang.Thread.run(Thread.java:745)))
// owning resource (resource2) is waiting for resource (resource1),
// while Thread (Thread1 blocking on ( at com.ipr.heap.dump.Project$1.run(Project.java:28) at java.lang.Thread.run(Thread.java:745))) owning resource (resource1) is waiting for resource (resource2)
}
}
monitors.add(monitor);
}
}
} catch (AddressException | ClassNotFoundException e) {
e.printStackTrace();
}
Generally speaking, here we do a traversal through all of the monitor synchronizers currently active in JVM, then searching for a pending thread on the current monitor, and finally we find the owner thread for this monitor. And the logic to determine that we are deadlocked is rather obvious: If monitor (resource1) has owner Thread (Thread1) while Thread (Thread2) is pending on it and monitor (resource2) has owner Thread (Thread2) while Thread (Thread1) is pending on it. Profit! If the problem is floating enough then we can wrap a launch of utility in some kind of Timertask (ScheduledExecutorService or whatever) and check for a presence of deadlocked threads within a determined interval.
Also, we could achieve this outcome from thread traversal perspective and our code could look as following:
class MonitorObject {
JavaThread pending;
JavaThread owner;
Object object;
Address ownerAddress;
MonitorObject(JavaThread pending, JavaThread owner,
OopHandle oopHandle, Address ownerAddress) throws ClassNotFoundException {
super();
this.pending = pending;
this.owner = owner;
this.object = objectReader.readObject(heap.newOop(oopHandle));
this.ownerAddress = ownerAddress;
}
}
List monitorObjects = new ArrayList<>();
for (JavaThread thread = threads.first(); thread != null; thread = thread.next()) {
ObjectMonitor pendingMonitor = thread.getCurrentPendingMonitor();
if (thread.getThreadState() != JavaThreadState.BLOCKED || pendingMonitor == null) {
continue;
}
MonitorObject currentMonitorObject = new MonitorObject(thread,
threads.owningThreadFromMonitor(pendingMonitor), pendingMonitor.object(), pendingMonitor.owner());
for (MonitorObject monitorObject : monitorObjects) {
JavaThread pending = monitorObject.pending;
JavaThread owner = monitorObject.owner;
Object object = monitorObject.object;
if (pending.isLockOwned(currentMonitorObject.ownerAddress)
&& currentMonitorObject.pending.isLockOwned(monitorObject.ownerAddress)) {
throw new IllegalStateException(
String.format(
"Deadlock detected! Thread (%s) owning resource (%s) is waiting for resource (%s), while Thread (%s) owning resource (%s) is waiting for resource (%s)",
pending.getThreadName(),
currentMonitorObject.object, object, owner.getThreadName(),
object, currentMonitorObject.object));
}
}
monitorObjects.add(currentMonitorObject);
}
This solution can also be used remotely. But in this case jsadebugd daemon should be running on the remote machine and our launcher needs to be rewritten like following:
DeadlockFinder deadlockFinder = new DeadlockFinder();
deadlockFinder.setDebugeeType(Tool.DEBUGEE_REMOTE);
deadlockFinder.start(new String[] {"[server_id@]"});
That’s it.
Rather a graceful solution, but not without a fly in the ointment. The most unpleasant drawback of this approach is the absence of good and even bad (it is really very hard to find at least anything) documentation.
Anyway with this approach you can only get a plain information regarding deadlocks, but you can't do anything with that, I mean if threads are deadlocked they are deadlocked forever (whatever you use synchronized blocks or ownable locks). So in order to have a tool as a replacement for locks with which you can somehow affect from outside, you might take a look at java.util.concurrent.Semaphore (with 1 permit settings) and expose it via JMX for example. And it might be used as following:
Semaphore semaphore = new Semaphore(1, true);
try {
semaphore.acquire();
// you code goes here
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
semaphore.release();
}
And if some threads are deadlocked on acquire(), it can be easily unlocked (call method release()) from a third-party thread without the need to be the owner of this monitor.
This particular solution was tested on Oracle Java 1.7, and since it is based on its internals, this may not be available in the future, or it will be – but with some admissible probability of changes. Either way, don’t hesitate to get in touch with us if you need assistance or advice!
 |
About the author: Aleksandrs Konstantinovs is a software developer with more than 7 years of experience, mainly with the JVM technology stack and client-server architecture. He has deep understanding of how garbage collector, JMM and concurrency work in JVM, and how it can influence the performance of the entire virtual machine. |
