I’ve been hunting a deadlock in an application of a customer we support with our RCP, EMF and OSGi know-how and turned out be a class-loader problem because Hibernate uses Class.forName().

The application is a pure OSGi-Application acting as an Application Server which is used to process database informations in an asynchronous fashion and heavily uses OSGi-DS to wire up the complete application. The problems we’ve been facing only happened at start up where many threads are accessing database informations in a highly concurrent fashion.

The deadlock information from the JVM looked like this:

Found one Java-level deadlock:
=============================
"Thread-23":
  waiting to lock monitor 0x18b30eac (object 0x09c93230, a org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader),
  which is held by "Thread-17"
"Thread-17":
  waiting to lock monitor 0x1977f16c (object 0x09c92fd0, a org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader),
  which is held by "Thread-12"
"Thread-12":
  waiting to lock monitor 0x18b30eac (object 0x09c93230, a org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader),
  which is held by "Thread-17"

where the threads where calling the following methods:

“Thread-23”:

        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at org.hibernate.util.ReflectHelper.classForName(ReflectHelper.java:192)
        at org.hibernate.impl.SessionFactoryImpl.getImplementors(SessionFactoryImpl.java:825)
        at org.hibernate.hql.QuerySplitter.concreteQueries(QuerySplitter.java:123)
        at org.hibernate.engine.query.HQLQueryPlan.<init>(HQLQueryPlan.java:92)
        at org.hibernate.engine.query.HQLQueryPlan.<init>(HQLQueryPlan.java:80)
        at org.hibernate.engine.query.QueryPlanCache.getHQLQueryPlan(QueryPlanCache.java:124)
        at org.hibernate.impl.AbstractSessionImpl.getHQLQueryPlan(AbstractSessionImpl.java:156)
        at org.hibernate.impl.AbstractSessionImpl.createQuery(AbstractSessionImpl.java:135)
        at org.hibernate.impl.SessionImpl.createQuery(SessionImpl.java:1770)

“Thread-17”:

        at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass_LockClassLoader(ClasspathManager.java:468)
        - waiting to lock <0x09c92fd0> (a org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader)
        at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(ClasspathManager.java:449)
        at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:216)
        at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:393)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:469)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
        at org.eclipse.osgi.internal.loader.buddy.DependentPolicy.loadClass(DependentPolicy.java:54)
        at org.eclipse.osgi.internal.loader.buddy.PolicyHandler.doBuddyClassLoading(PolicyHandler.java:135)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:494)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
        at org.eclipse.osgi.internal.loader.buddy.DependentPolicy.loadClass(DependentPolicy.java:54)
        at org.eclipse.osgi.internal.loader.buddy.PolicyHandler.doBuddyClassLoading(PolicyHandler.java:135)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:494)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
        at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at org.hibernate.util.ReflectHelper.classForName(ReflectHelper.java:192)
        at org.hibernate.impl.SessionFactoryImpl.getImplementors(SessionFactoryImpl.java:825)
        at org.hibernate.hql.QuerySplitter.concreteQueries(QuerySplitter.java:123)
        at org.hibernate.engine.query.HQLQueryPlan.<init>(HQLQueryPlan.java:92)
        at org.hibernate.engine.query.HQLQueryPlan.<init>(HQLQueryPlan.java:80)
        at org.hibernate.engine.query.QueryPlanCache.getHQLQueryPlan(QueryPlanCache.java:124)
        at org.hibernate.impl.AbstractSessionImpl.getHQLQueryPlan(AbstractSessionImpl.java:156)
        at org.hibernate.impl.AbstractSessionImpl.createQuery(AbstractSessionImpl.java:135)

“Thread-12”:

        at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass_LockClassLoader(ClasspathManager.java:468)
        - waiting to lock <0x09c93230> (a org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader)
        at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(ClasspathManager.java:449)
        at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:216)
        at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:393)
        at org.eclipse.osgi.internal.loader.SingleSourcePackage.loadClass(SingleSourcePackage.java:33)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:466)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
        at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at com.bizerba.basic.resource.teneo.TeneoHbDataStoreProvider$ChangeInformationInterceptor.afterTransactionBegin(TeneoHbDataStoreProvider.java:148)
        at org.hibernate.impl.SessionImpl.afterTransactionBegin(SessionImpl.java:1479)
        at org.hibernate.jdbc.JDBCContext.afterTransactionBegin(JDBCContext.java:259)
        at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:107)
        at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)

So what did we do to fix the problem once we managed to get to the bottom of it? We asked at the equinox IRC-Channel for help and Tom Watson pointed me to a bug 212262 describing more or less the problem we are seeing. As described in the bug we’ve added jvm options:

-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass

and ran our test-suite the whole night and could not reproduce the locks since then.

So if you happen to run Hibernate (other libraries who use Class.forName() might have the same problem) inside the Equinox-OSGi-Container (I have no idea if other OSGi-Implementation might face the same problem) and your system is doing many things in a concurrent way you should consider setting those 2 options to avoid deadlocks.