Friday, August 25, 2006

Distributed (XA) Transactions w/ Hibernate, Spring, JTA, JOTM

I recently struggled with getting XA transactions to work across Oracle 9i and SQL Server 2000 databases, using a combination of Hibernate, Spring, JOTM, and the SQL Server 2005 JDBC driver. I ran into some stickiness, and I still think I might be missing something - but XA transactions work. Recovery probably doesn't, but for our organization that's OK right now. In the off-chance that my probably-misguided attempts could prove useful to someone else in my situation, here is what I learned:

XAPool Is Cheating
Most places you look, XAPool is listed as a great companion to JOTM for supplying connection pooling. It is - if you need to make data sources which don't actually support XA transactions pretend to participate in XA transactions. But if you actually want two-phase commits as specified in the XA standard, you'd better not use XAPool, because it just wraps non-XA data sources. Using JOTM with XA gives you 'simulated' XA transactions - on the one hand, you can easily start and commit/roll back transactions on multiple data sources, but when it comes to commit time, if you commit your first data source and the second data source fails its commit, there's no way to roll back the first one. Which is why we have the 2PC concept in XA to begin with. Which makes XAPool a rather confusingly-named library. Definately not what I was looking for.

For real XA with JOTM, ditch XAPool and use the XADataSource implementations supplied by your vendor's JDBC driver. There might be an easier way, but I wrote a simple helper class to manage connection pooling and transaction enlistment using commons-pool and commons-dbcp. More on that in a bit.

Fixing JOTM
Just when I thought I had things set up correctly, I started getting intermitent and difficult-to-reproduce failures from my resource managers. Given my unfamiliarity with the terrain, it took me a while to realize that the source of the problem was JOTM, not my code/configuration. I am using JOTM 2.0.10, which as far as I know is the most recent official release. Unfortunately it appears to have a rather show-stopping bug - it gets its branch transaction ids mixed up.

When you start an XA transaction, your transaction manager (JOTM) makes up an Xid to identify the global transaction. For each resource manager which is enlisted in the transaction, the TM creates a branch id, which is typically the global id with an appended bit to identify the resource manager. That branch id is used to identify the transaction on each resource manager.

I had created a test application using two data sources, and had successfully committed/rolled back some transactions across both. When I configured my production application for XA, I kept getting intermittent errors from one or the other resource manager that a transaction id was invalid. I couldn't reproduce the problem in my test app. What was going on?

A little background for this bug: for a two-phase commit (2PC), the TM first sends a message to all of the resource managers involved in the transaction that it wants to commit, and so they should indicate whether they can do so, and also 'freeze' the transaction in question. Each resource manager then sends back a 'vote.' This vote can basically be 'commit,' 'rollback,' or 'I was enlisted but no changes were made to me, so I don't have to do anything.'

Turns out JOTM 2.0.10 doesn't handle that third vote so well. When a resource manager indicates that it doesn't need to commit, JOTM can its Xids for the various resource managers confused and end up sending the wrong branch Xid to a resource manager that does need to commit. Which is what was happening to me. This is a big deal, and happened to me almost immediately - the only reason I can think of that it wasn't identified and fixed was that XAPool is so often used, and with XAPool none of this 2PC stuff is actually happening, so XAPool probably always votes 'commit.' I haven't checked.

So I built JOTM from source with my fix. The offending class is org.objectweb.jotm.SubCoordinator. I was going to post a little examination of the bug, but honestly it isn't worth the effort. It's really glaring, and there's nothing particularly subtle or interesting about it. Here's the patch:

77a78,79
>
> private Vector loggedJavaxXids = new Vector();
83c85
< public void addResource( XAResource res, Xid xid )
---
> public void addResource( XAResource res, Xid xid, javax.transaction.xa.Xid javaXid )
90a93
> loggedJavaxXids.addElement(javaXid);
105a109,112
>
> public javax.transaction.xa.Xid getLoggedJavaxXid(int index) {
> return (javax.transaction.xa.Xid) loggedJavaxXids.get(index);
> }
825c832
< log.addResource(res,xid);
---
> log.addResource(res,xid,myjavaxxid);
1021c1028,1034
< javax.transaction.xa.Xid myjavaxxid = (javax.transaction.xa.Xid) javaxxidList.elementAt(i);
---
> /* whoops - this line is wrong. For example if the resource manager corresponding to
> * javaxxidList.elementAt(0) votes VOTE_READONLY and the next on the list votes
> * VOTE_COMMIT, the wrong xid will get matched with this resource and the commit
> * will fail.
> */
> // javax.transaction.xa.Xid myjavaxxid = (javax.transaction.xa.Xid) javaxxidList.elementAt(i);
> javax.transaction.xa.Xid myjavaxxid = (javax.transaction.xa.Xid) log.getLoggedJavaxXid(i);

Wiring the Pieces (Spring, Hibernate, and some Pooling)
Now that JOTM will work correctly, let's take a look at the Spring configuration necessary to get ourselves off the ground.


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">

<beans>
<bean id="sqlServerDataSourceTarget" class="com.microsoft.sqlserver.jdbc.SQLServerXADataSource">
<property name="URL" value="jdbc:sqlserver://server:port;databaseName=jtatest;user=testing_user;password=testing_user;"/>
</bean>

<bean id="oracleDataSourceTarget" class="oracle.jdbc.xa.client.OracleXADataSource">
<property name="URL" value="jdbc:oracle:thin:user/password@server:port:PPRD"/>
</bean>

<bean id="jotm" class="org.springframework.transaction.jta.JotmFactoryBean"/>

<bean id="txManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="userTransaction" ref="jotm"/>
</bean>

<bean id="sqlServerDataSource" class="com.mattmcgill.PoolingXADataSource">
<property name="xaDataSource" ref="sqlServerDataSourceTarget"/>
<property name="transactionManager" ref="jotm"/>
</bean>

<bean id="oracleDataSource" class="edu.taylor.domain.PoolingXADataSource">
<property name="xaDataSource" ref="oracleDataSourceTarget"/>
<property name="transactionManager" ref="jotm"/>
</bean>

<bean id="sqlServerSessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="sqlServerDataSource"/>
<property name="jtaTransactionManager" ref="jotm"/>
<property name="useTransactionAwareDataSource" value="true"/>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.SQLServerDialect</prop>
</props>
</property>
<property name="mappingResources">
<list>
<value>com/mattmcgill/jtatest/some-entity.hbm.xml</value>
</list>
</property>
</bean>

<bean id="oracleSessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="oracleDataSource"/>
<property name="jtaTransactionManager" ref="jotm"/>
<property name="useTransactionAwareDataSource" value="true"/>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.Oracle9Dialect</prop>
</props>
</property>
<property name="mappingResources">
<list>
<value>edu/taylor/jta/fee.hbm.xml</value>
</list>
</property>
</bean>
</beans>


In this example, XADataSources are configured for Oracle and SQL Server. Then a JOTM bean is configured using the helpful org.springframework.transaction.jta.JotmFactoryBean class. Next, we wrap our XADataSources in instances of PoolingXADataSource, a little helper class I wrote which makes sure that when a connection is obtained from the pool, it gets enlisted in any current thread-bound transaction that happens to be taking place. Here is the code for that class:

PoolingXADataSource.java

package com.mattmcgill

import java.io.PrintWriter;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;

import javax.sql.DataSource;
import javax.sql.XAConnection;
import javax.sql.XADataSource;
import javax.transaction.Status;
import javax.transaction.Transaction;
import javax.transaction.TransactionManager;

import org.apache.commons.dbcp.PoolableConnection;
import org.apache.commons.dbcp.PoolingDataSource;
import org.apache.commons.pool.PoolableObjectFactory;
import org.apache.commons.pool.impl.GenericObjectPool;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.DisposableBean;
import org.springframework.beans.factory.InitializingBean;

public class PoolingXADataSource implements DataSource, PoolableObjectFactory, InitializingBean, DisposableBean {
private static final Logger logger = Logger.getLogger(PoolingXADataSource.class);

private Map<Connection, XAConnection> xaConnectionLookup = new HashMap<Connection, XAConnection>();
private XADataSource xaDataSource;
private TransactionManager transactionManager;
private GenericObjectPool connectionPool;
private PoolingDataSource poolingDataSource;

public XADataSource getXaDataSource() {
return xaDataSource;
}

public void setXaDataSource(XADataSource xaDataSource) {
this.xaDataSource = xaDataSource;
}

public TransactionManager getTransactionManager() {
return transactionManager;
}

public void setTransactionManager(TransactionManager transactionManager) {
this.transactionManager = transactionManager;
}

public void afterPropertiesSet() throws Exception {
if (xaDataSource == null)
throw new RuntimeException("Must set xaDataSource");
if (transactionManager == null)
throw new RuntimeException("Must set transactionManager");
connectionPool = new GenericObjectPool(this);
connectionPool.setWhenExhaustedAction(GenericObjectPool.WHEN_EXHAUSTED_GROW);
connectionPool.setMaxActive(30);
connectionPool.setMinIdle(5);
for (int i=0;i<5;i++) connectionPool.addObject();
poolingDataSource = new PoolingDataSource(connectionPool);
}

public void destroy() throws Exception {
connectionPool.close();
}

public void activateObject(Object conn) throws Exception {
logger.debug("activateObject(" + conn + ") [" + xaDataSource + "]");
logger.debug(" active: " + connectionPool.getNumActive());
logger.debug(" idle: " + connectionPool.getNumIdle());
XAConnection xaConn = xaConnectionLookup.get(conn);
if (transactionManager.getStatus() != Status.STATUS_NO_TRANSACTION) {
Date start = new Date();
logger.info("Found transaction, associating connection");
Transaction tx = transactionManager.getTransaction();
tx.enlistResource(xaConn.getXAResource());
Date end = new Date();
logger.debug(" enlisting took " + (end.getTime() - start.getTime()) + " millis");
}
}

public void destroyObject(Object conn) throws Exception {
logger.debug("destroyObject(" + conn + ") [" + xaDataSource + "]");
logger.debug(" active: " + connectionPool.getNumActive());
logger.debug(" idle: " + connectionPool.getNumIdle());
logger.info("Destroying connection [" + xaDataSource + "]");
xaConnectionLookup.remove(conn);
}

public Object makeObject() throws Exception {
logger.debug("makeObject() [" + xaDataSource + "]");
logger.info("Opening connection [" + xaDataSource + "]");
XAConnection xaConn = xaDataSource.getXAConnection();
Connection conn = new PoolableConnection(xaConn.getConnection(), connectionPool);
xaConnectionLookup.put(conn, xaConn);
logger.debug("returned [" + conn + "]");
return conn;
}

public void passivateObject(Object conn) throws Exception {
logger.debug("passivateObject(" + conn + ") [" + xaDataSource + "]");
logger.debug(" active: " + connectionPool.getNumActive());
logger.debug(" idle: " + connectionPool.getNumIdle());
}

public boolean validateObject(Object conn) {
logger.debug("validateObject(" + conn + ") [" + xaDataSource + "]");
return true;
}

public Connection getConnection() throws SQLException {
logger.debug("getConnection() [" + xaDataSource + "]");
try {
return poolingDataSource.getConnection();
} catch (Exception ex) {
throw new RuntimeException("Couldn't borrow object from pool", ex);
}
}

public Connection getConnection(String arg0, String arg1)
throws SQLException {
throw new UnsupportedOperationException();
}

public PrintWriter getLogWriter() throws SQLException {
return xaDataSource.getLogWriter();
}

public void setLogWriter(PrintWriter writer) throws SQLException {
xaDataSource.setLogWriter(writer);
}

public void setLoginTimeout(int timeout) throws SQLException {
xaDataSource.setLoginTimeout(timeout);
}

public int getLoginTimeout() throws SQLException {
return xaDataSource.getLoginTimeout();
}
}


Don't Forget These!
After that, all that's left is to configure our SessionFactorys (which I do by way of Spring's LocalSessionFactoryBean, and we're all set. Except that nothing will work unless you remember to set the following properties on LocalSessionFactoryBean:

<property name="jtaTransactionManager" ref="jotm"/>
<property name="useTransactionAwareDataSource" value="true"/>

The Spring documentation is very clear on how to use this session factory definition in conjunction with a TransactionProxyFactoryBean to configure declarative transactions. I'll leave that part up to you =)

If anyone can tell me a better way to make this work with connection pooling which removes the need for the PoolingXADataSource class, please let me know! I feel like it should be simple and I must be missing something.

1 comment:

Unknown said...

Hi Matt, it is a great post, i have just a question. How can hibernate relate the entities that are in different data bases?.

For example i have User entity with a collection of Role entity and a collection of Authority entity, the User entity map to the table user in db1 but Role and Authority entities map to role and authority tables in db2.

I configure a little project to support jotm transactions just like you explain but it does not work, and i suspect that it because hibernate get an error when load the mappings.

I will appreciate your help.
Maykell