Last week I made an interesting observation. I was able to
loose persistent queue messages that were sent in a transaction to an ActiveMQ
broker.
I believe its best to share the findings.
SETTING THE SCENE
Suppose you have a MessageProducer that sends messages to an
ActiveMQ broker inside a transaction. This producer could be anything, a plain
Java JMS Producer client, a Spring JMS producer or as in my case a Camel route.
It also does not matter if the producer sends one or more messages within the
same transaction.
Next the ActiveMQ broker instance is configured for a
specific limit. In my test that limit was fairly low in
order to reproduce the problem quickly. In addition I did set the property
sendFailIfNoSpace so that the broker raises an exception back to the producer
once any of its global limits have been reached. So the configuration I used
reads
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="10 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="64 mb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="10 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
Without sendFailIfNoSpace="true" ActiveMQ would
block the producer until additional space is available on the broker. This
would most likely not reproduce the problem I am talking about.
I ran the ActiveMQ broker with these settings, kicked off my
Camel route and let it do what it can do best: route messages (which are
finally sent to a queue on my ActiveMQ broker). Btw, I did not have any
consumers attached to the broker; only the Camel route producer was connected. Still
the same problem can occur with slow consumers.
And finally my producer did not register an exception
listener. In my tests the producer was a Camel route and Camel currently does
not register an exception listener.
THE PROBLEM
Now, the problem starts when the broker reaches its configured
limits. Because of the property sendFailIfNoSpace="true", it will not
block the producer indefinitely but throw an exception back to the producer. In
my test I was hitting the storeUsage limit of 64 MB.
On the other hand when sending messages in a transaction,
then the actual send is done asynchronously. That means the thread that is
sending the message does not wait for the ack from the broker. It’s the
transport thread that receives the broker’s ack and deals with it. Using an
async send in the case of running inside a transaction is done for performance
reasons. By not waiting for the broker ack, you can send messages more quickly
to the broker. Until the transaction finally commits the messages received by
the broker will not be placed onto the brokers queue.
Putting this together the following happened in my test:
-
The producer started a new transaction. This information
was sent to the broker, the broker accepted the new transaction.
-
The producer then sent a message to the broker.
As this send was within an existing transaction it was done async. The call to
send the message returned immediately after sending the message. It did not
wait for the broker’s ack.
-
The broker did no accept this message but raised
a "javax.jms.ResourceAllocationException: Persistent store is Full, 100%
of 67108864" back to the producer.
-
The producers transport thread received the
exception but didn’t know how to handle it. Instead it printed the following
debug log message:
DEBUG - ActiveMQConnection - Async exception with no exception listener:
javax.jms.ResourceAllocationException: Persistent store is Full, 100% of
67108864. Stopping producer (ID:Mac.local-51375-1320685288867-2:1:1:1) to
prevent flooding queue://TEST.IN.
It flagged the fact that
there was an error but it did not mark the current transaction to be rolled
back. This would have been the job of an exception listener, but none was
registered.
-
The
producer then committed the transaction. From the producers point of view no
errors did happen (it never became aware of the JMSException) so it asked the
broker to commit the transaction.
-
The
broker committed the transaction just fine although the message did not get
stored on the broker’s queue. From the brokers point of view all went fine. The
producer did start a new transaction, it then sent some msgs, which the broker
rejected and then the broker got asked to commit the transaction. All fine from
the brokers point of view. It’s the producer’s task to deal with the
JMSException and to decide whether to rollback or not.
-
The
message(s) that got sent inside the transaction is (are) lost!
It does not necessarily require an
ActiveMQ broker that has reached its configured limits to run into this
problem. If the transport for any reason fails to send the msg to the broker
but works again on the transaction commit, then the same problem could arise. This
is less likely to occur though. Transport related problems (e.g. connection
loss) will most of the time affect the ability to commit the transaction as
well (and not only the individual message send). When there is an exception
during commit, it will correctly roll back the entire transaction.
You would
think that using JMS Transactions would guard you against message loss. In
general that is correct. However you need to make sure that any errors that can
occur within the transaction are being dealt with. In this scenario, there was
no exception listener registered in the producer that would deal with the
JMSException. So I lost messages because I did not deal with the exception
returned from the async send.
When using Camel as a message
producer, you currently cannot easily register an exception listener, which
then ensures the transaction gets rolled back in case of any problems.
Luckily there are some simple solutions to this problem:
THE SOLUTION
.. is to either
1) Question if you really need to send your messages inside
a transaction. When sending only one persistent message at a time inside a
transaction, there is not much benefit over sending the message without a
transaction. Persistent messages are sent synchronously by default so the producer
waits for the brokers ack before proceeding and it can deal directly with any
error returned from the broker. Transactions are very useful though when
sending multiple messages that need to all succeed or fail as one atomic
operation.
2) If you need to use transactions for sending your messages
to the broker, then configure the ConnectionFactory to always send messages
synchronously rather than using the default async mode. Changing to sync send
will have a small performance impact, particularly when sending a batch of messages
within one transaction. This performance impact however is probably negligible
in most scenarios.
You can switch to using a sync send by using the
alwaysSyncSend=true property on the ActiveMQConnectionFactor, e.g. for a Spring
configuration:
<bean id="AMQJMSConnectionFactory3" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="failover:(tcp://localhost:61620)" />
<property name="alwaysSyncSend" value="true"/>
</bean>
Alternatively, set jms.alwaysSyncSend=true on the broker URL,
e.g.
tcp://localhost:61616?jms.alwaysSyncSend=true, or
failover:(tcp://localhost:61616)?jms.alwaysSyncSend=true
3) Register a JMS exception listener programmatically so
that it reacts on any exceptions thrown by the broker in case of sending the message
async. For a plain Java JMS client you can use
ActiveMQConnectionFactory connectionFactory =
new ActiveMQConnectionFactory(brokerUrl); connectionFactory.setExceptionListener(new ExceptionListener() {
public void onException(JMSException ex) {
// handle the exception
}
});
You can also set this exception listener on an existing JMS
Connection object.
The somewhat tricky part may be to figure out which
transaction needs to be rolled back, in case of having multiple threads sending
messages concurrently to the broker.
I already mentioned this option is not easily done in case
where the producer is a Camel route. We hope to improve Camel to have an out of
the box exception listener registered in future. Till then
CAMEL-4616
captures this request.
If you want to be on the safe side, the go with option #2
and ensure a sync send of your messages.
