Ramblings on pretty much anything technical.

22 Dec 2011

Securing the ActiveMQ web console using LDAP based authentication

In my previous blog post I described how to configure LDAP based authentication and authorization in ActiveMQ while also allowing anonymous access to certain destinations in the broker. That previous post as well as this post is based on the LDAP tutorial in the ActiveMQ Security Guide provided by Fusesource.com.

In this article I want to expand on the previous post and show how to secure the ActiveMQ web console so that web users will be authenticated against user information stored in LDAP. All the configuration from the previous post can extended in order to secure the web console but I will provide full configuration here as well so that there is no need to copy and paste config snippets from the previous article.
I recommend the LDAP tutorial as a prerequisite to this article as it populates the LDAP server with the right security information (users, passwords, groups and roles), which is needed if you want to follow the steps of this blog post.

The ActiveMQ web console internally uses Jetty as the web container. So securing the web console basically means configuring the Jetty HTTP server to authenticate any web clients. The way how Jetty gets secured is completely independent from the way how ActiveMQ gets secured. However both support JAAS so the actual configuration principles are the same. Jetty also supports LDAP based authentication through its own JAAS LoginModule class org.eclipse.jetty.plus.jaas.spi.LdapLoginModule.

Prerequisites before taking off

The LDAP tutorial in the ActiveMQ Security Guide (version 5.5) makes some configuration that will not work quite well with the JAAS LDAP login module used by the ActiveMQ web console (i.e. Jetty). Two configuration changes are required in your LDAP user and group data to successfully authenticate any web users via Jetty's LdapLoginModule. Here are the required changes to your LDAP data:

1) Store passwords in plain text in LDAP:
The LDAP tutorial instructs you to store any user passwords as SHA hashes. See steps 18 and 19 of the Add User Entries part of the tutorial (and Figure 6.7). The LdapLoginModule from Jetty reads that password from LDAP as it is (i.e. SHA hash) and tries to match it against the password supplied by the web user, which is in plain text. So the SHA hashed password stored in LDAP is string compared against the plain text password, which will obviously fail.
Rather than storing the password as SHA hash in LDAP, store it in plain text.

This needs to be done for all users.
Storing the password in plain text will not change the ability of the ActiveMQ LDAPLoginModule class to authenticate users. This LoginModule takes a different approach. Rather than retrieving the password and comparing it against the password given by the user, it tries to bind the user (an operation on the LDAP server) whereby the password gets verified by the LDAP server and not by the LoginModule class.
Perhaps there is a way to configure the Jetty LdapLoginModule to also work with SHA hashed password, however I did not find it (also tried DIGEST based authentication but that did not help).

2) User to group mapping in LDAP must use users full dn + basedn name:

Steps 30-31 and figure 6.8 of the Add User Entries part of the LDAP tutorial explain how to make users member of a particular group. It uses the member attribute of the groupOfNames LDAP class and the mapping to a user is done via the users uid attribute, like in this example mapping that is taken from the LDAP tutorial:
member=uid=jdoe

This won't work with the Jetty LdapLoginModule implementation, as it queries for a users group using the user's full name (i.e. uid + base dn), e.g.

member=uid=jdoe,ou=User,ou=ActiveMQ,ou=system

and as a result won't find any user groups if the group members are only specified using the uid (i.e. jdoe).
To overcome this difference in the LDAP login module implementations, it is necessary to always provide the dn + basedn of the user that you want to add to a group.
E.g. rather than setting the members of a group like this ((see Figure 6.8 of the LDAP tutorial)

use this

This change of users to groups wiring however requires an update on the ActiveMQ login module configuration in login.config from
roleSearchMatching="(member=uid={1})"

to
roleSearchMatching="(member=uid={1},ou=User,ou=ActiveMQ,ou=system)"

That way both, the LDAP LoginModule of ActiveMQ and Jetty will be able to retrieve the groups (roles) a user is in.
With these two changes to the LDAP data we are ready to go and configure the web console for LDAP based authentication.

Steps to securing the web console

All of the Jetty configuration in ActiveMQ is stored in $ACTIVEMQ_HOME/conf/jetty.xml. Its Spring config already instantiates a security handler but authentication is turned off by default via the authenticate=false property. It needs to be enable in this bean:

</bean>

Changing the "authenticate" property to "true" is required but isn't enough. Any authenticated user needs to have the role specified by the "roles" attribute in order to get authorized. The administrator role name in LDAP is called "admins" but the Jetty securityConstraint bean uses "admin" out of the box. It is necessary to change the value of the "roles" property to "admins" as in the above example. A fully configured jetty.xml is provided at the end of this article.

Out of the box Jetty is configured to use a HashLoginService class, which simply reads the user credentials from a text file ${activemq.base}/conf/jetty-realm.properties. In order to authenticate against an LDAP server we need to configure a JAAS LoginService class:

</bean>

This JAASLoginService is configured for a LoginModuleName called "jetty-ldap". This name refers to the JAAS login module configuration in login.config. In the previous article I already provided a login.config to be used by ActiveMQ. You can simply add the following configuration to that same login.config:

/**

This LoginModule configuration is only used by Jetty when

authentication of the web console is enabled.

jetty-ldap {

org.eclipse.jetty.plus.jaas.spi.LdapLoginModule required

debug="true"

contextFactory="com.sun.jndi.ldap.LdapCtxFactory"

hostname="localhost"

port="10389"

bindDn="uid=admin,ou=system"

bindPassword="secret"

authenticationMethod="simple"

forceBindingLogin="false"

userBaseDn="ou=User,ou=ActiveMQ,ou=system"

userRdnAttribute="uid"

userIdAttribute="uid"

userPasswordAttribute="userPassword"

userObjectClass="inetOrgPerson"

roleBaseDn="ou=Group,ou=ActiveMQ,ou=system"

roleNameAttribute="cn"

roleMemberAttribute="member"

roleObjectClass="groupOfNames"

authenticated="true";

};

Finally Jetty needs to be told where to find this login.config file via a Java property to the JVM.
The easiest way is to add this propery to the bin/activemq script, e.g.

ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS_MEMORY -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=login.config"

That's about it. Assuming the data in LDAP is setup according to the prerequisites listed above, the ActiveMQ web console should now be secured. Trying to access http://localhost:8161 should now request a username and password from you which is then authenticated against the LDAP server.

Note we did not need to change the broker configuration (with the exception of pre-requisite #2) as the authentication done in the web console (based on Jetty) is completely independent of the authentication done by ActiveMQ.

Links to all files used in this post:
jetty.xml
login.config
activemq-ldap-with-anon-access.xml

The relevant documentation that helped me on this subject:
Jetty JAAS Tutorial
Jetty Realms Tutorial
Jetty 7 JavaDoc

Let me know if this post was helpful.
This configuration has been tested with ActiveMQ 5.5.1 and 5.6.0.

5 Dec 2011

ActiveMQ: LDAP based authentication and authorization

The FuseSource ActiveMQ Security Guide has two great chapters on how to configure ActiveMQ for authentication and authorization against an LDAP server. Chapter 5 even has a tutorial that contains step-by-step instructions on how to configure your LDAP server and ActiveMQ based on ApacheDS, an open-source LDAP server. If you need to secure your ActiveMQ broker, then I highly recommend this documentation.

The tutorial in the Security Guide shows the relevant configuration needed so that every JMS connection into the broker is authenticated and checked for authorization against the security information stored in an LDAP server.

Sometimes you may want to allow anonymous access to certain destinations on your broker that are not critical while securing access to your critical destinations. In this post I like to outline a possible solution for such use-case. I will assume the reader is generally familiar with LDAP based authentication and authorization in ActiveMQ. If not, I suggest to first consult the ActiveMQ Security Guide.

When configuring ActiveMQ for JAAS based authentication against an LDAP server, the file login.config could read as follows (pretty much a copy-and-paste from the FuseSource Security Guide):

/** JAAS LoginModule that uses LDAP based authentication.

Every connection into broker must supply username and password

for authentication to succeed. Anonymous access not allowed.

LDAPLogin {

org.apache.activemq.jaas.LDAPLoginModule required

debug=true

initialContextFactory=com.sun.jndi.ldap.LdapCtxFactory

connectionURL="ldap://localhost:10389"

connectionUsername="uid=admin,ou=system"

connectionPassword=secret

connectionProtocol=""

authentication=simple

userBase="ou=User,ou=ActiveMQ,ou=system"

userSearchMatching="(uid={0})"

userSearchSubtree=false

roleBase="ou=Group,ou=ActiveMQ,ou=system"

roleName=cn

roleSearchMatching="(member=uid={1})"

roleSearchSubtree=false;

};

The broker then needs to use the JAAS authentication plug-in referencing the LDAPLogin authentication realm:

</plugins>

Using such configuration, every JMS connection will be authenticated using the LDAPLoginModule, no matter if it provides a username and password. Needless to say the authentication will fail if no username/password is supplied. Anonymous access won't be allowed in this configuration.

So in order to also allow anonymous access to the broker we can leverage the ActiveMQ GuestLoginModule.
The GuestLoginModule allows JMS connections that are not configured for username/password to still access a secured broker. It is typically used in conjunction with other JAAS login modules and basically successfully authenticates a JMS connection using a configurable username and group name. Here is a sample configuration of this login module:

org.apache.activemq.jaas.GuestLoginModule sufficient

debug=true

org.apache.activemq.jaas.guest.user="guest"

org.apache.activemq.jaas.guest.group="guests";

Authentication will succeed even if no username/password was supplied by the JMS client. Clients will be authenticated as user "guest" and belong to the group "guests". As you see, these names are configurable.

In JAAS configuration, multiple login modules can be combined in one JAAS authentication realm. They will then be tried in order.

The basic idea is to configure authentication so that the broker first invokes the GuestLoginModule, before trying the LDAPLoginModule. Here is the configuration:

/** JAAS LoginModule configuration that combines LDAPLoginModule with

GuestLoginModule.

LoginModules can be combined in JAAS.

See http://fusesource.com/docs/broker/5.5/security/Auth-JAAS-GuestLoginModule.html

for more information.

LDAPLogin-with-Anon-Access {

org.apache.activemq.jaas.GuestLoginModule sufficient

debug=true

credentialsInvalidate=true

org.apache.activemq.jaas.guest.user="guest"

org.apache.activemq.jaas.guest.group="guests";

org.apache.activemq.jaas.LDAPLoginModule requisite

debug=true

initialContextFactory=com.sun.jndi.ldap.LdapCtxFactory

connectionURL="ldap://localhost:10389"

connectionUsername="uid=admin,ou=system"

connectionPassword=secret

connectionProtocol=""

authentication=simple

userBase="ou=User,ou=ActiveMQ,ou=system"

userSearchMatching="(uid={0})"

userSearchSubtree=false

roleBase="ou=Group,ou=ActiveMQ,ou=system"

roleName=cn

roleSearchMatching="(member=uid={1})"

roleSearchSubtree=false;

};

</plugins>

The GuestLoginModule config above uses an additional property credentialsInvalidate that when set to "true" will only authenticate requests that do not have a username/password supplied. Each connection without username gets authenticated as user "guest" and belongs
to the group "guest".
Any other JMS connections that contain username and password will be authenticated using the next login module; the LDAP LoginModule. So its important to set credentialsInvalidate=true as otherwise the GuestLoginModule authenticates all requests no matter whether or what username/password is supplied.

Authenticating anonymous connections is only half the story. Without further authorization anonymous users would be allowed all operations on the broker, which is definitely not what we want.
With additional authorization anonymous access to the broker can be restricted to only specific destinations with restricted rights on each destination.
The FuseSource Security Guide also explains how to configure the LDAP Authorization plug-in and how to create groups and destinations in the LDAP server. A step-by-step guide for adding all required entries is given in chapter 5.
Based on the configuration proposed in that documentation it is additionally also necessary to

define all those destinations in LDAP that anonymous access will be allowed on and
grant the required permissions to the group "guests" to these destinations.

If for example anonymous access should be allowed to queue "example.A", then the destination "example.A" needs to be defined in LDAP. Further "read", "write" and potentially "admin" privileges need to be given to the group "guests" on this destination.
This procedure needs to be applied to all destinations that should allow anonymous access.

Finally, every new connection into the broker triggers some advisory messages. Without giving "guests" access to these advisory topics, the connection will fail, typically with an error similar to this:

java.lang.SecurityException:
User null is not authorized to create: topic://ActiveMQ.Advisory.Connection

It is necessary to also grant "admin", "write" and potentially "read" rights for the group "guests" to ActiveMQ.Advisory topics.

This will allow for anonymous access on certain destinations while enforcing username/password based authentication on all other destinations in the broker.

Links to all files used in this post:
login.config
activemq-ldap-with-anon-access.xml
activemq-ldap.xml - no anonymous access

14 Nov 2011

Consuming Topic messages in Spring and Camel

When using a Spring to consume topic messages, make sure to cache the JMS consumer somewhere. Otherwise your consumer may not receive all messages.

Consider the following Camel route definition that consumes messages from a JMS topic.

</bean>

</bean>

</bean>

...

</route>

</camelContext>

The camel-jms component is configured to use a Springs SingleConnectionFactory and a cache level of CACHE_SESSION. This combination will cause problems!

When using Springs SingleConnectionFactory, it will not cache the consumer. This ConnectionFactory only reuses the same connection but does not cache any other JMS resources on top of the connection (i.e. session and consumer).

At the same time Springs DefaultMessageListenerContainer (which is used by Camel to consume messages from a JMS broker) does not cache the consumer either as its cache level is set to CACHE_SESSION. The result of this configuration is that a new JMS consumer instance get created by Springs DMLC prior to requesting the next message from the JMS broker. After the message has been dispatched and processed, the consumer is destroyed (and unregistered from the broker).
In addition a JMS topic works differently from a JMS queue in the way that if there is no subscriber registered, the broker will discard the topic message. As with the above configuration the JMS consumer gets recreated for every message by Spring (and registered in the JMS broker), there are certain time windows when the broker does not have a topic subscriber registered and so it discards the messages it receives. The result will be that your Spring JMS topic consumer will not receive all of the messages from the broker.

When consuming topic messages in Spring, it is necessary to cache the JMS consumer. Either by using a connection factory that supports consumer caching like the Spring CachingConnectionFactory or by configuring the cache level of the DMLC to use CACHE_CONSUMER.

Please note that the ActiveMQ PooledConnectionFactory does not cache consumers!

This only applies to topic consumers in Spring. Topic producers will not use these cache level settings and hence don't have this problem.

7 Nov 2011

Loosing messages despite of sending within a transaction

Last week I made an interesting observation. I was able to loose persistent queue messages that were sent in a transaction to an ActiveMQ broker.

I believe its best to share the findings.

SETTING THE SCENE

Suppose you have a MessageProducer that sends messages to an ActiveMQ broker inside a transaction. This producer could be anything, a plain Java JMS Producer client, a Spring JMS producer or as in my case a Camel route. It also does not matter if the producer sends one or more messages within the same transaction.

Next the ActiveMQ broker instance is configured for a specific limit. In my test that limit was fairly low in order to reproduce the problem quickly. In addition I did set the property sendFailIfNoSpace so that the broker raises an exception back to the producer once any of its global limits have been reached. So the configuration I used reads

</memoryUsage>

</storeUsage>

</tempUsage>

</systemUsage>

Without sendFailIfNoSpace="true" ActiveMQ would block the producer until additional space is available on the broker. This would most likely not reproduce the problem I am talking about.

I ran the ActiveMQ broker with these settings, kicked off my Camel route and let it do what it can do best: route messages (which are finally sent to a queue on my ActiveMQ broker). Btw, I did not have any consumers attached to the broker; only the Camel route producer was connected. Still the same problem can occur with slow consumers.

And finally my producer did not register an exception listener. In my tests the producer was a Camel route and Camel currently does not register an exception listener.

THE PROBLEM

Now, the problem starts when the broker reaches its configured limits. Because of the property sendFailIfNoSpace="true", it will not block the producer indefinitely but throw an exception back to the producer. In my test I was hitting the storeUsage limit of 64 MB.

On the other hand when sending messages in a transaction, then the actual send is done asynchronously. That means the thread that is sending the message does not wait for the ack from the broker. It’s the transport thread that receives the broker’s ack and deals with it. Using an async send in the case of running inside a transaction is done for performance reasons. By not waiting for the broker ack, you can send messages more quickly to the broker. Until the transaction finally commits the messages received by the broker will not be placed onto the brokers queue.

Putting this together the following happened in my test:

- The producer started a new transaction. This information was sent to the broker, the broker accepted the new transaction.

- The producer then sent a message to the broker. As this send was within an existing transaction it was done async. The call to send the message returned immediately after sending the message. It did not wait for the broker’s ack.

- The broker did no accept this message but raised a "javax.jms.ResourceAllocationException: Persistent store is Full, 100% of 67108864" back to the producer.

- The producers transport thread received the exception but didn’t know how to handle it. Instead it printed the following debug log message:
DEBUG - ActiveMQConnection - Async exception with no exception listener: javax.jms.ResourceAllocationException: Persistent store is Full, 100% of 67108864. Stopping producer (ID:Mac.local-51375-1320685288867-2:1:1:1) to prevent flooding queue://TEST.IN.
It flagged the fact that there was an error but it did not mark the current transaction to be rolled back. This would have been the job of an exception listener, but none was registered.

- The producer then committed the transaction. From the producers point of view no errors did happen (it never became aware of the JMSException) so it asked the broker to commit the transaction.

- The broker committed the transaction just fine although the message did not get stored on the broker’s queue. From the brokers point of view all went fine. The producer did start a new transaction, it then sent some msgs, which the broker rejected and then the broker got asked to commit the transaction. All fine from the brokers point of view. It’s the producer’s task to deal with the JMSException and to decide whether to rollback or not.

- The message(s) that got sent inside the transaction is (are) lost!

It does not necessarily require an ActiveMQ broker that has reached its configured limits to run into this problem. If the transport for any reason fails to send the msg to the broker but works again on the transaction commit, then the same problem could arise. This is less likely to occur though. Transport related problems (e.g. connection loss) will most of the time affect the ability to commit the transaction as well (and not only the individual message send). When there is an exception during commit, it will correctly roll back the entire transaction.

You would think that using JMS Transactions would guard you against message loss. In general that is correct. However you need to make sure that any errors that can occur within the transaction are being dealt with. In this scenario, there was no exception listener registered in the producer that would deal with the JMSException. So I lost messages because I did not deal with the exception returned from the async send.

When using Camel as a message producer, you currently cannot easily register an exception listener, which then ensures the transaction gets rolled back in case of any problems.

Luckily there are some simple solutions to this problem:

THE SOLUTION

.. is to either

1) Question if you really need to send your messages inside a transaction. When sending only one persistent message at a time inside a transaction, there is not much benefit over sending the message without a transaction. Persistent messages are sent synchronously by default so the producer waits for the brokers ack before proceeding and it can deal directly with any error returned from the broker. Transactions are very useful though when sending multiple messages that need to all succeed or fail as one atomic operation.

2) If you need to use transactions for sending your messages to the broker, then configure the ConnectionFactory to always send messages synchronously rather than using the default async mode. Changing to sync send will have a small performance impact, particularly when sending a batch of messages within one transaction. This performance impact however is probably negligible in most scenarios.

You can switch to using a sync send by using the alwaysSyncSend=true property on the ActiveMQConnectionFactor, e.g. for a Spring configuration:

</bean>

Alternatively, set jms.alwaysSyncSend=true on the broker URL, e.g.

tcp://localhost:61616?jms.alwaysSyncSend=true, or
failover:(tcp://localhost:61616)?jms.alwaysSyncSend=true

3) Register a JMS exception listener programmatically so that it reacts on any exceptions thrown by the broker in case of sending the message async. For a plain Java JMS client you can use

ActiveMQConnectionFactory connectionFactory =

new ActiveMQConnectionFactory(brokerUrl); connectionFactory.setExceptionListener(new ExceptionListener() {

public void onException(JMSException ex) {
// handle the exception
}
});

You can also set this exception listener on an existing JMS Connection object.

The somewhat tricky part may be to figure out which transaction needs to be rolled back, in case of having multiple threads sending messages concurrently to the broker.

I already mentioned this option is not easily done in case where the producer is a Camel route. We hope to improve Camel to have an out of the box exception listener registered in future. Till then CAMEL-4616

captures this request.

If you want to be on the safe side, the go with option #2 and ensure a sync send of your messages.