Creating ActiveMQ Broker cluster topologies using Fuse Fabric

Fuse Fabric is a very powerful open source integration platform (iPaaS) developed by Red Hat (former FuseSource). If you don't know what Fabric is, have a look at the project web site: http://fuse.fusesource.org/fabric/.
Introducing Fabric in this post would mean to simply repeat the documentation of the Fabric web site, so I'll spare that.

Instead, this blog post will attempt to deep dive a bit into the fabric:mq-create command. I hope this post will be useful to anyone who uses Fabric and wants to setup highly dynamic broker networks using Fabric. Let's get right into it...



The fabric:mq-create command allows to create broker instances that run in their own OSGi containers. For help on general usage type

fabric:mq-create --help

In Fabric every broker has a broker name and is part of a broker group. The broker name is the last argument specified on the mq-create command and names an individual broker. The group name is specified using the --group argument. If that argument is omitted, the broker gets the default group name called 'default' assigned. These concepts of broker names and group names are important as the next sections will show.
fabric:mq-create creates a new Fabric profile that contains the specified broker configuration. The profile name is the same as the broker name which is specified as the last argument to mq-create. This profile can then be easily deployed to one or multiple containers.
The fabric:mq-create command is quite powerful and can also be used to create more complex broker topologies. Here is a quick walk-through of some of the most common topologies.

Single Master/Slave Broker Pair

For high availability of a broker one may want to configure a master/slave broker pair. The main idea in Fabric is that two broker instances that use the same group name and the same broker name will form a master/slave pair. Using this idea a master/slave pair can be created using

fabric:mq-create --create-container broker1,broker2 MasterSlaveBroker

This command creates a Fabric profile called MasterSlaveBroker. in this profile it configures an ActiveMQ broker with the same name MasterSlaveBroker. As no --group argument was specified the group name will be 'default'. The option --create-container broker1,broker2 also deploys this profile to two new OSGi container instances called broker1 and broker2. As both container instances deploy the very same profile, they will instantiate a broker with the same broker name and the same group name and as such form a master/slave pair.
Note: The option --create-container is really optional. Its also possible to simply create the Fabric profile first using mq-create and then in a later step deploy this profile to different container using fabric:container-create-* commands.

Broker network with two interconnected Master/Slave pairs

Extending the previous topology to create two broker pairs that are interconnected using a network connector. Each pair consists of a master and slave broker instance. This can be created by invoking

fabric:mq-create
    --group network-brokers1
    --create-container broker1_1,broker1_2
    --networks network-brokers2 
    --networks-password admin
    --networks-username admin
    MasterSlaveBroker1

fabric:mq-create
    --group network-brokers2
    --create-container broker2_1,broker2_2
    --networks network-brokers1 
    --networks-password admin
    --networks-username admin
    MasterSlaveBroker2

These commands create two Fabric profiles. Each profile configures an ActiveMQ broker with the names MasterSlaveBroker1 and MasterSlaveBroker2. Each broker configuration also sets a group name so that the broker instances will be part of that group. Further the MasterSlaveBroker1 that is part of the group network-brokers1 configures a network connector to the broker instances in the group network-brokers2 and vice versa.
Using --create-container we instantiate two OSGi container in each command that each deploy the relevant Fabric profile and create the broker instances. The two container in each --create-container argument will form a master/slave broker pair as they both use the same broker name (either MasterSlaveBroker1 or MasterSlaveBroker2) and the same group name (either network-brokers1 or network-brokers2).
By default the brokers created by mq-create are secured and require authentication. So when configuring the network bridge it is then necessary to supply username and password credentials in order to successfully establish a network bridge. Username and password are supplied using the arguments --networks-password admin --networks-username admin.
Altogether these two commands create four broker instances out of which only two will be master broker instances, the other two will be slave instance. Each master/slave pair belongs to one broker group. A network bridge between the two master will be established in both directions. If one of the master broker dies or gets shut down, the slave broker will take over within a few seconds and the network bridges will get re-established from and to the new master broker.

Fully connected Broker Mesh

The above example sets up a master/slave pair for each broker group, where only one instance is active at a time. Its also possible to configure for multiple active broker instances within the same group. For the broker instance to be active independent of other instances it simply needs to have a unique broker name within the same group. These instances can also be networked using a full mesh topology.
When running

fabric:mq-create
    --group BrokerClusterMesh
    --networks BrokerClusterMesh
    --create-container MeshBroker1 
    --networks-password admin
    --networks-username admin
    BrokerClusterMesh1

fabric:mq-create
    --group BrokerClusterMesh
    --networks BrokerClusterMesh
    --create-container MeshBroker2 
    --networks-password admin
    --networks-username admin
    BrokerClusterMesh2

it will again create two broker profiles named BrokerClusterMesh1 and BrokerClusterMesh2. Each profile configures an ActiveMQ broker. Both broker configurations are part of the same group BrokerClusterMesh. Using --create-container, each profile gets deployed to exactly one OSGi container. Since both broker instances have their own broker name configured (BrokerClusterMesh1 and BrokerClusterMesh2) they will be both active broker instances within the same group BrokerClusterMesh. Using --network BrokerClusterMesh a network bridge is configured in each broker that points to its own network group name. This in essence will create a network bridge from each broker to all the other broker instances within the same group and form a full mesh topology. In this example only two broker instances get created so its a fairly small mesh. However you can easily add another broker to the group, e.g. by running

fabric:mq-create
    --group BrokerClusterMesh 
    --networks BrokerClusterMesh 
    --create-container MeshBroker3 
    --networks-password admin 
    --networks-username admin
    BrokerClusterMesh3

and all broker instances (the new one that is introduced as well as the two existing instances) will each reconfigure their network connectors to connect to all the other broker instances in this group. So a full mesh of 3 broker instances gets created. This mesh can be expanded with additional instances if needed. Once a new instance is introduced all broker instances reconfigure their network bridges accordingly.
Note: Its generally not advisable to create large broker meshes (e.g. > 4 broker instances) as depending on the use case it will cause some chatter on advisory messages that these broker instances exchange.

Full Broker Mesh with Master/Slave pair on each Broker

Combining the last two use cases its also possible to configure a full broker mesh where each broker consists of a master/slave pair. This is achieved using the following commands:

fabric:mq-create
    --group BrokerClusterMeshWithSlave 
    --networks BrokerClusterMeshWithSlave 
    --create-container MeshBroker1,MeshBroker2 
    --networks-password admin 
    --networks-username admin 
    BrokerClusterMeshWithSlave1

fabric:mq-create 
    --group BrokerClusterMeshWithSlave 
    --networks BrokerClusterMeshWithSlave 
    --create-container MeshBroker3,MeshBroker4 
    --networks-password admin 
    --networks-username admin 
    BrokerClusterMeshWithSlave2

This command differs from the previous example only in the --create-container argument. The previous example deploys the broker configuration to only one container. Now we deploy each broker configuration to two containers. Each container within --create-container will use exactly the same broker configuration (i.e. the same broker name and the same group name) and both instances will therefore form a master/slave broker pair. Each master broker will create a network bridge to all other active broker instances within the same network group.
Its of course possible to add additional master/slave pairs to this broker group if needed and all active broker instances (i.e. master broker instances) will reconfigure their network bridge dynamically as new brokers enter or leave the network group. To add another master/slave broker pair to the mesh you can simply run

fabric:mq-create 
    --group BrokerClusterMeshWithSlave 
    --networks BrokerClusterMeshWithSlave 
    --create-container MeshBroker5,MeshBroker6 
    --networks-password admin 
    --networks-username admin 
    BrokerClusterMeshWithSlave3

Additional notes

- Each of the broker profiles that get created by the above mq-create examples all create a profile that has the profile mq-base as the parent.
- The profile mq-base contains a broker.xml configuration file which serves as the basis broker configuration for all the broker instances created above. So you could upfront adjust this broker.xml in mq-base to your needs (e.g. configure systemUsage and policy entries) and then create your broker instances using mq-create and they will all leverage this configuration.
- The broker configuration in mq-base does not configure any network bridges. The configuration about network bridges is not stored in the broker.xml when using mq-create. Instead it is stored in the configuration file org.fusesource.mq.fabric.server-.properties that is created when running mq-create. This file stores all network connector related configuration using the 'network.' keyword. E.g.

network.password=admin
network=BrokerClusterMeshWithSlave
group=BrokerClusterMeshWithSlave
network.userName=admin
broker-name=BrokerClusterMeshWithSlave1

The network bridge gets created and configured based on this configuration and not based on activemq.xml! This allows for fairly dynamic configuration changes at runtime.
- The network connector [configuration](http://activemq.apache.org/networks-of-brokers.html) of ActiveMQ allows further fine grained configuration. This configuration can be added manually to the configuration file org.fusesource.mq.fabric.server-.properties in each profile by prefixing the configuration property with 'network.'. E.g. in order to set the property decreaseNetworkConsumerPriority=true on the network connector, one can simply add

network.decreaseNetworkConsumerPriority=true

to org.fusesource.mq.fabric.server-.properties. Likewise with all the other network connector properties.