16 Apr 2009

Slower performance with the new servicemix-jms endpoints?


There are two servicemix-jms components available in the later versions of ServiceMix 3. The traditional jms endpoint that is configured using <jms:endpoint> and the new jms endpoint that uses <jms:consumer>; and <jms:provider>. The new JMS endpoints reuse much of Spring JMS. Very recently I noticed a pretty large performance difference between these two JMS consumers, in the way that the new servicemix-jms consumer was much slower than the traditional consumer while using a pretty default configuration for both.
The reason for that is that out of the box the new JMS endpoint is configured to use the Spring JMS DefaultMessageListenerContainer without any caching, so all JMS resources get re-created and closed for every message being sent or received. This of course degrades performance a lot.

There are two simple solutions:

1) Configure caching for this DMLC

<jms:consumer service="test: NewJMSConsumer"
endpoint="endpoint"
targetservice="test:bean"
targetendpoint="endpoint"
destinationname="queue/TEST.FOO"
connectionfactory="#AMQConnectionFactory"
cachelevel="3">


Using cacheLevel="3" (CACHE_CONSUMER) is the highest cache option and caches the pretty much all JMS resources for each listener thread (JMS connection, session and message consumer). Other levels are 2 (CACHE_SESSION ), 1 (CACHE_CONNECTION) and 0 (CACHE_NONE). If no cacheLevel is specified, CACHE_NONE is used!

2) Or configure for a different Spring JMS listener

<jms:consumer service="test:NewJMSConsumer"
endpoint="endpoint"
targetservice="test:bean"
targetendpoint="endpoint"
destinationname="queue/TEST.FOO"
connectionfactory="#AMQConnectionFactory"
listenertype="server">


The ServerSessionMessageListenerContainer type internally uses a ServerSessionPool (which as the name suggests pools the server sessions) and also results in reasonable performance. However this listener type might not be suitable for all deployments.

Both options should boost performance and perform equally fast as the old JMS endpoint. These jms consumer attributes are both documented but it is easy to overlook them and just use the default configuration.
I recommend solution 1, using the DefaultMessageListenerContainer and enable caching.

Finally there seems no reason why caching could not be turned on by default. So I raised JIRA SM-1841 at Apache.

23 Jan 2009

Using the Camel Aggregator correctly

I was recently working on a project that required the use of a Camel aggregator. Since I had not used the aggregator much before I started by reading through the Camel Aggregator documentation. Some questions remained though.
In my use case I needed to aggregate every five incoming messages of the same format together into a new exchange. Sounds very simple but I did not find it that straight forward. And that's the motivation for this post.
To have an example at hand; let's say I want to aggregate messages of the following general XML format (with different values of course):

<artist genre='Alternative'>
<name>The White Stripes</name>
<album>Elephant</album>
<album>Get Behind Me Satan</album>
<album>White Blood Cells</album>
</artist>


Suppose that's the message format of the incoming messages. I will build the aggregator according to this format.
As mentioned in the Camel documentation, each aggregator is configured by a correlation expression and an aggregation strategy.
The correlation expression is used to define which messages should be aggregated together. When the correlation expression is being evaluated at runtime on a particular message, it returns a value that is used as the correlation id under which messages are being aggregated. All messages that evaluate to the same correlation id are aggregated together. A mistake I made initially was assuming that all messages that do not match my correlation expression will not be aggregated. But every correlation expression will always evaluate to a particular correlation id for every message. Suppose I want to aggregate all those messages that have an alternative genre (genre='Alternative') and I define my aggregation strategy accordingly (we will explore that later). All messages that have a different genre set will still be aggregated somehow, depending on what id the correlation expression evaluates to for these messages at runtime. We will see more detailed examples of that later, but this is important to keep in mind.
The aggregation strategy on the other hand contains the logic for combining all message exchanges into a single aggregated message. The default aggregation strategy is org.apache.camel.processor.aggregate.UseLatestAggregationStrategy, which only aggregates the latest incoming message, discarding older messages. This is great for throttling purposes but did not serve my use case.

Generally a Camel aggregator is set up in Java DSL as follows (I will only use Java examples, XML based configuration will be quite similar):


from("somewhere").aggregate("somehow").to("some destination");


Let's not care too much about the from(…) and to(…) part of this route, they will be defined by your business use case. It is the aggregate(…) part that we are interested in here.
As mentioned above the aggregator is configured via the correlation expression (which messages to aggregate together) and the aggregation strategy (how to aggregate messages). So it comes as no surprise that the aggregate() method can take the following parameters:


public AggregatorType aggregate(Expression correlationExpression, AggregationStrategy strategy);


When defining a correlation expression, one can either correlate on a particular message header or on the message content. Pretty much all examples I have seen so far did correlate messages on the existence of a particular message header element. In Camel headers are named and can take values - they are basically (name, value) pairs.
Suppose some of my messages also have a header called "MusicCollection" with a value "Yes" set. I can then aggregate my messages based on this header and define my correlation expression as follows:


.aggregate(header("MusicCollection"), …).


This correlation expression will evaluate to the value of the header "MusicCollection" at runtime (the String "Yes" in the above example). If no such header is set, the correlation expression will evaluate to null at runtime, which can still be used as a key for aggregating messages.
But what If I don't want to or cannot use message headers? Perhaps the message was received from an external system without any headers. I can then also correlate messages based on their message payload as well. Using an XPath expression I can test if a specific element or structure is present in the message payload. In my example the messages start with as the root element, so it seems natural to define a correlation expression like this:


.aggregate(xpath("/artist", …).



This however won't work as expected. Internally the aggregator will use the Camel XPathBuilder to evaluate the XPath expression and the above expression will return a new instance of org.w3c.dom.NodeList for every new message. The aggregation however can only occur on a correlation expression that evaluates to the same object or primitive value at runtime for all messages to be aggregated (as that object or primitive value is used internally as a HashMap key for storing the aggregated messages into exchanges). So we need to define an XPath expression that evaluates to the same object (or key) at runtime. The trick is to select a proper XPath expression and to cast the evaluation result into a type like java.lang.String, such as in the following example:


.aggregate(…).xpath("name(/artist)='artist'", String.class).


I had to move the XPath expression after the aggregate() method call because of the additional String.class argument. The above example will take the name of the root element of my message (which will be "artist") and test if it equals the string "artist". For the sample message shown in the beginning, this expression will evaluate to the String "true". For any other XML messages that don't start with , this XPath expression will evaluate to the String value "false" and these messages will still be aggregated according to my aggregation strategy.
Aggregating on only the root XML element does not seem very useful. I might want to aggregate on the genre attribute of the element instead. Here is the corresponding correlation expression:


.aggregate(…).xpath("/artist/@genre", String.class).


This does serve my use case. If I want to aggregate all messages with an "Alternative" genre I could use:


.aggregate(…).xpath("name(/artist[@genre='Alternative'])", String.class).


Notice this will aggregate all messages starting with together into a new exchange. All other messages that have a different genre will also be aggregated together into a different exchange, no matter what their genre value is. If you wondered why I use the XPath function name() here, then that is because using name() will return the String "artist" as the correlation key for the example message above (the name of the XML element that matches my XPath expression). If I omitted the name() function, it would return a string representation of the node set itself. In the case of my example message the correlation key would be the string "White Stripes Elephant Get Behind Me Satan White Blood Cells". Using "artist" as the key seems better somehow.
If you are unsure to what key your own correlation expression evaluates at runtime, place a breakpoint into method org.apache.camel.processor.aggregate.DefaultAggregationCollection.add() and debug your Camel route from there. Check the value of the computed correlationKey object.
You can of course define your correlation expression in different expression languages than XPath such as XQuery, SQL and others.

That should cover the correlation expression part of the aggregator. The second part deals with the aggregation strategy. Remember, the strategy defines how messages are aggregated together. The default strategy used by Camel is the UseLatestAggregationStrategy, which out of x received messages only takes the latest message. If however I want to aggregate multiple messages together into a single message, I need to write my own aggregation strategy. This isn't a big deal and the Camel aggregator documentation shows an example implementation. I have re-used that code for my example as well.


public class MyAggregationStrategy implements AggregationStrategy {
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
Message newIn = newExchange.getIn();
String oldBody = oldExchange.getIn().getBody(String.class);
String newBody = newIn.getBody(String.class);
newIn.setBody(oldBody + newBody);
return newExchange;
}
}


In this implementation we extract the messages of both Exchanges old and new, then simply concatenate the two messages together and place the result on the new Exchange. Nothing fancy here. We can then instantiate this class and pass its reference into the aggregator:


.aggregate(…, new MyAggregationStrategy()).


So we are done, right? Well, what we have not defined yet is a criterion as to when the aggregation is done. There are again multiple ways of doing so. If I want to aggregate a fixed number of incoming messages, I can easily use the batch options:


.aggregate(new MyAggregationStrategy()).
xpath("("/artist/@genre", String.class).
batchSize(5).
to(…);


In this example the aggregation will finish whenever it has processed 5 incoming messages no matter as to how they got aggregated (into which exchanges). Or I could specify a timeout after which the aggregated message is sent, irrespective of how many message have been aggregated:



.aggregate(new MyAggregationStrategy()).
xpath("("/artist/@genre", String.class).
batchTimeout(5000L).
to(…);


If I want to finish the aggregation when there are 5 messages aggregated together into one new exchange, then a more complex logic is required, which I can specify using the completedPredicate() expression:


.aggregate(new MyAggregationStrategy()).
xpath("("/artist/@genre", String.class).
completedPredicate(header("aggregated").isEqualTo(5)).


The example above assumes that there is a header called "aggregated" set and whenever its value reaches the number 5, the aggregation will finish. Of course some application level code needs to set this header and also increment the counter. This would typically be done in the custom aggregation strategy implementation, where increasing that number can be based on any application logic (see attached demo, which illustrates this approach).
I can also call a particular method on a Spring bean for deciding when to finish the aggregation by using a BeanExpression:


.aggregate(new MyAggregationStrategy()).
xpath("("/artist/@genre", String.class).
completedPredicate(
new BeanExpression("aggregatorStrategy", "isCompleted"))
).
to(...)


This requires a Spring bean with name "aggregatorStrategy" being defined in the Camel Spring configuration (your camel-context.xml) which exposes a public method called isCompleted().


<bean id="aggregatorStrategy"
class="com.progress.cs.camelaggregatortest.MyAggregationStrategy">


Check the attached demo for an example.
Finally, the entire aggregator code in Java DSL for my use case now reads:


from("…").
aggregate(new MyAggregationStrategy()).
xpath("/artist/@genre", String.class).
batchSize(5).
to("…");


Using a Camel aggregator can appear rather complex at first. But once you get to see some working examples it becomes much clearer (as with all things in life). I personally had some troubles getting the aggregator to work with an XPath based correlation expression. That part I wanted to share most of all in this article.

And finally here is the demo for this article. After extracting, follow the steps in README.txt to run it quickly.

29 Jul 2008

WS-Security woes in ServiceMix 3.2

Previously I had to get WS-Security UsernameToken based authentication working in ServiceMix 3.2 for a customer. All the customer wanted was to have an external SOAP client connecting securely to a service deployed in ServiceMix and have the client being authenticated and authorized by ServiceMix. Sounds like a straight forward use-case and the fact that there already is a ws-security demo in ServiceMix made me hope this would be a simple project.
As Murphy's Law dictates, it turned out to be the opposite.

My initial hope was to simply define a CXF-BC consumer that is WS-Security enabled and that authenticates the client and passes on the request to a CXF-SE service. So overall use-case is
External client -> CXF-BC consumer -> CXF-SE service.
This can be set up using Maven in less than 10 minutes. The tricky part was to configure security. The ws-security demo of ServiceMix already shows the configuration for using XML-Signature and XML-Encryption, so I could simply copy that part. For using WS-Security functions ServiceMix simply leverages WSS4J, so it becomes a matter of correctly configuring WSS4J interceptors for the incoming request and outgoing reply.
Adding WS-Security UsernameToken based authentication configuration to the WSS4J interceptor is relatively simple too:

<bean class="org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor"
id="TimestampSignEncrypt_Request">
<constructor-arg>
<map>
<entry key="action" value="Timestamp Signature Encrypt UsernameToken"/>
<entry key="signaturePropFile" value="alice.properties"/>
<entry key="decryptionPropFile" value="bob.properties"/>
<entry key="passwordCallbackClass"
value="org.apache.servicemix.samples.cxf_ws_security_usernametoken.KeystorePasswordCallback"/>
</map>
</constructor-arg>
</bean>

I simply had to add UsernameToken to the <action> element in the above WSS4JInInterceptor configuration (config for XML-Signature and XML-Encryption was directly taken from ws-security demo). I won't care about the client configuration here, as this could be any SOAP client but need to add that my client was configured to transmit username "alice" with password "password".
For authorization to work, I had to add these credentials to conf/users-passwords.properties

# users-passwords.properties
smx=smx
alice=password

and make alice a member of the administrators role in conf/groups.properties (for a real use-case I would have defined a new role):

# groups.properties
admin=smx,alice

Finally to restrict only users of the administration role to invoke my service, I had to change conf/security.xml

<sm:authorizationMap id="authorizationMap">
<sm:authorizationEntries>
<sm:authorizationEntry service="*: SOAPServiceWSSecurity"
roles="admin" />
</sm:authorizationEntries>
</sm:authorizationMap>

Easy I thought and went to testing it. But it did not work. There was simply no authentication happening. Only when debugging through the code I realized the CXF-BC component did not delegate these credentials to JAAS for authentication! So there was no authentication possible in CXF-BC! I raised a bug report and took a deep breath...
The good news is that by now this bug got resolved and things work as expected. However you need a very recent version of ServiceMix to have this fix included. Now if a client sends a WS-Security header containing username and password, these will be authenticated correctly by JAAS inside ServiceMix.
Still at the time of discovering the bug I had to get things working for the customer and got to know that the HTTB-BC component supports username/password based authentication. So I started to replace my CXF-BC consumer with a HTTP-BC consumer. And this is where the real pain started.
The configuration again is straight forward and does not look much different from the CXF-BC consumer configuration. However, for authentication to work in HTTP-BC, I must set soap="true" in my bc consumer config:

<beans xmlns:http="http://servicemix.apache.org/http/1.0"
xmlns:greeter="http://apache.org/hello_world_soap_http"
xmlns:soap="http://servicemix.apache.org/soap/1.0">
<http:endpoint service="greeter:SOAPServiceWSSecurityHTTP"
endpoint="endpoint"
role="consumer"
targetService="greeter:SOAPServiceWSSecurity"
targetEndpoint="endpoint"
locationURI="http://localhost:9000/"
defaultMep="http://www.w3.org/2004/08/wsdl/in-out"
soap="true">
<http:policies>
<soap:ws-addressing />
<soap:ws-security receiveAction="UsernameToken" keystore="default" />
</http:policies>
</http:endpoint>
</beans>


This is because the WSSecurityHandler that is called by the HTTP-BC consumer expects a DOM representation of the SOAP header, which won't get created unless soap="true" is set. With that I got the HTTP-BC component to authenticate the client, but the request failed to be un-marshaled by the CXF-SE runtime. Setting soap="true" in my http consumer had the side effect of stripping off the SOAP header from the request before sending it on to the next component inside ServiceMix. The CXF-SE runtime however requires a valid SOAP request, which it now did not get. Aargh...

So what to do with this dilemma? The easiest option I could think of was to insert a Camel component between the HTTP-BC consumer and the CXF-SE service that adds a default SOAP envelope to the In message using xslt transformation and removes the SOAP envelope from the Out message produced by the CXF-SE component again (as the http-bc component will add an envelope when soap="true" is set.) Not an ideal solution and rather a hack than a proper workaround. Another option would have been to use a custom CXF interceptor that takes care of adding/stripping the SOAP envelope but I felt a Camel XSLT component will be simpler to use.
I inserted the Camel component that did add/remove the SOAP envelope. Still I had a problem with the reply. The reply message had to be signed and encrypted but that was not the case, despite my configuration being correct. After another few hours of debugging (I am new to the codebase) I realized this part had yet not been implemented and I had hit another bug. Because of that bug outgoing SOAP reply message cannot be encrypted nor signed when using the HTTP-BC component.

I still found some other issues on the way that I raised accordingly. Password digests as defined by the WS-Security UsernameToken spec are currently not supported (passwords need to be send in clear text in the SOAP header - but the header can still be encrypted, so this is no security problem). And the call to authorization in ServiceMix does not produce any logging. Finally I raised a few documentation issues.

Lessons learned:
As you can see I hit a number of issues on the way. But the good news is that these will now be addressed and fixed in future versions of ServiceMix, making the product more stable and easier to use. The most important bug is already resolved so there is no need to use a HTTP-BC consumer anymore when doing WS-Security (and I highly recommend against it).

The demo that I worked on is already incorporated into the cxf-ws-security demo of ServiceMix in the current release.

The most difficult part for me was to hit very low level error messages that did not tell me much about the real cause of the problem. This can be quite frustrating and the only way I got to understand the error was to debug through the source code. So error reporting needs a hell lot of improvement in ServiceMix.

Secondly, the low-level XML configuration is highly error-prone. ServiceMix really needs proper tooling that free the developer from configuring xml files. It is not only easy to make mistakes in the config, there are also so many configuration options that have implications unknown to the regular developer (like the soap="true" story).

28 May 2008

... the namespace on the "message" element, is not a valid SOAP version.

Recently I was working on a SMX usecase:
external client -> CXF BC consumer -> CXF SE

All configured nicely it still failed at runtime with

INFO: Interceptor has thrown exception,
unwinding now org.apache.cxf.binding.soap.SoapFault:
"http://java.sun.com/xml/ns/jbi/wsdl-11-wrapper",
the namespace on the "message" element, is not a valid SOAP version.


It took me a while to figure out what's wrong. In the end I only had to set useJBIWrapper="true" in the config of my CXF SE component:


<beans xmlns:cxfse="http://servicemix.apache.org/cxfse/1.0">
<cxfse:endpoint useJBIWrapper="true">
<cxfse:pojo>
<bean class="org.apache.servicemix.samples.cxf_ws_security.GreeterImpl" />
</cxfse:pojo>
</cxfse:endpoint>
</beans>


Those error messages really need to get improved in SMX. There are too many low level errors raised that give you no indication of what the real problem is.

15 Apr 2008

Chronic - a date and time parser for Ruby

I have recently come across Chronic, a very elegant Date and Time parser library for Ruby. It can be easily installed via RubyGems and can process all sorts of dates and times written in natural language formats. If you need to parse date in Ruby this might be the library you have been looking for.

There is an interesting post on parsing US and non-US date formats in this post.

7 Apr 2008

Kerberos authentication in Java

Have you used Kerberos authentication in Java using JGSS? The GSSContext interface defines the two methods initSecContext() and acceptSecContext() for an initiator and acceptor of a Kerberos authentication request. JGSS is configured in a file typically called jaas.conf, where you specify the Kerberos keytab files, Kerberos principal name and other properties that configure the authentication process.

According to the Kerberos authentication algorithm, only the initiator (or client) needs to contact the Kerberos authentication server, asking for a service ticket. The service ticket is later passed on to the server that the client wants to authenticate against and it contains all the information that the server (or acceptor) requires for authenticating the client. That implies the server does not need to communicate with the Kerberos authentication server when authenticating a client, as it is emphasized in pretty much every description of the Kerberos protocol.

Still when using JGSS you might have noticed that your service does contact the Kerberos authentication server as part of the authentication process. The server will request a ticket granting ticket from the KDC even though this is not needed. This behavior has caused me some headaches in the past as I did not understand why the server would need to contact the KDC.

There is a not very much documented property in jaas.conf that will solve this issue. If in your com.sun.security.jgss.accept stanza of jaas.conf you specify isInitiator=false, then the server will not request a TGT. Officially this property is only supported from Java SE 6 b89 onwards, however I verified that the later patches of Java 5 also include it.
So when setting up jaas.conf, I recommend adding isInitiator=false to the configuration of the acceptor.

This topic is also discussed here.

4 Apr 2008

Let's change the default transport protocol used in ServiceMix 3

Today I want to post about a potential issue that I came across recently when running tests using the FUSE ESB (which is IONA's distribution of Apache ServiceMix).

Currently ServiceMix 3.2 configures the underlying ActiveMQ broker to simply use tcp as the transport protocol. This is defined in conf/servicemix.properties:

activemq.url = tcp://${activemq.host}:${activemq.port}

While this configuration works fine and is sufficient for using ServiceMix in development, it will not recover from any network failures and is therefore not suitable for running in a production system. In case of a network failure or connection inactivity timeout, ActiveMQ will simply raise an exception but won't try to reconnect. You will need to restart ServiceMix in order to recover from such error. For automatic reconnection to happen, the failover protocol should be used instead. Hence the default configuration should be changed to

activemq.url = failover://(tcp://${activemq.host}:${activemq.port})

The failover protocol should really be the default protocol and enabled out of the box. I logged SM-1302.

29 Feb 2008

Bug in gdm under SuSE 10.3 (and perhaps in other Linux distributions)

Anyone who is running Linux with GNOME in VMWare might have come across a rather serious problem with gdm lately after doing an upgrade of the system with the latest patches. I am running OpenSuSE 10.3 and installed the latest patches just a few days ago. The patch installation went fine but next morning when I tried to reboot my Linux image, gdm failed to start up. Gdm initially started up but then crashed just before or shortly after showing the login prompt. When gdm was started the first time it ran a few seconds longer before crashing. You won't even have enough time to login. On subsequent restarts, GDM crashed almost instantly for every restart attempt. It attempted 6 restarts, then raised the following error to the console, slept for 2 minutes and started over again:

"The display server has been shut down about 6 times in the last 90 seconds, it is likely that something bad is going on. Waiting for 2 minutes before trying again on display:0"

The log in /var/logs/Xorg.0.log should had the following lines at the end:
Backtrace:
0: /usr/bin/X(xf86SigHandler+0x81) [0x80e6d81]
1: [0xffffe420]
2: /usr/bin/X [0x817b185]
3: /usr/bin/X(CompositePicture+0x150) [0x8161be0]
4: /usr/bin/X [0x8167a1f]
5: /usr/bin/X [0x8164d75]
6: /usr/bin/X [0x815809e]
7: /usr/bin/X(Dispatch+0x1af) [0x808f68f]
8: /usr/bin/X(main+0x47e) [0x807717e]
9: /lib/libc.so.6(__libc_start_main+0xe0) [0xb7d1ffe0]
10: /usr/bin/X(FontFileCompleteXLFD+0x1e5) [0x8076501]

Fatal server error:
Caught signal 11. Server aborting


After various tries I was still unable to resolve the problem but found a way to bring up the X Windows system. I had to login as root from console and run gdm -stop. The next time gdm was restarted (after the 2 min had elapsed) it did come up fine.

So I went on the web to see if someone else had encountered the same problem as well. When I checked SuSE's bug database, I realized I was not the first one to encounter this issue. A bug was already raised https://bugzilla.novell.com/show_bug.cgi?id=350318.

Apparently the culprit is a patch to the Control-Center2 (https://bugzilla.novell.com/show_bug.cgi?id=337434) that seens to break gdm.

This is quite a severe problem and only seems to affect Linux running inside VMWare. I am not yet sure if other distributions are effected as well, most likely they are.

25 Feb 2008

simple Ruby SOAP client

I am fairly new to Ruby and am quite amazed how simple and elegant this language can be. In particular I was surprised how simple it is to write a SOAP client in Ruby. Where in most programming languages even a simple SOAP client takes about two screenshots of code (if not more), a Ruby SOAP client can be boiled down to these few lines of code:


#soap_client.rb
require 'soap/wsdlDriver'

WSDL_URL="file://C:/hello_world.wsdl"

soap = SOAP::WSDLDriverFactory.new(WSDL_URL).create_rpc_driver("SOAPService", "SoapPort")

#invoke WSDL operation sayHi
result = soap.sayHi()
printf "Returned message reads: " + result + "\n"

#invoke WSDL operation greetMe
result = soap.greetMe("New Ruby User")
printf "Returned message reads: " + result + "\n"



In this code I am referring to a wsdl definition that defines two WSDL portType operations named sayHi() and greetMe() that both return a string.
In essence the code boils down to 4 lines of code that are of relevance. Of course this code is everything but object oriented and it is somewhat unfair to compare it against a properly implemented SOAP client written in other object oriented programming languages. Still this example is surprisingly simple and straight forward. I have used this code a number of times in order to quickly implement a SOAP test client. All you need to change is the name of the business method you actually want to invoke on (plus their arguments) and the location of the WSDL file. Thanks to Ruby you don't even need to recompile your changes.
Sure, most SOAP toolkits also provide a way to generate a fully working client and server without requiring you to write any code. Still the generated code is tied to the WSDL interface. Changes to one of the operation signatures typically require a regeneration of your test client/server.
Ruby rocks (and I am just starting to hit the tip of the iceberg)!