This great post from Tomas on
receiving ack/nack notifications from the MSMQ[C] adapter prompted me to write up
something I had researched a few weeks back…
But first, a bit of background…
When doing work with message queueing systems, a very common (and well supported)
convention exists when two systems are exchanging messages: System ‘A’ sends a message
to System ‘B’. When System ‘B’ wishes to reply, it uses the message identifier of
the ‘request’ message as the correlation identifier of the ‘reply’ message.
The reply queue used is either well known by both parties, or is communicated
as a property on the request message. For purposes of discussion, let’s call
this the “queueing contract”.
Both MSMQ and MQSeries have MessageId and CorrelationId properties on messages.
When looking at messages in a reply queue, System ‘A’ can peek at the messages, looking
for a particular CorrelationId prior to issuing a receive. Alternatively, it
might use the CorrelationId as a look-up into working state of some sort.
To facilitate request-response interactions, MQSeries has had a ReceiveByCorrelationId
API for a long while, whereas MSMQ added this in 3.0 (that is, on Windows XP and Windows
2003.) (Actual property and method names may vary in real life…)
There are, of course, many deployed systems in the wild that implement the queueing
contract – using both MSMQ and MQSeries. You may find yourself needing to integrate
with these. At first blush, it seems that consuming a service such as this should
be a breeze from within BizTalk orchestrations – after all, accomplishing
correlation is a first-class-citizen feature for BizTalk, so how tough can this
be?
Correlation in BizTalk is indeed a first-class citizen…for data within your
documents (or promoted message context.) The classic example is to define
a property schema that embodies a business identifier like “PONumber”, and then visit
all schemas that will be used across a set of message interactions – promoting
the appropriate field in each as a “PONumber”. Then, a correlation type
and correlation set are defined which reference that property, and Send
and Receive shapes are set to either initialize or follow the correlation as
appropriate.
The problem with an orchestration consuming the queuing contract we described
earlier is that this contract occurs at the transport layer. To
be a consumer of the queueing contract, an orchestration would like to send a message,
and initialize a correlation based on the MessageId…but the MessageId is produced
further downstream, at the point an actual MSMQ or MQSeries message is generated by
an adapter. Moreover, an orchestration would subsequently like to receive a
message, following an initialized correlation – but the CorrelationId isn’t promoted by
the MSMQ adapter (though it is for the MQSeries adapter…)
Solving for the MSMQ Adapter
The MSMQ Adapter gives us a small amount of help to get past this…We can
in fact do our initial send operation through a solicit-response port (rather than
a one-way port) and get back the MessageId that was generated by the adapter — it
actually comes back in a single-element document with a tag named “MSMQMsgId”.
Starting here, the approach I took to solving the whole of the problem looked like
this:
-
Send a message through a MSMQ solicit-response port. The response operation
in the orchestration designer can be of type “String”. -
On the response half of the solicit-response port, execute a pipeline & pipeline-component
that extracts the returned MessageId and promotes it as the CorrelationId (using
the standard MSMQ adapter namespace for that property.) (For tracing purposes
& useability, the MessageId is also returned in a <string> wrapper by the
pipeline component rather than the native <MSMQMsgId> wrapper.) -
When the initial message (containing the correlation id in both content and context!)
is received back into the orchestration, it is sent back out to initialize
a correlation set (typed as MSMQ.CorrelationId.) This Send shape is just a “dummy
send”…it exists just to initialize the correlation, because that is the only means
given to us in the orchestration world. I used Patrick
Wellink’s “NOPE” Adapter for this purpose, but there would be other
options as well (including Tomas’
Null Send Adapter, which I just recently rediscovered and which apparently performs
better under load.) -
Next, we receive the “real” response message – following the correlation that was
just initialized. Because MSMQ.CorrelationId is not promoted by default,
the response is brought through a second pipeline component that performs that service
for us.
See the diagram below (and the downloadable
sample) for more detail.
(click)
What can go wrong with this approach? Well…there is race condition that can
occur if the “real” response message (step 4 above) arrives before steps 2 and 3 can
execute and establish the correlation (and associated instance subscription.)
In other words, if the “real” response message comes back before the orchestration
has had a chance to express interest in that particular message, a routing failure
will occur. This might be highly unlikely for your scenario (given response
times of the service you are interacting with) but it is something to test for, nonetheless.
(With BizTalk 2006’s ability to route failed messages, you could potentially catch
and retry in this scenario…)
(Note – if you are going to rely on the technique in the sample, consider asking Microsoft
support for BizTalk 2004 QFE 1647, which addresses an issue with truncated correlation
identifiers being returned to BizTalk…)
Solving for the MQSeries Adapter
Consuming the queueing contract with the MQSeries adapter for BizTalk can be done
with a technique quite close to the one just discussed – but with a little less work. The
response to the initial solicit-response interaction returns with MQMD_MsgID already
copied to a MQSeries.BizTalk_CorrelationId property. Likewise, the when
receiving the “real” response message, MQMD_CorrelationId is copied to the
MQSeries.BizTalk_CorrelationId property. This means all correlation can be done
with this one property – no pipelines/pipeline-components are needed!
From the documentation:
(click)
(The “dummy send” – though not pictured – would be required here as well to initialize
the correlation set.)
In addition, there is a much cleaner solution that the BizTalk 2004
MQSeries adapter provides. It turns out that MQSeries allows the “caller”
to assign the message ID (unlike MSMQ), so within BizTalk, you can in fact set
MQMD_MsgID and MQMD_CorrelationId to the same value prior to doing
your initial send. Now, you can initialize a correlation set based on MQMD_CorrelationId
with the first outgoing message, and follow the correlation for the “real”
response message. Slick!
From the documentation:
(click)
Solving for the MQSeries Adapter in BizTalk 2006
Ahhhh, here we have what looks like real simplicity…There is a new feature in the
BizTalk 2006 MQSeries adapter termed “Dynamic Receive” (referenced in the BizTalk
2006 Adapter Enhancements document.) For a solicit-response port, Dynamic
Receive allows context properties on the outbound message to determine which server,
queue manager, and queue should be “watched” for the response message.
This is used in conjunction with a “match” option – which allows you to wait for a
particular message based on CorrelationID (or any of several other properties, such
as MessageID, GroupID, SequenceNumber, Offset, or MsgToken…)
Not only does this give us quite a bit of flexiblity for where response messages
should be found (we don’t need a fixed receive location anymore) but it also elminates
the whole of the problem of implementing the “queueing” contract within orchestrations.
With this feature in place, an orchestration simply uses one solicit-response
port, and the work is done.
Needless to say, it would be fantastic to get similar support from the MSMQ Adapter
(at least the ability to have a single solicit-response port with a similar “match”
criteria option, even without the dynamic receive location.) But I don’t think
that is in the cards for BizTalk 2006…
Notes on the download:
-
See the BTMSMQCorrelation.PortBindings.xml file for the names of the two queues and
two file directories that must be created for this sample. -
Install Patrick
Wellink’s “NOPE” Adapter or choose a different strategy for dummy sends
(like Tomas’ Null Send
Adapter.) -
Use the included Deployment
Framework-based deployment, or deploy manually. -
To run, launch the “TestQueueServer” console app that will implement the queueing
contract (request/response) -
The sample will import and run just fine in BizTalk 2006, though you will have to
deploy manually. (Still working on the 2006 version of the deployment framework
– more later…)
Hopefully this is helpful to those doing queueing work with BizTalk – enjoy!
>