We had an issue, the DTA database was empty, nothing was being tracked, and we were seeing a weird error message that pointed at something else in the event log:
Either another TDDS is processing the same data or there is an orphaned session in SQL server holding TDDS lock. Either another TDDS is processing the same data or there is an orphaned session in SQL Server holding TDDS lock. SQL Server: \. Database: BizTalkDTADb
Paul has a week of workflow web-casts on going for those of you who missed PDC or missed our sessions at PDC. The full detail and links to recordings are here: http://blogs.msdn.com/pandrew/articles/460630.aspx
Wednesday: Workflow + Messaging + Services: Developing Distributed Applications with Workflows
Thursday: Developing Event Driven State Machine Workflows
Friday: Extending Workflow Capabilities with Custom Activities
I worked through a problem recently with a client that really took me by surprise
– because I would think that many BizTalk shops would be running into this issue regularly.
So! Here goes with an explanation and a solution.
We ran into this problem initially using the MSMQ (not MSMQT) adapter with BizTalk
2004. We had roughly 10 MSMQ receive locations, as well as a few send ports
that were using the loopback
adapter. These were all executing in the same host.
The initial symptom was that the loopback adapter appeared to not work – messages
were just not getting through! They sat in the “delivered, not consumed state”
for no good reason. But we quickly reproduced the problem with just MSMQ receive
locations (i.e. without the loopback adapter.)
On a single processor virtual machine, the repro looked like this: Create four
MSMQ receive locations, and one MSMQ send port (with the send port subscribed
to one of the receive ports, just to keep things easy.) No messages
will flow through the send port at all.
To repro: This
download has a binding file with receive ports/locations for local (non-tx) private
queues Q1-Q4, plus a send port for local private queue NONTXQ with a filter for the
first receive port. There is also a bit of VB script to put a message into a
queue…If you turn off one of the receive locations for Q2-Q4, you’ll find things
work just fine. If you don’t, then (reiterating) no messages will flow through
the send port.
What was the resolution? Well, with BizTalk 2004 Service Pack 1 installed,
you can create a “CLR Hosting” key under the registry service definition.
In our case, we actually had to increase these values – you should determine the values
you need through testing. Consider having min worker threads equal to 7x the
number of MSMQ receive locations, and max worker threads equal to 10x the
number of MSMQ receive locations. (More on these numbers later…)
Does the documentation address this? Good question. If you look at
the topic “Managing Multiple Receive Locations” in the MSMQ adapter documentation, you
will find some reference to this. It indicates you should create a “CLR Hosting”
key as described above…but no actual values are mentioned (clearly just
a documentation mishap.)
But why do these have to be tweaked at all? Good question. The
documentation for the MSMQ adapter has some unfortunate quotables, like:
To increase performance, Microsoft BizTalk® 2004 Adapter for MSMQ
is multi-threaded. If you have many receive locations, there may not be enough threads
available for all the receive locations. This prevents some of the receive locations
from picking up messages.
The reality is that you really shouldn’t have to starve any particular receive location
because of a lack of threads…you should just wind up with increased latency.
But, such is not the implementation of the MSMQ adapter (at least for BizTalk
2004.)
Some background: The MSMQ adapter has a “Batch Size” parameter and a “Serial
Processing” parameter that can be set per receive location. “Batch Size” determines
how many messages the adapter will attempt to read from the queue (and submit to the
message box) on each iteration. “Serial Processing” determines whether one thread
is engaged in the peek/get/submit activity per receive location (Serial Processing
= ‘true’) or multiple threads (Serial Processing = ‘false’). If “Serial
Processing” is true, the “Batch Size” is forced to one regardless of the actual setting.
So what is the execution flow for a given receive location? The internal class
MsmqReceiverEndpoint is instantiated per receive location, and when it initializes,
it calls ThreadPool.QueueUserWorkItem with
a reference to itself. If “Serial Processing” is false…it does this
exactly seven (7) times.
What does it do with the QueueUserWorkItem callback? Well, when MsmqReceiverEndpoint.ProcessWorkItem
is called, it enters into a do/while loop that doesn’t exit until the
endpoint (receive location) becomes invalid (i.e. the receive location is shut town.)
In other words, ProcessWorkItem sits on a .NET thread pool thread – and if Serial
Processing is false, it sits on seven of them. The do/while loop executes
a peek on the queue (with a hard-coded 10 second timeout), and if there are messages
waiting, it receives up to “Batch Size” and submits them to the message box. (It
will give up attempting to receive a “Batch Size” worth of messages if the 10 second
timeout is reached on any attempt within the batch receive loop – i.e. if you drop
a single message on a queue, and the batch size is greater than one, expect to wait
10 seconds before further activity begins…) The behavior of consuming seven
threads per queue leads to the recommendation of MinWorkerThreads = 7x MSMQ receive
locations provided above.
Now, I confess – I’m not a BizTalk adapter expert. But, this design
seems to be in conflict with the advice offered in “Writing
Effective BizTalk Server Adapters“, where it says:
Don’t starve the .NET thread pool: …While starving
the .NET thread pool is a risk to all asynchronous programming in .NET, it is particularly
important for the BizTalk Server adapter programmer to watch out for this. It
has impacted many BizTalk Server adapters: take great care not to starve the .NET
thread pool. The .NET thread pool is a limited but widely shared resource. It
is very easy to write code that uses one of its threads and holds onto it for ages
and in so doing blocks other work items from ever being executed….If you have multiple
pieces of work to do (for example copying messages out of MQSeries into BizTalk Server),
you should execute one work item (one batch of messages into BizTalk Server) and simply
requeue in the thread pool if there is more work to do. What ever you do, don’t
sit in a while loop on the thread.
Is this fixed in BizTalk 2006? Surely it is… And, in fact,
it sure seems to be in Beta 1. The design of the adapter is a bit different…First, “Serial
Processing” refers to whether additional messages will be received from
the queue prior to the “EndBatchComplete” event being set (downstream
of IBTDTCCommitConfirm.Done.) (This part of “Serial Processing” is
true for BizTalk 2004 as well, along with forcing the batch size to one.) “Serial
Processing” in BizTalk 2006 does not affect how many threads will be reading
from your queue – you will have just one (despite what the Beta 1 docs say…),
unless you have multiple host instances in play. (That one thread using
a large batch size and operating with serial processing set to ‘false’ – not
blocking on the actual message box submission – should keep up with a fairly
large message arrival rate, but multiple host instances might be needed for your particular
case.)
More importantly, the ProcessWorkItem implementation returns immediately after a single
peek/get/submit operation (and simply calls QueueUserWorkItem again, per the advice
cited above.) (Side note: There seems to be some room in the design for the
idea that you in fact woudn’t return immediately if more than a threshold number
of messages were received, but currently this condition is “if # of messages received
> BatchSize”, which won’t ever happen.)
So what should I do for now with BizTalk 2004? For those using the
MSMQ Adapter with BizTalk 2004…consider whether you can set “Serial Processing”
equal to true. Keep in mind this forces you to a batch size of 1, so this
might not work depending on your message arrival rate. If you test this configuration
and find an unacceptable performance loss, consider setting the MinWorkerThreads value
to 7x the number of MSMQ receive locations you are maintaining, and
MaxWorkerThreads to roughly 10x (to provide breathing room.)
As an alternative, spread your receive locations among multiple hosts (though avoid
an over-proliferation of hosts – that has its own issues.)
And never draw any conclusions until you have performance tested at load with your
final host configuration – that is, your final allocation of send handlers, receive
handlers, send ports, and receive locations among your hosts! Other adapters
may affect the outcome if they involve polling on the receive side, or polling on
the “response” side of a solit-response send port. (If they use a thread pool
thread to do their work, they can be affected by any adapter that consumes threads
whether they themselves are written correctly or not!) Finally, I’ve heard from
a gentlemen who has done extensive testing that the threading parameters above
are useful/necessary when using large numbers of MSMQT receive locations as well.
Never a dull day in BizTalk land…!
I worked through a problem recently with a client that really took me by surprise
– because I would think that many BizTalk shops would be running into this issue regularly.
So! Here goes with an explanation and a solution.
We ran into this problem initially using the MSMQ (not MSMQT) adapter with BizTalk
2004. We had roughly 10 MSMQ receive locations, as well as a few send ports
that were using the loopback
adapter. These were all executing in the same host.
The initial symptom was that the loopback adapter appeared to not work – messages
were just not getting through! They sat in the “delivered, not consumed state”
for no good reason. But we quickly reproduced the problem with just MSMQ receive
locations (i.e. without the loopback adapter.)
On a single processor virtual machine, the repro looked like this: Create four
MSMQ receive locations, and one MSMQ send port (with the send port subscribed
to one of the receive ports, just to keep things easy.) No messages
will flow through the send port at all.
To repro: This
download has a binding file with receive ports/locations for local (non-tx) private
queues Q1-Q4, plus a send port for local private queue NONTXQ with a filter for the
first receive port. There is also a bit of VB script to put a message into a
queue…If you turn off one of the receive locations for Q2-Q4, you’ll find things
work just fine. If you don’t, then (reiterating) no messages will flow through
the send port.
What was the resolution? Well, with BizTalk 2004 Service Pack 1 installed,
you can create a “CLR Hosting” key under the registry service definition.
In our case, we actually had to increase these values – you should determine the values
you need through testing. Consider having min worker threads equal to 7x the
number of MSMQ receive locations, and max worker threads equal to 10x the
number of MSMQ receive locations. (More on these numbers later…)
Does the documentation address this? Good question. If you look at
the topic “Managing Multiple Receive Locations” in the MSMQ adapter documentation, you
will find some reference to this. It indicates you should create a “CLR Hosting”
key as described above…but no actual values are mentioned (clearly just
a documentation mishap.)
But why do these have to be tweaked at all? Good question. The
documentation for the MSMQ adapter has some unfortunate quotables, like:
To increase performance, Microsoft BizTalk® 2004 Adapter for MSMQ
is multi-threaded. If you have many receive locations, there may not be enough threads
available for all the receive locations. This prevents some of the receive locations
from picking up messages.
The reality is that you really shouldn’t have to starve any particular receive location
because of a lack of threads…you should just wind up with increased latency.
But, such is not the implementation of the MSMQ adapter (at least for BizTalk
2004.)
Some background: The MSMQ adapter has a “Batch Size” parameter and a “Serial
Processing” parameter that can be set per receive location. “Batch Size” determines
how many messages the adapter will attempt to read from the queue (and submit to the
message box) on each iteration. “Serial Processing” determines whether one thread
is engaged in the peek/get/submit activity per receive location (Serial Processing
= ‘true’) or multiple threads (Serial Processing = ‘false’). If “Serial
Processing” is true, the “Batch Size” is forced to one regardless of the actual setting.
So what is the execution flow for a given receive location? The internal class
MsmqReceiverEndpoint is instantiated per receive location, and when it initializes,
it calls ThreadPool.QueueUserWorkItem with
a reference to itself. If “Serial Processing” is false…it does this
exactly seven (7) times.
What does it do with the QueueUserWorkItem callback? Well, when MsmqReceiverEndpoint.ProcessWorkItem
is called, it enters into a do/while loop that doesn’t exit until the
endpoint (receive location) becomes invalid (i.e. the receive location is shut town.)
In other words, ProcessWorkItem sits on a .NET thread pool thread – and if Serial
Processing is false, it sits on seven of them. The do/while loop executes
a peek on the queue (with a hard-coded 10 second timeout), and if there are messages
waiting, it receives up to “Batch Size” and submits them to the message box. (It
will give up attempting to receive a “Batch Size” worth of messages if the 10 second
timeout is reached on any attempt within the batch receive loop – i.e. if you drop
a single message on a queue, and the batch size is greater than one, expect to wait
10 seconds before further activity begins…) The behavior of consuming seven
threads per queue leads to the recommendation of MinWorkerThreads = 7x MSMQ receive
locations provided above.
Now, I confess – I’m not a BizTalk adapter expert. But, this design
seems to be in conflict with the advice offered in “Writing
Effective BizTalk Server Adapters“, where it says:
Don’t starve the .NET thread pool: …While starving
the .NET thread pool is a risk to all asynchronous programming in .NET, it is particularly
important for the BizTalk Server adapter programmer to watch out for this. It
has impacted many BizTalk Server adapters: take great care not to starve the .NET
thread pool. The .NET thread pool is a limited but widely shared resource. It
is very easy to write code that uses one of its threads and holds onto it for ages
and in so doing blocks other work items from ever being executed….If you have multiple
pieces of work to do (for example copying messages out of MQSeries into BizTalk Server),
you should execute one work item (one batch of messages into BizTalk Server) and simply
requeue in the thread pool if there is more work to do. What ever you do, don’t
sit in a while loop on the thread.
Is this fixed in BizTalk 2006? Surely it is… And, in fact,
it sure seems to be in Beta 1. The design of the adapter is a bit different…First, “Serial
Processing” refers to whether additional messages will be received from
the queue prior to the “EndBatchComplete” event being set (downstream
of IBTDTCCommitConfirm.Done.) (This part of “Serial Processing” is
true for BizTalk 2004 as well, along with forcing the batch size to one.) “Serial
Processing” in BizTalk 2006 does not affect how many threads will be reading
from your queue – you will have just one (despite what the Beta 1 docs say…),
unless you have multiple host instances in play. (That one thread using
a large batch size and operating with serial processing set to ‘false’ – not
blocking on the actual message box submission – should keep up with a fairly
large message arrival rate, but multiple host instances might be needed for your particular
case.)
More importantly, the ProcessWorkItem implementation returns immediately after a single
peek/get/submit operation (and simply calls QueueUserWorkItem again, per the advice
cited above.) (Side note: There seems to be some room in the design for the
idea that you in fact woudn’t return immediately if more than a threshold number
of messages were received, but currently this condition is “if # of messages received
> BatchSize”, which won’t ever happen.)
So what should I do for now with BizTalk 2004? For those using the
MSMQ Adapter with BizTalk 2004…consider whether you can set “Serial Processing”
equal to true. Keep in mind this forces you to a batch size of 1, so this
might not work depending on your message arrival rate. If you test this configuration
and find an unacceptable performance loss, consider setting the MinWorkerThreads value
to 7x the number of MSMQ receive locations you are maintaining, and
MaxWorkerThreads to roughly 10x (to provide breathing room.)
As an alternative, spread your receive locations among multiple hosts (though avoid
an over-proliferation of hosts – that has its own issues.)
And never draw any conclusions until you have performance tested at load with your
final host configuration – that is, your final allocation of send handlers, receive
handlers, send ports, and receive locations among your hosts! Other adapters
may affect the outcome if they involve polling on the receive side, or polling on
the “response” side of a solit-response send port. (If they use a thread pool
thread to do their work, they can be affected by any adapter that consumes threads
whether they themselves are written correctly or not!) Finally, I’ve heard from
a gentlemen who has done extensive testing that the threading parameters above
are useful/necessary when using large numbers of MSMQT receive locations as well.
Never a dull day in BizTalk land…!
Jesus Rodriguez has created a super cool adapter for BizTalk Server 2006 that talks to SQL Service Broker in SQL Server 2005 so that you can use SSB as a transport for SQL to SQL communications within a broader BizTalk Server ecosystem. His adapter is available on gotdotnet and you can read more about it on his blog,
http://weblogs.asp.net/gsusx/archive/2005/09/26/425989.aspx
Thanks, Jesus!
The BizTalk 2006 version and API is 100% backwards compatible with the 2004 version and API. BizTalk 2006 is built against .NET2.0 while the BizTalk 2004 assemblies were built against .NET1.1. Although I’m not working for Microsoft this surely means that for the BizTalk 2006 version the Microsoft BizTalk product team has:
1. Checked the re-used BizTalk 2004 code parts against the new .NET version for incompatibilities and resolved all of them for the BizTalk 2006 release.
2. Rewritten parts of the original BizTalk 2004 implementation in order to benefit from the new framework version (such as a performance gain) without breaking compatibility with the previous BizTalk version.
3. Added a lot of new BizTalk functionality written from scratch in .NET2.0 code.
In general .NET framework supports installing and running different framework versions side-by-side on a single machine. By default your custom old .NET1.1 applications will run with .NET2.0 when you run them on a machine with 2.0 installed. If you detect incompatibilities you can instruct .NET to load the old runtime version you originally built your application against (by means of configuration). Or you can choose to fix the incompatibility in your code and recompile for .NET2.0.
Whole different story for your newly written .NET2.0 applications: they only run with 2.0 or higher compatible version (for example 2.1).
Microsoft BizTalk 2006 is a ‘new application’ and is ‘built against’ .NET 2.0 (beta) and as a consequence will only run with .NET framework 2.0 or higher compatible version. The BizTalk 2006 host instances will load and execute code using CLR .NET2.0 and as a consequence the assemblies it loads will execute within .NET 2.0. Your assemblies used inside orchestrations, your custom adapters and your pipeline components and their dependencies will all have to be compatible with .NET2.0. You don’t have the option to instruct to load the .NET framework version you built your assembly against because loading 2 different CLR versions in the same process is not supported. Here is a good post from Junfeng Zhang about the subject: The CLR Side by Side and Compatibility.
This means that as a BizTalk developer you should at least perform step 1 from the above steps for all your BizTalk 2004 custom code. If you’re lucky you detect no incompatibilities and it will work by default. But if you do detect incompatibilities then you’ll have to re-code and re-compile. For more information on the .NET CLR see Suzanne Cook’s weblog
What do you do if you have over 1000 suspended orchestrations, each one would have put a message to the event log when it suspended, however the event log entry does not tell you which instance of the orchestration generated the message, given they could all be the same orchestration called many of times that is failing.
How to know which one to look into, and recover from?
The SDK of BizTalk 2006 Beta2 will include a new version of the adapter base classes. These will not be backwards compatible with the original BizTalk 2004 SDK classes.
The API has changed and the internals have changed (uses System.Transactions for example). Adapters that have been originally built with the 2004 adapter base classes should continue to use and be deployed with this version.The .NET1.1 compiled assembly of the old base classes should by default work with.NET2.0 CLR.
You could of course prefer to re-compile for .NET2.0 but you will get some warnings during compilation time.
Think of this implementation as a BizTalk orchestration imitating the functionality of a Windows Service i.e. you have the ability to start, interrupt and stop the orchestration, otherwise like a service it keeps on running (when not processing BizTalk dehydrates it). The orchestration asynchronously spawns other orchestrations and waits for call back from them.
We used this implementation to aggregate results from multiple instances of (asynchronously spawned) orchestrations each running their own long running transactions which could last for a number of days. The input messages were split up into different types by Worker orchestrations (started asynchronously) each record being individually processed. The Worker orchestrations then called back the Aggregator Service orchestration which aggregated results into different message types before pushing the final result aggregated messages out at scheduled times during the day.
Other implementations of this pattern are out there but so far I’ve found none as simple and elegant as the one shown here.
• To keep things simple I have not shown any implementation of how to actually aggregate messages to learn this I suggest you google “BizTalk aggregator pattern”. There are a few implementations out there. I’m hounding Anil to blog his latest implementation of this pattern which apparently performs better than the ones currently blogged.
• Orphans & Zombies!!! If you terminate the Aggregator Service you may end up with orphaned orchestrations and zombie messages which it started asynchronously but have not called back to the service yet.
• Terminating the asynchronously spawned orchestrations will not hinder the running of the Aggregator Service but obviously you will loose the message data that that orchestration is processing.
• I wanted to have some fun so re-wrote the sample in BizTalk 2006 Beta1 but works fine in 2004.
• As you will see I have just provided a simple example of the pattern. I have not included the implementation of scopes, compensation and exception handling or aggregation of the messages.
• The example I have given here is for aggregating messages however there are many other uses.
So lets start with a few pictures and then I’ll explain how it all works.
Diagram 1: The AggregatorService orchestration
Diagram 2: The Worker orchestration
The AggregatorService orchestration looks complex at first so we’ll divide it into two parts to make the explanation easier. The first part is starting up the aggregator service and asynchronously spawning the worker orchestrations shown in the following diagram look familiar? It’s basically a sequential parallel convoy that kinda loops back on itself. I’ll go through in order shape by shape in the following table; I’ve left out the group shapes as these are basically comments.
Diagram 3: Start service and asynchronously spawn worker orchestration
Shape | Purpose |
Receive (Start Command) | Start up the service like a windows service you start only one instance of this orchestration up and leave it running continuously |
Send (Start command response) | Send back a response that the server has been started but the main purpose of this send is to initialize a correlation set which is used by the following two Receive shapes in the convoy |
Parallel actions left branch | The process you can see implement by the left hand side of the parallel action shape is to asynchronously spawn another orchestration or orchestrations to do some work with an incoming message (the input) |
Loop | This is an infinite loop which continues looping until the orchestration is terminated in HAT or a Stop command message is received on the 2nd branch of the Listen shape |
Listen | Listens for an input data message to do some processing on or a Stop command to terminate the orchestration “service“ |
Receive (Left listen branch) (Input data) | Receive a data input |
Start Orchestration (Left listen branch) | Asynchronously start the worker orchestration passing it a self correlated port and the message data input. The self correlated port acts like a delegate for the Worker orchestration to call the AggregatorService orchestration back on with a result which may be aggregated |
Receive (Right listen branch) (Stop command) | Receives a command to stop the orchestration |
Terminate (Right listen branch) | Terminate orchestration see Orphans and Zombies in notes above |
The second part of the AggregatorService orchestration is concerned with receiving call backs containing results to be aggregated from the Worker orchestration anytime in the future when the worker has finished processing could be seconds to weeks. Again I’ll go through the shapes one by one
Diagram 4: Receive asynchronous call backs and aggregate messages
Shape | Purpose |
Parallel actions right branch | The purpose of this branch is to receive the processed messages from the callbacks and aggregate those messages |
Construct message | Construct an empty message to aggregate into this is an initialization step |
Message Assignment | Using a DOM document load the message seed XML (usually this is the root node without any record data) and assign the DOM document to the aggregated message variable |
Loop | This is an infinite loop which continues looping until the orchestration is terminated in HAT or a Stop command message is received on the 2nd branch of the Listen shape |
Listen | We are either waiting for a call back message from the Worker orchestration or if we haven’t received anything for 1 minute push out the aggregated message and reinitialize it.
Yes I know the logic isn’t full proof here but it’s just for the sake of example. It’s easy to implement better scheduling logic i.e. put a test to see if the scheduled time has passed in the left branch of the listen shape and then send and reinitialize the aggregated message. |
Receive (Left listen branch) (Call back) | Receive the processed data via a callback from the self correlated port on the Worker orchestration |
Construct message (Left listen branch) | Construct a temp message and aggregate the current aggregated message with the processed data we have just received |
Transform message (Left listen branch) | Map using inline xslt to aggregate the two messages |
Construct message (Left listen branch) | Overwrite the current aggregate message with the temp message |
Transform message (Left listen branch) | Simple message assignment |
Delay (Right listen branch) | wait for 1 minute |
Decide (Left branch, right listen branch) | Test to see if there are any records in the aggregated message no point sending out an empty message |
Send (Left decide branch, right listen branch) | Send out the aggregated message |
Construct message (Left decide branch, right listen branch) | Construct an empty message to aggregate into this is an initialization step |
Message Assignment (Left decide branch, right listen branch) | Using a DOM document load the message seed XML (usually this is the root node without any record data) and assign the DOM document to the aggregated message variable |
Just briefly all my Worker orchestration does is simulate some processing of the input data and wait for 10 seconds before calling back on the self correlated port which is one of its arguments.
R. Addis
Firstly I haven’t blogged for a while as I’ve been hard at work down in Wellington. Two of us implemented a credit card issuing and re-issuing integration app (5 partners) in just under six weeks. We were very impressed with the speed of developing in BizTalk which enabled the achievement of this deadline, finally a real RAD tool. So having a wee breather before I’m off to Redmond to Microsoft’s BPI & Workflow Conference with some other integration folks from Synergy I’m really looking forward to this. The three technical tracks on show are: BizTalk 2006, Windows Workflow Foundation and Windows Communications Foundation (Indigo) The Business Track on Realising business value from BPI looks very promising. So I’ll try to post any intriguing insights from this, the best part of which will be a consolidation of my understanding in Microsoft’s future BPI direction.