by stephen-w-thomas | Sep 9, 2005 | Stephen's BizTalk and Integration Blog
It’s late on a Friday. I am still at the client site. I just canceled my flight home. It is my anniversary weekend (yep it is on 9/11, sure sucks). We are hours from going live… But, what does Exceptions.Types.InterceptorException GetTempFileName Failed mean?
I ran into this problem while testing some recently updated Orchestrations that use Convoys to process messages received from multiple source systems in order. A small change was made that removed a Send Shape and replaced it with a custom .net component call to a java web service. Seems simple enough.
When the process encounters an exception, it uses the file adapter to write the file to disk and stops message process. That all tested perfectly.
When the process completes successfully but no other messages have arrived in the convoy. That all tested perfectly.
When the process completes successfully but new messages have arrived in the convoy the follow error occurs:
Exceptions.Types.InterceptorException
GetTempFileName Failed.
Exception type: StreamingException
Source: Microsoft.BizTalk.Streaming
Target Site: System.IO.Stream CreatePersistentStream()
Help Link:
Additional error information:
Not only do I get an exception, but the process continues forever in an endless loop with a 100% CPU spike.
So, what could be the problem and more importantly how can it be fixed?
The answer lies with the change that was made by replacing the Send Shape with a .net component call. This removed a persistence point in the Orchestation. In fact, it removed the only persistence point for the happy path (i.e. no errors) inside the Orchestration. So, when this message is processing, a 2nd message arrives in the convoy, and the next Receive Shape is hit the whole process goes down hill.
Solution: I added an Atomic Scope (persistence point) right after the Receive Shape. Problem solved.
A better solution: Isolate the units of work inside the Orchestration into logical units and use Atomic scopes to track progress through the Orchestation. In the event of a Server Crash, the Orchestrations would restart in more desirable state.
Even better solutions? I’m sure.
by stephen-w-thomas | Jun 12, 2005 | Downloads
This lab will show you step-by-step how to use the Flat File Disassembler, Delivery Notification, and a Sequential Convoy to map a large flat file document.
Get more information from the original blog post on this topic: http://www.biztalkgurus.com/biztalk_server/biztalk_blogs/b/biztalk/archive/2005/06/12/sequential-flat-file-message-processing-lab.aspx
by stephen-w-thomas | Apr 22, 2005 | Stephen's BizTalk and Integration Blog
The Convoy Deep Dive white paper and sample code is now available on MSDN. You can read the paper on-line and download the sample code here.
For anyone who read the “beta” version of the paper, Scenario 2 went through a minor change. I now use an atomic scope to build the output message rather then a simple string concatenation. This eliminates messages collecting inside the message box in the “Delivered, not consumed” status.
It seems that the status of a message is not updated until a persistence point is reached or the Orchestration dehydrates. I would guess this is the case with any messages that are used inside an Orchestration and not just inside a looping-receive Convoy. But, I have not verified this.
Overall, this change provides a better solution since the Orchestration state is being saved when each new message arrives and can be easily be restated/recovered in the event of a system failure.
by stephen-w-thomas | Apr 1, 2005 | Downloads
Microsoft BizTalk Server 2004 supports the use of convoy message processing, a new messaging pattern. Convoys are a messaging pattern and not a feature of BizTalk Server 2004. A convoy is not something a customer can use, like the Rules Engine, but instead it is something that the business process dictates and requires.
This paper discusses examples of business scenarios that require convoy message processing. After establishing a business need through the scenarios, this paper explains basic convoy theory and the different types of convoy messaging patterns. This paper discusses a solution to each business scenario and reviews each included sample.
View this whitepaper online at: http://msdn2.microsoft.com/en-us/library/ms942189.aspx
by stephen-w-thomas | Feb 10, 2005 | Stephen's BizTalk and Integration Blog
I have talked a lot in the past about Convoys. But, under the covers what really makes Convoy message processing different? What makes a Convoy a Convoy?
To sum it all up into one sentence: Convoys require database level message-to-Orchestration correlation independent of any running Orchestration instances.
Is this confusing? Yes. But trust me, more detailed information is coming soon on this topic. For now, I want to briefly go over the basic internals of how Convoys work in BizTalk Server 2004.
When you deploy an Orchestration:
The presence of a Convoy is detected at the time a BizTalk 2004 Orchestration is enlisted. At this point, an entry is inserted into the ConvoySets table inside the Message Box that lists the properties used in the Convoy Set. This is limited to only three properties because each property is listed in a column inside the table. The subscriptions that can activate this Convoy are marked with the uidConvoySetID for that Convoy Set.
When a message arrives:
Every time a new message arrives into BizTalk it is checked to see if it is part of a Convoy or if it should start a Convoy. If it should start a Convoy, an entry is inserted into the ConvoySetInstances table that routed messages to the correct instance. If it is part of an existing Convoy (i.e. it has the sample Convoy properties of an existing Convoy set), it is routed to the Service Instance listed inside the table. Otherwise, it just follows the normal subscription process.
Convoy Virtual Subscription:
This database level message routing creates a type of virtual subscription on top of the standard subscriptions inside the message box. This is set up internally using a series of tables.
I have put together a simple SQL query to view all Orchestrations that belong to a Convoy and what the Convoy virtual subscription properties are for those instances. I have not tested this code under all Convoy scenarios so please use caution when running the query and it may yield unexpected results. Note that for this viewer to return any data you must have running Orchestration instances that are part of a Convoy.
DOWNLOAD: Sample Convoy Virtual Subscription Viewer
CRITICAL: This application and included SQL will query critical BizTalk SQL Tables. This should not be run on production servers. Review all code prior to execution and use extreme caution when viewing any BizTalk related database tables. Use this code at your own risk.
This sample is also an excellent example of why I am a BizTalk Developer rather then a .net Developer :).
Here are some of my samples that use Convoys that can be used with this viewer.
DOWNLOAD: Concurrent Receive Convoys (post date 11/5/2004)
DOWNLOAD: Sequential Receive Convoys (post date 8/23/2004)
by stephen-w-thomas | Feb 10, 2005 | Downloads
This tool can be used to see the Convoy properties associated with any running Orchestration processing messages as a Convoy. Note that this queries critical BizTalk tables. Run with care. This tool works with BizTalk 2004 and BizTalk 2006.
Get more information from the original blog post on this topic: http://www.biztalkgurus.com/blogs/biztalk/archive/2005/02/11/Convoy-Subscription-Viewer-for-BizTalk-Server-2004.aspx
by stephen-w-thomas | Nov 7, 2004 | Stephen's BizTalk and Integration Blog
In the past, I have tended to focus on sequential convoys. Now, I decided to take a look at Concurrent Convoys. First off, I was sure in older versions of the documentation they were called Parallel Convoys? Maybe I’m mistaken or maybe the name changed over the past few months…. In any case, I will get with the times and call them Concurrent Convoys. No matter what you call them, the power and usefulness of this type of convoy is the same.
Why use Concurrent Convoys? Concurrent convoys allow you to receive multiple items in any order that all need to be processed by the same Orchestration before the next step in the process can happen.
Setting up and using a Concurrent Convoy is rather straight forward. All you have to do is drop a Parallel Shape inside your Orchestration. Then, drop in your multiple Receive Shapes. If this is the first shape in your Orchestration then each Received must be set with Activate = True and all must initialize the same Correlation Set. Otherwise, they must all follow the same Correlation Set that was initialized by a Send Shape. That is all your need to do!
I have included a sample with two different Orchestrations. One Orchestration shows a Concurrent Convoy with Active Receives and one without Active Receives.
The Orchestration with Activate Receives has two sets of sample files. The set with ID 100 will produce 8 total records in the output file. The set with ID 500 produces 6 records. Note that since all three messages arrive on different ports, Receive Port mapping is possible to get all inbound documents into a common format (not shown in my sample).
The Orchestration without Activate Receives uses the same set of sample files but it requires a Start Message to be received and a message to be sent out to initialize the correlation. This will produce the same output of 8 and 6 records. In this Orchestration, all the Receives uses the same Receive Port.
For information on how to run the samples, please see the Read Me file.
Make sure you keep an eye on the BTSSubscriptionViewer tool (located in the SDK). Note the differences between the two Orchestrations.
DOWLOAD: Sample Concurrent Receive Convoys
Bonus: I also show a simple say to merge all three messages together into a single output. This uses a map inside the Orchestration.
by stephen-w-thomas | Oct 5, 2004 | Stephen's BizTalk and Integration Blog
Here is another sample of a Sequential Convoy in BizTalk 2004. This is a little different then most, it contains a Parallel Action shape that allows for concurrent parallel processing of inbound messages.
What does this accomplish? It is all about control. This process allows for processing a pre-defined number of messages at the same time in a controlled manner.
Performance? Ok, it is not the fasting running Orchestration I have ever seen. Actually, in some of my tests this parallel action processed 1/2 to 1/3 less messages per minutes then a single process convoy. In case you are wondering, running 100 messages has 203 persistence points.
What’s the point then? Well, I could see a need for something like this in a de-batching scenario when the parallel actions will take an unknown account of time to complete and the overall status of all the messages is important.
Download the sample below. I have 4 parallel actions processing messages. I added a random delay to simulate message submission to an outside system or web service. I have a Win Form test harness for easy running. If you want to test the performance, download my single convoy example and remove the delay shapes inside this sample.
The key to this solution is the Synchronized Scope shapes that maintain the overall message count inside the Orchestration.
DOWNLOAD: Parallel Sequential Convoy Sample
See the read me file for set up instructions. Make sure you run “Clean Up” before you rerun and samples.
A better approach? I think it would be better to remove the processing logic inside the Parallel Actions and call an additional Orchestration to do any work. This would increase the overall flexibility and maintenance of the solution.
Take Away: Parallel Convoys allow for better control of your de-batching process but the overall throughput of your messages is decreased.
by stephen-w-thomas | Aug 22, 2004 | Stephen's BizTalk and Integration Blog
I need to start off with a note about the Parallel Sequential Convoy sample that I am working on. I am still working on it and it will probably be a few weeks before it is ready to be posted. I have a sample working that does Parallel Sequential processing of a single message type. I am now trying to get that to work with multiple message types in a single Orchestration, although I think this breaks one of the rules of convoys. In any case, please check back later for that sample.
In the mean time, here is a sample working with a Sequential Receive Convoy inside an Orchestration. This takes in an initial message to start the Orchestration. This message will tell the Orchestration how many messages to process before it ends. Rather then this approach, I could have used a time out or a set number of messages to tell the process when to end.
This sample includes a Test Harness Win Form to run the process. For set-up instructions, please see the ReatMe.txt. No executable code is included with the samples; you will need to compile the code yourself.
Download: Get the sample code here.
Key Take Home Points:
– This approach could be prone to zombies
– The Receive Port must be the same for the Start and Worker messages
– The Receive Port must be marked for ordered delivery
What’s coming next?
– A parallel sequential convoy sample
– A zombie maker “feature”
Take Away: Running the sample is kind of fun and working with convoys is cool.
