by stephen-w-thomas | Aug 17, 2004 | Stephen's BizTalk and Integration Blog
Updated 8/8/2005:
Please note the information below is outdated and I would never recommend mapping messages of this size inside Biztalk 2004. Biztalk 2006 will have support for larger messages by breaking up the message into chunks. More information on large message support can be found on the Performance Blog.
Original Post:
Ok, I hear it all the time: “Can BizTalk handle large files?”
But, really the question should be: “How large of a file do you want BizTalk to process?”
Large is a subjective term. For the purposes of this post, large is a 150 MB XML Document. I see lots of blogs with negative posts on big messages in BizTalk 2004. My experiences have been quite the opposite. But, I’ll let you make the determination yourself. Here are my test results.
My Test Computer: P4 2.4 GHz with 1.25 GB Ram.
CRITICAL: Working with large files such as this will consume a HUGE amount of system resources. My CPU was maxed at 100% and RAM utilization was extensive. A better approach is to break the large file down, but that is a subject for another post coming soon…
Sample File Size: 180 MB
Schema Nodes: Around 400
All Mapping on Send Port, map size is around 1.2 MB
Test1: Straight line mapping to itself
Run Time: 7 minutes 13 seconds
Test2: Straight line mapping to itself
Functoids: Around 80 String Manipulation
Run Time: 8 minutes 26 seconds
Test3: Straight line mapping to itself
Functoids (over 250):
– Around 80 String Manipulation
– Around 50 Logical
– Around 50 Value Mapping
– Around 80 Scripting with calls to an external assembly
Run Time: 9 minutes 42 seconds
Take Away: I was surprised and impressed with the processing time of large files inside BizTalk. But, I’ll let you make the final determination…
by stephen-w-thomas | Aug 10, 2004 | Stephen's BizTalk and Integration Blog
Scott posted a message back in June about untyped receive ports inside on Orchestration. Untyped Messages means you receive or send a message as System.Xml.XmlDocument rather then a specific schema type. This is an incredibly powerful feature in BizTalk 2004!
Business Case:
Lets say you want to have one Orchestration (i.e. a single business process) to process many different types of messages all in the same way. Additionally, say you want to change some values in your message and send all of your messages out through the same Send Shape (i.e. send messages as Xml.XmlDocument).
Lets say you can have a Book Review and a Movie Review. You want them both to be processed in the same way though the same Orchestration, maybe send information to an outside web service or something like that. Additionally, you have to extract out promoted properties of the message inside the Orchestration and make decisions inside the Orchestration on them. This can not be done with Xml.XmlDocument since the Orchestration will not allow you to access them (they are in the message context, just you can not get at them).
Confused? How about we look at a sample.
DOWNLOAD: Get the sample here!
Set-up is easy, just unzip the SampleProperties folder and put it on your C: drive. Then, build and deploy the SampleProperties project. I use early binding so the send and receive ports will be created for you.
To run the sample, drop the 4 Start….xml message into c:\SampleProperties\In. You will get 4 messages in your Out folder. Plus, 4 events will be written to your event log. Do not forget to look inside the expressions shapes inside the Orchestration for comments. If all else fails, read the ReadMe.txt file.
Key Take Home Points:
– Common properties must be promoted in all of the different schemas (look at the promotions in both BookReview.xsd and MovieReview.xsd)
– Messages for type XmlDocument can be cast into different types
– Typed messages can be cast back into XmlDocument
– Blank Schema can be created and properties changed (look at Movie Review branch)
– This is using Property Promotion and Demotion in the XML Pipelines
– As a test, set the Send Pipeline to Pass Through and see the difference in the Movie Review output data.
– This will also work for distinguished fields
CRITICAL: This process is kind of risky since you can pass in any XML Document. You can end up with an invalid cast exception or XPath query not valid.
What is this Movie Review Branch inside the Orchestration really doing?
- Taking in a message of type Xml.XmlDocument
- Casts the In message to CastMovieIn (strongly typed to a schema)
- Sets Orchestration variables based on promoted properties inside the MovieReview schema
- Creates a new XML Document (CastMovieOut) and loads a blank schema for that type
- Changes values inside CastMovieOut
- Creates a new message, Out, as XmlDocument
- Casts the CastMoveOut to Out
Take Away: Using XmlDocument can greatly increase the flexibility inside your Orchestration.
by stephen-w-thomas | Aug 10, 2004 | Downloads
Untyped messages inside an Orchestration allow for many different types of messages to be received by the same Orchestration using System.Xml.XmlDocument. This sample shows basic use of this feature.
Key Take Home Points:
– Common properties must be promoted in all of the different schemas (look at the promotions in both BookReview.xsd and MovieReview.xsd)
– Messages for type XmlDocument can be cast into different types
– Typed messages can be cast back into XmlDocument
– Blank Schema can be created and properties changed (look at Movie Review branch)
– This is using Property Promotion and Demotion in the XML Pipelines
– As a test, set the Send Pipeline to Pass Through and see the difference in the Movie Review output data.
– This will also work for distinguished fields
Get more information from the original blog post on this topic: http://www.biztalkgurus.com/biztalk_server/biztalk_blogs/b/biztalk/archive/2004/08/10/working-with-untyped-send-and-receive-ports-in-biztalk-orchestrations.aspx
by stephen-w-thomas | Aug 2, 2004 | Stephen's BizTalk and Integration Blog
Did anyone read Kevin Smith’s blog on ACK/NACK and run out to try it? Well, I did and found it a more time consuming then I expected. I have put together a sample that shows how to catch the SOAP exception and get access to the error message. I hope that after looking at this sample Kevin’s excellent post on NACK’s will make a little more sense. It did for me.
DOWNLOAD: Get the sample here!
Set-up is easy, just unzip the SampleNACK folder and put it on your C: drive. Then, build and deploy the SampleNACK project. You will need to manually create a Send Port. I set up a File Send Port going to c:\some_location_that_does_not_exist. Set retries to 0. To run it, drop the StartFile.xml message into c:\SampleNACK\In. Your NACK will show up in c:\SampleNACK\Out. Do not forget to look inside the expressions shapes inside the Orchestration for comments. If all else fails, read the ReadMe.txt file.
CRITICAL: Getting at the HTTP error using Delivery Notification will not work for transport type of HTTP or SOAP due to an “issue”. For more information please see Microsoft KB840008.
Key Take Home Points:
– Delivery Notification is not available on Early Bound Ports.
– Must import System.Web.Services to cast the SOAP exception
– Set Send Port retries to 0
– Must use a Synchronized scope
Under the covers:
Ok, so you ask what is BizTalk doing inside the little Delivery Notification property? Keep in mind this is my interpretation.
When a message is sent through a send port with Delivery Notification set to transmitted, a correlation set is initialized. A correlation token is assigned to the outbound message. A subscription is started based on this correlation token. This token is stored in the context property of the ACK/NACK and promoted. When the ACK/NACK is returned, it is routed back to the calling Orchestration. You get all this just by setting a little property to “Transmitted”! Cool!
Take Away: Delivery Notification is an easy way to catch exceptions as long as you understand how to get at the error message.
by stephen-w-thomas | Jul 30, 2004 | Stephen's BizTalk and Integration Blog
This is a challenge to overcome in BizTalk 2004. It seems that the message never hits the Send Pipeline and as far as I can tell no error is written to HAT. The only place this error is written to is the Event Log. But this is ok if you are using MOM or something else to watch the event log.
The messages will be Suspended inside HAT and marked as resumable.
CRITICAL: Note that this is not the case if a map fails on the Receive Port. These messages are non-resumable.
by stephen-w-thomas | Jul 29, 2004 | Stephen's BizTalk and Integration Blog
Microsoft released a new version of the product documentation on July 28th.It is available for download at: http://go.microsoft.com/fwlink/?linkid=20616
by stephen-w-thomas | Jul 28, 2004 | Stephen's BizTalk and Integration Blog
I just wanted to take a second to get everyone on the same page since these properties are VERY important in BizTalk.
If you are a solo developer or you have never renamed a root node, then you may not understand the importance of these properties.First off, what are they?
Node Name: This is the name that is displayed inside Visual Studio for the root node in the schema and for each field element in a property schema.
Root Node Type Name: Basically, this is what BizTalk will call your root node. This does not have to be the same as the Node Name, thus the potential for confusion.
The root node will default to the same name as your root node name when you first create the node or when you rename the root node by changing the Node Name property. But, it is possible to change the Root Node Type Name, either by accident or on purpose, to something else that is non-meaningful to anyone.
Ok, but how will this really effect me? BizTalk sets a message context property called BTS.MessageType. This is a concatenation of the document namespace and Root Node Name. This message type property will show up in HAT and can also cause problems when working with non-unique root nodes with no namespace, but that is a whole other post.
What purpose does this serve? This will allow you to have two schemas with no namespace or the same namespace and the same root node name, but different root node type name. The benefits of this may not be obvious, but it will allow you to set the message type property inside the pipeline. This is required if you want to use the message for routing, mapping, or inside a strongly typed orchestration.
If you do have two messages with no or the same namespace and the same root node type name you may get the follow error at runtime:
There was a failure executing the receive pipeline: "Microsoft.BizTalk.DefaultPipelines.XMLReceive" Source: "XML disassembler" Receive Location: "d:\bt\in\*.xml" Reason: The disassembler cannot retrieve the document specification by using this type: "Root123". Either the schema is not deployed correctly, or more than one schema is deployed for the same message type.
The same holds true for schemas name and type name on the file properties of a schema. The schema name has a Type Name property that is used inside the BizTalk Type Picker to reference that schema. If this name gets changed on accident, it may become difficult for other developers to reference your schema.
CRITICAL: Unlike the Root Node, the file Type Name property will NOT change if the schema is renamed.
Take Away: Be mindful of the Type Name properties on the file and root node level and double check them to make sure they are set correctly.
by stephen-w-thomas | Jul 28, 2004 | Stephen's BizTalk and Integration Blog
Greeting and welcome to my BizTalk Blog.
I guess you might want to know a little about who is going to be writing this blog. Before working with BizTalk Server, I ran my own retail internet business selling toys and display cases on the net. Ok, that does not sound very exciting or technical but I did all the web site development along with all the order packing, phone answering, receiving, etc. Oh, I guess I should back up a little more and should mention that before that I was working for International Paper in a paper mill as a Process Engineer. I went to the University of Iowa and have a Chemical Engineering degree. And, No, we do not grow potatoes in Iowa.
I have been working with BizTalk Server since February 27th 2001. I remember that date well… since that was my first day in Seattle on a new project and the next day we were shaken up by the 6.8 earthquake. Who knew Seattle was on a fault line?I have spent over 2 years working with BizTalk 2000 and 2002. I have been working exclusively with BizTalk 2004 since I got my hands on the Beta in early July 2003. I started on first BizTalk 2004 project in August 2003 working (or trying to work with) the Beta.
Now, after RTM, I am working on my 3rd BizTalk project.What do I hope to achieve with my Blog? Good question. I hope to post simple solutions to common, complex problems. Like working with LARGE files (1GB+), using XPath and C# inside the mapper, custom pipelines, orchestration throttling, batching, de-batching, and the list goes on and on.
My approach is always consistent: Make it as simple as possible with little custom code!
