Presentation -> Using BizTalk 2006

by community-syndication | Feb 8, 2006 | BizTalk Community Blogs via Syndication

Short notice, but I am doing a presentation tomorrow (Thurs. Feb 9) at the Metro Toronto .Net Users Group.
You can read more about it here:
http://www.metrotorontoug.com/User+Group+Events/231.aspx

Demos will include dealing with binary documents (Word and Excel) in BizTalk 2006.
The demos will also include:

– The new Sharepoint and POP3 adapters for BizTalk 2006.
– Parsing Excel documents to XML in BizTalk.
– Integration of Sql Server 2005 Service Broker with BizTalk 2006.
– Some of the new and improved features of BizTalk 2006.

So if you are in the area, feel free to drop by.

XSLT performance when mapping large documents in BizTalk

by community-syndication | Feb 8, 2006 | BizTalk Community Blogs via Syndication

Recently I had to map a document with many thousand rows. I could not split the document because before I could split it, the document’s nodes had to be sorted.

With such large files you generally test it using a small subset to avoid waiting for maps to complete, I built an XSLT which worked great, I thought.

When you use a select filter such as “not(KeyValue=preceding-sibling::row/ KeyValue)” you end up with a huge performance hit the larger the document gets. My map went from 2 seconds for 50 rows to 10 minutes for a few thousand.

How to improve performance when you have large XML files to map that you can’t split? Try using xsl:key instead, which builds an index of keys from which you can much more efficiently select.

Here is a sample XSLT that demonstrates how to use the xsl:key:

Using DelayedFileSystemWatcher sample

by community-syndication | Feb 6, 2006 | BizTalk Community Blogs via Syndication

See attached file for a sample on how to use DelayedFileSystemWatcher class described in my previous post FileSystemWatcher generates duplicate events – How to workaround

Source Links property in BizTalk Maps

by community-syndication | Feb 6, 2006 | BizTalk Community Blogs via Syndication

Here’s a cool mapping tip I found on Vijendra’s blog.

While mapping an element from the source schema to the target schema, it is possible to copy either the value contained within the element or the element name itself.

You achieve this by setting the Source Links property of the link connecting the elements as shown below.

In the above figure above the value mapped to the target element will be “CustomerName”.

The values that can be set are:
1. Copy text value – which is the default that copies the content of the element
2. Copy name – copies the name of the source node instead of its value
3. Copy text and subcontent value – concatenates the value of all child nodes

Now you might be wondering what is this 3rd value (Copy text and subcontent value) that can be set…

This is used if you want to concatenate all the child element values of the source node into a single element in the target as shown below.

Here is an example:

Input Message
<ns0:Root xmlns:ns0=”http://MapTest.SourceSchema“>
  <Record>
    <FirstName>FirstName_0</FirstName>
    <LastName>LastName_0</LastName>
    <Street>Street_0</Street>
    <City>City_0</City>
  </Record>
  <Record>
    <FirstName>FirstName_1</FirstName>
    <LastName>LastName_1</LastName>
    <Street>Street_1</Street>
    <City>City_1</City>
  </Record>
</ns0:Root>

Output Message
<ns0:Root xmlns:ns0=”http://MapTest.DestinationSchema“>
  <Record>
    <Data>FirstName_0LastName_0Street_0City_0</Data>
  </Record>
  <Record>
    <Data>FirstName_1LastName_1Street_1City_1</Data>
  </Record>
</ns0:Root>

Now here is another interesting mapping

The mapping above transforms the elements in the source schema into a key-value pair as shown below.

Each element node from the source schema links to the two elements in the target schema, one link copies the node name and the other copies the node value.

Input Message
<ns0:Root xmlns:ns0=”http://MapTest.SourceSchema“>
  <Record>
    <FirstName>FirstName_0</FirstName>
    <LastName>LastName_0</LastName>
    <Street>Street_0</Street>
    <City>City_0</City>
  </Record>
</ns0:Root>

Output Message
<ns0:Root xmlns:ns0=”http://MapTest.KeyValuePair“>
  <Record>
    <FieldName>FirstName</FieldName>
    <FieldValue>FirstName_0</FieldValue>
  </Record>
  <Record>
    <FieldName>LastName</FieldName>
    <FieldValue>LastName_0</FieldValue>
  </Record>
  <Record>
    <FieldName>Street</FieldName>
    <FieldValue>Street_0</FieldValue>
  </Record>
  <Record>
    <FieldName>City</FieldName>
    <FieldValue>City_0</FieldValue>
  </Record>
</ns0:Root>

Jobs

by community-syndication | Feb 6, 2006 | BizTalk Community Blogs via Syndication

(Shameless plug)

Looking to work in a high quality position for a high quality consultancy? Look to the left where the latest positions at Conchango are available.

Oh, and if you do happen to get a job here don’t forget to say where you found out about the position 🙂

I don’t think I need to say any more…

My article (Introduction to BizTalk Server 2004) on AspToday

by community-syndication | Feb 5, 2006 | BizTalk Community Blogs via Syndication

Check out my article on AspToday

Introduction to BizTalk Server 2004

Debatching a flat file in BizTalk 2004/2006

by community-syndication | Feb 5, 2006 | BizTalk Community Blogs via Syndication

Sometimes you receive a batch file containing multiple records and you have to split each record into a separate message and process them individually.

This kind of splitting of a batch file is also known as debatching.

Depending on your requirement, you can either debatch an incoming message inside the pipeline or inside the orchestration.

The technique used to debatch a message inside the pipeline depends on whether the input is an XML file or a flat file.

For an XML file, you will need to use an XML disassembler, an envelop schema and a document schema, which I will discuss some other time on separate blog post / article.

Here I will show you how you can debatch a flat file.

Let us assume that we need to debatch a flat file that contains the following structure.

Field1,Field2,Field3,Field4,Field5,Field6
Field1,Field2,Field3,Field4,Field5,Field6
Field1,Field2,Field3,Field4,Field5,Field6
Field1,Field2,Field3,Field4,Field5,Field6
Field1,Field2,Field3,Field4,Field5,Field6
Field1,Field2,Field3,Field4,Field5,Field6

First you need to create a flat file schema that looks similar to the one shown in the figure below.

Make sure you have specified the correct child delimiters for the Root (0x0D 0x0A) node as well as the Record (,) node.

Now select the Record node and set its MaxOccurs property to 1. This is actually the key to debatching. The flat file disassembler will automatically separate each record into individual message.

Next, as you would normally do, add a receive pipeline to the project and add a flat file disassembler and set the Document schema property to the schema we created.

That’s all there is to it.

Now suppose you have a flat file that also has a header record or a trailer (footer) record or both as shown below.

FirstName,LastName,RegNo,School,City
SubjectName1,Score1
SubjectName2,Score2
SubjectName3,Score3
SubjectName4,Score4
SubjectName5,Score5
SubjectName6,Score6
SubjectName7,Score7
SubjectName8,Score8
TotalScore

You would normally create a flat file schema as shown below to parse as one single message.

But for the purpose of debatching the body records, you need to create 3 separate schemas as shown below. Make sure the MaxOccurs of the Body record is set to 1.

And in the flat file disassembler, you need to set the Header schema, Document schema and Trailer schema properties appropriately.

Note: You might need to set a unique Tag Identifier property for each of the schemas, and also prefix the same tag identifier on the records as shown below, so that the flat file parser can distinguish the header, body and footer records.

HFirstName,LastName,RegNo,School,City
BSubjectName1,Score1
BSubjectName2,Score2
BSubjectName3,Score3
BSubjectName4,Score4
BSubjectName5,Score5
BSubjectName6,Score6
BSubjectName7,Score7
BSubjectName8,Score8
FTotalScore

It is possible to preserve the header record into the message context, by setting the Preserve header property of the flat file disassembler to true, so that you can use it as a header record for each of the debatched message.

To use the preserved header, you need to set the Header schema property of the flat file assembler to the same header schema you used in the disassembler. Here’s how the output will look.

HFirstName,LastName,RegNo,School,City
BSubjectName1,Score1

HFirstName,LastName,RegNo,School,City
BSubjectName2,Score2

HFirstName,LastName,RegNo,School,City
BSubjectName3,Score3

HFirstName,LastName,RegNo,School,City
BSubjectName4,Score4

HFirstName,LastName,RegNo,School,City
BSubjectName5,Score5

HFirstName,LastName,RegNo,School,City
BSubjectName6,Score6

HFirstName,LastName,RegNo,School,City
BSubjectName7,Score7

HFirstName,LastName,RegNo,School,City
BSubjectName8,Score8

FileSystemWatcher generates duplicate events – How to workaround

by community-syndication | Feb 4, 2006 | BizTalk Community Blogs via Syndication

I had to use the FileSystemWatcher class these days and I found out that it raises multiple duplicate events. If you are not familiar with FileSystemWatcher class, this class allows you to watch a particular folder and receive notifications/events every time there is change happening in that folder. You can subcribe to all sorts of events: file created, file changed, file renamed, file accessed, directory created, directory renamed, directory accessed, etc.

The problem is that multiple duplicate events are being raised. For instance, if I copy a file over an existing file, instead of raising one file changed event, the FileSystemWatcher object will raise 5 !!! such file change events that are identical. If you are creating a file, the FileSystemWatcher class will raise a file created event and at least a file changed event.

I think that some of these issues are caused by the Win32 APIs and they are normal, others (like the file createad and file changed events when creating a file) are probably caused by applications. Many applications will first create an empty file and then they will update the file with the file content hence the file created, file changed events being raised when creating a new document.

This can be worked around by queueing the events (let’s say over a 1 second period) and then remove any duplicates. And this is what I did. I implemented my own version of FileSystemWatcher class (named DelayedFileSystemWatcher) that mimics MOST of the FileSystemWatcher behavior and it removes duplicate events.

DelayedFileSystemWatcher class will delay processing of any events that are being raised. All events that are raised (except error events) by the inner FileSystemWatcher object will be queued to a collection. The ElapsedEventHandler method will be invoked automatically (through a System.Timer.Timer event) every 1 second. This method will analyze all events in the queue. The first time an event is found, it will be marked as “Delayed” and it won’t be processed. The next time the method is executed, the already “Delayed” events will be raised but first all events that are duplicate of “Delayed” events will be removed. An event in the queue will either be delayed (processing postponed for next timer event), raised (it’s already delayed and it’s the first event of his kind), or deleted (delayed or not but there was an event in front of it that was similar/duplicate).So this class mimics FileSystemWatcher but it removes duplicate events that show up within 2 timer events. A second event is duplicate of a first event if both events are of the same type and have all properties equal or if the second event is a an update/changed event, first event is a create event and both events refer to the same file.

See code file attached. DISCLAIMER: Use this code at your own risk. No support is provided and this code has NOT been tested.

[Last file update – February 09, 2006]

Filtering records using Maps and conditional looping

by community-syndication | Feb 4, 2006 | BizTalk Community Blogs via Syndication

Quite often you come across situations in BizTalk, where you receive a message that contains hundreds of records, when you are actually interested only in a subset of those records that match a certain criteria and want to create a message that contains only the relevant records.

There are numerous ways of doing it such as:

• Debatch and Aggregate (Scatter-Gather Pattern)
  
• Using xslt in maps
  
• Using xpath in orchestration
  
• Using an external component

There is also another simple and straightforward way of doing it using BizTalk maps known as conditional looping, which you will see in a moment.

Let me illustrate this using an example. Assume that you receive a Customer message that contains all your customers. However, your requirement is a Customer message that contains only those customers who match the following criteria:

City = ‘New York’
Rating = ‘High’

A simple map shown below will do the trick.

When the outputs of a Looping functoid in conjunction with any of the Logical functoids are connected to the same target node, the behavior is as follows:

• If the output of the logical functoid is false, the looping functoid will suppress the creation of the target node.
  
• If the output of the logical functoid is true, the looping functoid allows the creation of the target node.

The Equal functoids here return true, if the values from the respective nodes match the configured value (hardcoded inside the functoid). we then connect the output of both the Equal functoids to a Logical AND functoid, so that the final output is true only if both the Equal functoids are true.

This way you can actually add more conditions by using any combination of Logical fuinctoids available in BizTalk.

BizTalk 2006 -> Example using the new POP3 and Sharepoint Adapters

by community-syndication | Feb 3, 2006 | BizTalk Community Blogs via Syndication

Below is an example using the new POP3 and Sharepoint Adapters (Sharepoint Adapter now out of the box) for BizTalk 2006. This example demonstrates the following:

1) Processing binary documents in BizTalk.
2) A method to process multiple attachments in an incoming mail message.
3) Improvements to the Sharepoint Adapter.

You can download the code at the end of this blog.

The example goes something like this.

A candidate is applying for a job. The candidates resume (Word) and an informational spreadsheet (Excel) are attached to an email message as below:

The resume is in word format, and the excel spreadsheet is as below:

The mail message with attachments is read by BizTalk server. BizTalk server will then add a new item to a Sharepoint Document library. The Word document from the mail message is added as the item’s document and the Excel spreadsheet is parsed to populate the First Name and Last Name columns as below:

The sample works as below:

1) A BizTalk Receive Port/Receive Location is configured using the POP3 receive Adapter as below:

Note: The Apply MIME Decoding property is set to false. This is because we want the raw MIME message to be delivered to an orchestration. The two attachments of the message will be dealt with in the orchestration (see below).

2) An orchestration as below will then subscribe to the incoming encoded MIME message:

3) In the orchestration, a receive pipeline (ConstructCandidateInfo Construct Shape) is executed to extract the Excel message out of the encoded MIME message and to also parse the Excel message to an XML format.

The pipeline looks as below:

Note: A MIME decoder is used in the pipeline to extract out the Excel Attachment
Note: The ODBC File decoder  is used in the pipeline to parse the Excel Attachment to an  XML message. This is so the FirstName and LastName can be used to populate the First Name and Last Name columns in the Sharepoint document library list.
Note: Executing receive pipelines in an orchestration is a new feature for BizTalk 2006.

The code to execute the pipeline in the orchestration is as below:

// Execute the Pipeline -> ReceivePipelineCandidateInfo.
// This will create a message with the XML in it
varPipelineOutPutMessages = 
Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(BizTalkPop3AndSharepointExample.ReceivePipelineCandidateInfo), 
msgMimeCandidate);
msgCandidateInfo = null;
varPipelineOutPutMessages.MoveNext();
varPipelineOutPutMessages.GetCurrent(msgCandidateInfo);
 
4) In the orchestration, a receive pipeline (ConstructResume Construct Shape) is executed to extract the Word document.

The pipeline looks as below:

Note: A MIME decoder is used in the pipeline to extract out the Word Attachment

The code to execute the pipeline in the orchestration is as below:

// Execute the Pipeline -> ReceivePipelineCandidateResume.
// This will create a message with the Word Document in it
varPipelineOutPutMessages = 
Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(BizTalkPop3AndSharepointExample.ReceivePipelineCandidateResume), 
msgMimeCandidate);
msgCandidateResume = null;
varPipelineOutPutMessages.MoveNext();
varPipelineOutPutMessages.GetCurrent(msgCandidateResume);

// Get the FirstName and LastName that are in the Parsed XML Excel message.
// set these into the WSS ConfigPropertiesXml
varXmlDomCandidateInfo = msgCandidateInfo;
varXmlNode = varXmlDomCandidateInfo.SelectSingleNode(“//FirstName”);
strFirstName = varXmlNode.InnerText;
varXmlNode = varXmlDomCandidateInfo.SelectSingleNode(“//LastName”);
strLastName = varXmlNode.InnerText;
// Set the first names and last names so they will
// appear in the Document Library Columns
strWSSConfigPropertiesXml = “<ConfigPropertiesXml><PropertyName1>First Name</PropertyName1><PropertySource1>” + strFirstName + 
“</PropertySource1><PropertyName2>Last Name</PropertyName2><PropertySource2>” + strLastName + “</PropertySource2></ConfigPropertiesXml>”;
msgCandidateResume(WSS.ConfigPropertiesXml) = strWSSConfigPropertiesXml;
// Set the name of the File, when created in the Sharepoint Document Library
msgCandidateResume(WSS.Filename) = strFirstName + ” ”  + strLastName + “.doc”;

Note: msgCandidateResume(WSS.ConfigPropertiesXml) = strWSSConfigPropertiesXml;
is used to set the First Name and Last Name columns in the document library.

5) The msgCandidateResume message is then sent to the WSS library via a configured
Send Port using the Sharepoint Adapter as below:

Note: That the Column 01, Column 01 Value, Column 02, Column 02 Value, FileName are not configured.
These are dynamically set by the orchestration as explained in 4).

You can download the code HERE. Also look at the ReadMe

Conclusion

1) The new adapters and adapter enhancements for BizTalk 2006 really open up the doors for integrating all sorts of different applications. For a complete list of adapters that will be shipped with BizTalk 2006, please go HERE  
2) You can use the new features of BizTalk 2006 (like executing Receive Pipelines in an Orchestration), to easily process complex messages in BizTalk.
3) Adrian Hamza and his team have developed a top notch, feature rich Sharepoint adapter for BizTalk 2006. For more information on the Sharepoint adapter, please visit Adrian’s blog HERE. Adrian also has recorded a series of WebCasts on the Sharepoint Adapter:

WSS Adapter Training Videos:

WSSAdapter-PropsInOrchestration.wmv (28.58 MB)
WSSAdapter-SendReceiveCBR.wmv (10.25 MB)
WSSAdapter-SetupAndConfig-Short.wmv (9.58 MB)
WSSAdapter-InfoPathIntegration.wmv (15.11 MB)

You can download these at BetaPlace (BizTalk Server 2006 section)
http://beta.microsoft.com.

To get access to the BizTalk Server 2006 area at BetaPlace, please goto :
http://www.microsoft.com/biztalk/evaluation/bts2006beta.mspx