Filtering records using Maps and conditional looping

by community-syndication | Feb 4, 2006 | BizTalk Community Blogs via Syndication

Quite often you come across situations in BizTalk, where you receive a message that contains hundreds of records, when you are actually interested only in a subset of those records that match a certain criteria and want to create a message that contains only the relevant records.

There are numerous ways of doing it such as:

• Debatch and Aggregate (Scatter-Gather Pattern)
  
• Using xslt in maps
  
• Using xpath in orchestration
  
• Using an external component

There is also another simple and straightforward way of doing it using BizTalk maps known as conditional looping, which you will see in a moment.

Let me illustrate this using an example. Assume that you receive a Customer message that contains all your customers. However, your requirement is a Customer message that contains only those customers who match the following criteria:

City = ‘New York’
Rating = ‘High’

A simple map shown below will do the trick.

When the outputs of a Looping functoid in conjunction with any of the Logical functoids are connected to the same target node, the behavior is as follows:

• If the output of the logical functoid is false, the looping functoid will suppress the creation of the target node.
  
• If the output of the logical functoid is true, the looping functoid allows the creation of the target node.

The Equal functoids here return true, if the values from the respective nodes match the configured value (hardcoded inside the functoid). we then connect the output of both the Equal functoids to a Logical AND functoid, so that the final output is true only if both the Equal functoids are true.

This way you can actually add more conditions by using any combination of Logical fuinctoids available in BizTalk.

BizTalk 2006 -> Example using the new POP3 and Sharepoint Adapters

by community-syndication | Feb 3, 2006 | BizTalk Community Blogs via Syndication

Below is an example using the new POP3 and Sharepoint Adapters (Sharepoint Adapter now out of the box) for BizTalk 2006. This example demonstrates the following:

1) Processing binary documents in BizTalk.
2) A method to process multiple attachments in an incoming mail message.
3) Improvements to the Sharepoint Adapter.

You can download the code at the end of this blog.

The example goes something like this.

A candidate is applying for a job. The candidates resume (Word) and an informational spreadsheet (Excel) are attached to an email message as below:

The resume is in word format, and the excel spreadsheet is as below:

The mail message with attachments is read by BizTalk server. BizTalk server will then add a new item to a Sharepoint Document library. The Word document from the mail message is added as the item’s document and the Excel spreadsheet is parsed to populate the First Name and Last Name columns as below:

The sample works as below:

1) A BizTalk Receive Port/Receive Location is configured using the POP3 receive Adapter as below:

Note: The Apply MIME Decoding property is set to false. This is because we want the raw MIME message to be delivered to an orchestration. The two attachments of the message will be dealt with in the orchestration (see below).

2) An orchestration as below will then subscribe to the incoming encoded MIME message:

3) In the orchestration, a receive pipeline (ConstructCandidateInfo Construct Shape) is executed to extract the Excel message out of the encoded MIME message and to also parse the Excel message to an XML format.

The pipeline looks as below:

Note: A MIME decoder is used in the pipeline to extract out the Excel Attachment
Note: The ODBC File decoder  is used in the pipeline to parse the Excel Attachment to an  XML message. This is so the FirstName and LastName can be used to populate the First Name and Last Name columns in the Sharepoint document library list.
Note: Executing receive pipelines in an orchestration is a new feature for BizTalk 2006.

The code to execute the pipeline in the orchestration is as below:

// Execute the Pipeline -> ReceivePipelineCandidateInfo.
// This will create a message with the XML in it
varPipelineOutPutMessages = 
Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(BizTalkPop3AndSharepointExample.ReceivePipelineCandidateInfo), 
msgMimeCandidate);
msgCandidateInfo = null;
varPipelineOutPutMessages.MoveNext();
varPipelineOutPutMessages.GetCurrent(msgCandidateInfo);
 
4) In the orchestration, a receive pipeline (ConstructResume Construct Shape) is executed to extract the Word document.

The pipeline looks as below:

Note: A MIME decoder is used in the pipeline to extract out the Word Attachment

The code to execute the pipeline in the orchestration is as below:

// Execute the Pipeline -> ReceivePipelineCandidateResume.
// This will create a message with the Word Document in it
varPipelineOutPutMessages = 
Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(BizTalkPop3AndSharepointExample.ReceivePipelineCandidateResume), 
msgMimeCandidate);
msgCandidateResume = null;
varPipelineOutPutMessages.MoveNext();
varPipelineOutPutMessages.GetCurrent(msgCandidateResume);

// Get the FirstName and LastName that are in the Parsed XML Excel message.
// set these into the WSS ConfigPropertiesXml
varXmlDomCandidateInfo = msgCandidateInfo;
varXmlNode = varXmlDomCandidateInfo.SelectSingleNode(“//FirstName”);
strFirstName = varXmlNode.InnerText;
varXmlNode = varXmlDomCandidateInfo.SelectSingleNode(“//LastName”);
strLastName = varXmlNode.InnerText;
// Set the first names and last names so they will
// appear in the Document Library Columns
strWSSConfigPropertiesXml = “<ConfigPropertiesXml><PropertyName1>First Name</PropertyName1><PropertySource1>” + strFirstName + 
“</PropertySource1><PropertyName2>Last Name</PropertyName2><PropertySource2>” + strLastName + “</PropertySource2></ConfigPropertiesXml>”;
msgCandidateResume(WSS.ConfigPropertiesXml) = strWSSConfigPropertiesXml;
// Set the name of the File, when created in the Sharepoint Document Library
msgCandidateResume(WSS.Filename) = strFirstName + ” ”  + strLastName + “.doc”;

Note: msgCandidateResume(WSS.ConfigPropertiesXml) = strWSSConfigPropertiesXml;
is used to set the First Name and Last Name columns in the document library.

5) The msgCandidateResume message is then sent to the WSS library via a configured
Send Port using the Sharepoint Adapter as below:

Note: That the Column 01, Column 01 Value, Column 02, Column 02 Value, FileName are not configured.
These are dynamically set by the orchestration as explained in 4).

You can download the code HERE. Also look at the ReadMe

Conclusion

1) The new adapters and adapter enhancements for BizTalk 2006 really open up the doors for integrating all sorts of different applications. For a complete list of adapters that will be shipped with BizTalk 2006, please go HERE  
2) You can use the new features of BizTalk 2006 (like executing Receive Pipelines in an Orchestration), to easily process complex messages in BizTalk.
3) Adrian Hamza and his team have developed a top notch, feature rich Sharepoint adapter for BizTalk 2006. For more information on the Sharepoint adapter, please visit Adrian’s blog HERE. Adrian also has recorded a series of WebCasts on the Sharepoint Adapter:

WSS Adapter Training Videos:

WSSAdapter-PropsInOrchestration.wmv (28.58 MB)
WSSAdapter-SendReceiveCBR.wmv (10.25 MB)
WSSAdapter-SetupAndConfig-Short.wmv (9.58 MB)
WSSAdapter-InfoPathIntegration.wmv (15.11 MB)

You can download these at BetaPlace (BizTalk Server 2006 section)
http://beta.microsoft.com.

To get access to the BizTalk Server 2006 area at BetaPlace, please goto :
http://www.microsoft.com/biztalk/evaluation/bts2006beta.mspx

ESB Project: our first milestone….

by community-syndication | Feb 2, 2006 | BizTalk Community Blogs via Syndication

I still owe you some architectural overview discussions, but first I wanted to “converge the timeline” and make these posts current by bringing you up to where we are at in the project today. The architectural posts will come soon (really!). I have been somewhat buried in develop/deploy mode, so documentation of all sorts has languished, but that will change soon.

Right now, we are working towards our first milestone.

Milestone 1 Composition
At a glance, it’s a very easy milestone to hit, and contains limited functionality. It is however very significant because it will prove out the processes we are putting in place, as well as the foundation we’re laying down, for both hardware and software. As a firm believer in iterative development, this approach suits me fine, and it has proven its value as we have moved through the process.

From a BizTalk perspective, the initial milestone consists of:

•Dynamic transformation. Only two external sources will be available in this initial release for map selection: hard coding of a map name based on the receive location, and the ability to call the BizTalk rules engine, passing through some data about the message being transformed. Subsequent releases will have more selection options, and I will detail those and the mechanism we’ll be using in future posts.

•Exception handling. The version of exception handling being deployed is extremely simplistic compared to what’s coming. The reason this is here at all right now is that there is a chance of failures in dynamic transformations (eg: invalid map name, map not deployed, mapping failure, etc), so we had to provide something.

•Heartbeat. Think of this as a “ping service”. It’s simply a Web service that calls a BizTalk orchestration, and the orchestration echoes back what it was sent as a parameter. Braindead simple, but it will allow us to give the Java side of a house a WSDL they can hit to prove interoperability, and has us deploying an initial Web service into our environment (along with virtual directories, app pools, and all the other little challenges that could potentially involve). We are also bundling in a simple UI to confirm the Web service is working.

•JMS pipeline component. We have a JMS pipeline component that promotes/demotes JMS header properties into/out-of message content. To test this we need to pull a JMS message off an MQ queue, and deposit it in another queue. JMS header properties on the outbound message should be preserved and be identical to the inbound ones.

•Functional tests. From my side we have identified four tests that will be run. Given time constraints, this will only be partially automated, but the automated ones will push NUnit and BizUnit onto the QA servers. Ultimately, the goal is to have most, if not all, tests run fully-automated inside NUnit.

From a non-BizTalk perspective, the initial milestone includes:

•Systinet UDDI registry running on AIX (that’s “Unix” to you purely-MSFT people who may not know :))
•AmberPoint Web services management, also running on AIX (including client and service metrics collection)

From an infrastructure perspective, the initial milestone includes:

•High availability MQ Series (AIX only for now, a Windows HA MQ cluster will follow)
•High availability BizTalk (self-clustered like you do with BizTalk servers, plus tuned with multiple MessageBoxes)
•High availability SQL Server (clustered, and tuned)

This is a formal, structured environment. We will not have the luxury of admin control over QA boxes, nor will we be doing our own deployments (we’re not allowed to touch the boxes). In order to succeed, this needs to be done in a meticulous and well documented way, and that’s what we’re doing.

First stage
Our first stop was to deploy to a “QA” Virtual Machine. This would mark the first installation that was not on my (the developer) VM. We have multiple MSIs, and as expected, it did not work properly the first time. We missed some dependencies, and had to solve some non-development issues, so we iterated through this process several times before we got it right. The fact that we are using a virtual machine for this stage saved us lots of time, we can snapshot the machine, do an install, determine what’s missing, revert back to the snapshot, and do it all over. We could achieve the same result using a physical machine and something like Ghost, but this is much MUCH more convenient.

The steps I went through are:

•I have both a Dev and a QA VM running at the same time (we’re using VMWare)
•In the Dev VM, I build the MSIs and drag them into a shared folder on the host
•I switch to the QA VM, and revert to a “clean” state by restoring a snapshot
•I drag the MSIs over from the shared host folder and do the install
•I load and run the NUnit tests inside the NUnit GUI
•I repeat as necessary to correct any issues

The whole process worked very well, and I was highly productive. Without virtual machines, this whole process would have taken MUCH longer. It was also pretty cool that I was able to work on this while riding the BART (Bay Area Rapid Transit, a train I take most days). I can’t imagine BizTalk development without VM technology!

Build/Deploy Team Composition
From a team perspective, it was at this stage that I involved Tom Canter of MidTech Partners, who has been working alongside me on this project. Up until this point, the division of labor between us was that I did the BizTalk architecture and development (building out the core ESB Engine), while he defined and deployed the complex server infrastructure that all this will run on. We work very well together, and we have a clean division of labor, and complementary skill sets. When explaining our roles to people, I just say “I’m software, Tom’s hardware”. Between us, we can do pretty well anything that needs to be done.

Tom and I started working together on the deployment as we were near the end of the “create the installation bits” stage. I got it to the point I was happy with project names and project granularity, and had preliminary installations working. Tom took it from there and fleshed out the scripting part, continually moving towards the goal of “single click deployment” for the ESB engine and functional tests. He had a lot of challenges with file paths, as we wanted to be able to install in any environment, including multi-BizTalk environments, and we didn’t know which drive or path we’d be installing to when we created the MSIs. In addition, the functional test receive locations needed to reflect the installation path, and we had other path dependencies for the BizUnit/NUnit test cases (which are XML files). Then, we had two key variants on the installs: a runtime install needed for Dev, QA, Staging, Production, and a developer install that any developer c