Mapping: Conditional Output

by community-syndication | Dec 30, 2005 | BizTalk Community Blogs via Syndication

In this blog entry I will show two different ways to use logical functoids to affect the output of the mapping process.  Just as in the last mapping blog entry these two examples are at the top of the list of mapping questions people ask me.

The first example will show how to split the incoming data into multiple grouped output records.  So, lets say I receive a shipping list that contains all of the orders for that day.  This listing includes the addresses for locations all over the world.  The destination requires that I group the output based on country.

The source schema looks like:

With the following as the source instance (with fake data, of course):

<ns0:Root xmlns:ns0=”http://Map_blog_sample.Source”>

  <Address>

    <ShipTo>Jim Smith</ShipTo>

    <Address>12 Old Main Street</Address>

    <City>RiverCity</City>

    <State>Maine</State>

    <Zip>12345</Zip>

    <Country>USA</Country>

  </Address>

  <Address>

    <ShipTo>Jim Anderson</ShipTo>

    <Address>87 West End Trail</Address>

    <City>Winnipeg</City>

    <State>Ontario</State>

    <Zip>HP7-21A</Zip>

    <Country>Canada</Country>

  </Address>

  <Address>

    <ShipTo>Phil Hewlett</ShipTo>

    <Address>4897 Bushnell Drive</Address>

    <City>Forest Glen</City>

    <State>Wisconsin</State>

    <Zip>54321</Zip>

    <Country>USA</Country>

  </Address>

</ns0:Root>

And the destination schema looks like:

In order to make this split happen we need to include 2 Equal functoids from the Toolbox in the Logical Functoids section.  Drag a line from the Country Node to each Equal functoid.  Whenever I drag a line to a functoid I like to label the line.  You can label the line by clicking on the line and entering text in the Label property in the Properties Windows.  This is helpful when I open the Input Parameters window for the functoid so that I can see a meaningful label instead of a long XPATH statement.

In the Input Parameters window of the first Equal functoid add a constant and label it USA.  Do the same for the second Equal functoid but label the constant Canada.  Now drag a line from the USA functoid to the USAddresses record and drag a line from the Canada functoid to the ForeignAddresses record.  Now all that is left is to drag the lines from the source fields to the matching destination fields.

The map looks like this:

And the destination XML looks like this:

<ns0:Root xmlns:ns0=”http://Map_blog_sample.Destination”>

        <USAddresses>

                <ShipTo>Jim Smith</ShipTo>

                <Address>12 Old Main Street</Address>

                <City>RiverCity</City>

                <State>Maine</State>

                <Zip>12345</Zip>

        </USAddresses>

        <USAddresses>

                <ShipTo>Phil Hewlett</ShipTo>

                <Address>4897 Bushnell Drive</Address>

                <City>Forest Glen</City>

                <State>Wisonsin</State>

                <Zip>54321</Zip>

        </USAddresses>

        <CanadianAddresses>

                <ShipTo>Jim Anderson</ShipTo>

                <Address>87 West End Trail</Address>

                <City>Winnepeg</City>

                <Province>Ontario</Province>

                <Zip>HP7-21A</Zip>

        </CanadianAddresses>

</ns0:Root>

One thing to notice is the difference between the input and the output order.  There is no way to guarantee the order through the mapping process.  This is because when the BizTalk Mapper generates the XSLT it examines the destination schema structure and then it propagates through any grid layers to pull out the values from the source schema structure.   So, after saying that there is no guarantee to the order there is a way to coach the process.  The order in which elements and attributes appear in an output instance message is dependent on the order of the records and fields of the destination schema.

The second example will show how to suppress a destination node based on the existence of or lack of a source node.

In this scenario the country element will only be filled out if the address is not in the United States – we want the output to include only those records that are in foreign countries.

In this scenario we will use the same input instance as well as the same source and destination schemas. 

This time the mapper will use the Logical Existence functoid as well as the Value Mapping functoid (5 for this example). 

The logical existence functoid will return true if the tags exists but there is no value.  For the logical existence to return false then the entire node must not be present.  In our case we want the logical existence functoid to return false when it iterates over a record where the address would be in the United States. 

To create the map follow these steps:

1)  drag a line from the Country node to the logical existence functoid. 

2)  drag a line from the logical existence functoid to each of the 5 Value Mapping functoids.  It is important that these lines be the first lines connected to the value mapping functoids

3) drag a line from the ShipTo element to the first Value Mapping functoid.  Since I may end up at the link in the functoids Property Window I will name the link. 

4) repeat step 3 for each of the remaining elements.  Do not map the Country element.

5) last, drag a line from the Logical Existence functoid to the ForeignAddresses record. 

This last item is interesting.  The logical functoids have a special ability in that when they return a true or false and are connected to a destination element or record they will suppress the output of that element or record when the value of the functoid is false.  This is exactly what we want since we do not want a record output for the United States Addresses. 

If the line from the logical existence functoid is not drawn to the ForeignAddresses record then the output will look like:

<ns0:Root xmlns:ns0=”http://Map_blog_sample.Destination”>

        <ForeignAddresses></ForeignAddresses>

        <ForeignAddresses>

                <ShipTo>Jim Anderson</ShipTo>

                <Address>87 West End Trail</Address>

                <City>Winnepeg</City>

                <Province>Ontario</Province>

                <Zip>HP7-21A</Zip>

        </ForeignAddresses>

        <ForeignAddresses></ForeignAddresses>

        <ForeignAddresses>

                <ShipTo>Jake Swenson</ShipTo>

                <Address>12 West Winfield Drive</Address>

                <City>Rome</City>

                <Province></Province>

                <Zip>YA1-23U</Zip>

        </ForeignAddresses>

</ns0:Root>

Notice that there are extra ForeignAddresses tags.  These are the tags where the input record was a United States address (the country element was missing). 

The map should look like this:

If we follow steps 1 through 5 then the output should look like:

<ns0:Root xmlns:ns0=”http://Map_blog_sample.Destination”>

        <ForeignAddresses>

                <ShipTo>Jim Anderson</ShipTo>

                <Address>87 West End Trail</Address>

                <City>Winnepeg</City>

                <Province>Ontario</Province>

                <Zip>HP7-21A</Zip>

        </ForeignAddresses>

        <ForeignAddresses>

                <ShipTo>Jake Swenson</ShipTo>

                <Address>12 West Winfield Drive</Address>

                <City>Rome</City>

                <Province></Province>

                <Zip>YA1-23U</Zip>

        </ForeignAddresses>

</ns0:Root>

Message Header vs. Context Terminology

by community-syndication | Dec 28, 2005 | BizTalk Community Blogs via Syndication

Is anyone confused about the conflicting terminlogy I have seen in the BizTalk documentation regarding BizTalk message headers and contexts? I have always understood a message to be just a context and a body.  There is no ‘header’ in a BizTalk message that has been received by an adapter.   For instance, here is an exerpt from the product documenation and I have seen this terminlogy conflict in other whitepapers and documentation.

A message is made up of a header section and a body section:

%u00b7                      Headers are simple name-value pairs. The names are all drawn from various property schemas. The system provides some core schemas and BizTalk Server applications can add their own.

I believe this should read:

BizTalk Samples

by community-syndication | Dec 28, 2005 | BizTalk Community Blogs via Syndication

I am looking for input on the practicality of using the BizTalk Adapter Framework to build a custom adapter.

• For those of you who have used it what were your experiences, good or bad?
  
• Can you describe the development process as to how you took the Framework and turned it into your adapter?
  
• What documentation did you use?
  
• Does Microsoft provide adequate documentation on the process?
  
• Would a step-by-step ‘how to’ document cut down on development time and be useful?

BizTalk 2004: Mapping Empty or Missing attributes and elements with different combinations of parameters: Required/Optional, Min/MaxOccurs, Default, Fixed

by community-syndication | Dec 23, 2005 | BizTalk Community Blogs via Syndication

Mapping Empty or Missing attributes and elements with different combinations of parameters: Required/Optional, Min/MaxOccurs, Default, Fixed

When I create maps it is annoying to guess which values I’ll get if I have to mapping Empty or Missing attributes/elements. I’ve

made an investigation on the base of a Standard “XML Schema Part 0: Primer. W3C Recommendation, 2 May 2001”.
http://www.w3.org/TR/2001/REC-xmlschema-0-20010502/#OccurrenceConstraints (below is this article).
I added the last column with my comments to the original table (in BizTalk mapping are some differences from the standard!).

I used the BTS2004 Mapper and “Test Map” command. I’ve changed the properties of attribute and element in a target schema.
I waited that the behaviour of mapping should be the same as in standard.
Results are different in several points:
1) “Fixed” and “Default” properties don’t change mapping. They only change validation and they have a different effect on validation.
2) “Required” property of attributes: If source attribute is missing the mapping creates the attribute with Empty value.
3) Something strange with “min/maxOccures” properties of elements: If they are (1,1), (-,1), (1,-), or (-,-) and source element is missing the mapping creates an element with Empty value! Only these combinations generate an element with Empty value, other combinations do not create an element.

“
2.2.1 Occurrence Constraints
The comment element is optional within PurchaseOrderType because the value of the minOccurs attribute in its declaration is 0. In general, an element is required to appear when the value of minOccurs is 1 or more. The maximum number of times an element may appear is determined by the value of a maxOccurs attribute in its declaration. This value may be a positive integer such as 41, or the term unbounded to indicate there is no maximum number of occurrences. The default value for both the minOccurs and the maxOccurs attributes is 1. Thus, when an element such as comment is declared without a maxOccurs attribute, the element may not occur more than once. Be sure that if you specify a value for only the minOccurs attribute, it is less than or equal to the default value of maxOccurs, i.e. it is 0 or 1. Similarly, if you specify a value for only the maxOccurs attribute, it must be greater than or equal to the default value of minOccurs, i.e. 1 or more. If both attributes are omitted, the element must appear exactly once.

Attributes may appear once or not at all, but no other number of times, and so the syntax for specifying occurrences of attributes is different than the syntax for elements. In particular, attributes can be declared with a use attribute to indicate whether the attribute is required (see for example, the partNum attribute declaration in po.xsd), optional, or even prohibited.

Default values of both attributes and elements are declared using the default attribute, although this attribute has a slightly different consequence in each case. When an attribute is declared with a default value, the value of the attribute is whatever value appears as the attribute’s value in an instance document; if the attribute does not appear in the instance document, the schema processor provides the attribute with a value equal to that of the default attribute. Note that default values for attributes only make sense if the attributes themselves are optional, and so it is an error to specify both a default value and anything other than a value of optional for use.

The schema processor treats defaulted elements slightly differently. When an element is declared with a default value, the value of the element is whatever value appears as the element’s content in the instance document; if the element appears without any content, the schema processor provides the element with a value equal to that of the default attribute. However, if the element does not appear in the instance document, the schema processor does not provide the element at all. In summary, the differences between element and attribute defaults can be stated as: Default attribute values apply when attributes are missing, and default element values apply when elements are empty.

The fixed attribute is used in both attribute and element declarations to ensure that the attributes and elements are set to particular values. For example, po.xsd contains a declaration for the country attribute, which is declared with a fixed value US. This declaration means that the appearance of a country attribute in an instance document is optional (the default value of use is optional), although if the attribute does appear, its value must be US, and if the attribute does not appear, the schema processor will provide a country attribute with the value US. Note that the concepts of a fixed value and a default value are mutually exclusive, and so it is an error for a declaration to contain both fixed and default attributes.

The values of the attributes used in element and attribute declarations to constrain their occurrences are summarized in Table 1.
“

BizTalk exception handling with web services

by community-syndication | Dec 22, 2005 | BizTalk Community Blogs via Syndication

Recently I needed to include exception handling in a web service call to an external party. I learned a couple of things and thought I’d highlight them here.

Exception handling doesn’t always behave the way you expect when you are trying to catch exceptions when calling a web service. While using one is very useful some things must be considered or you will not get the result you wanted.

Here is one way to do it as well as a few things to keep in mind when you want to set one up:

• Include the external web service call (both send and receive shapes) in either a non-transactional scope shape or a long running transaction.
• Decide if you need to catch any specific exceptions, if you don’t, using the General Exception will suffice. If you need to catch specific errors, you can use multiple catch exception blocks.
• It may sound obvious to say this, but make sure your orchestration doesn’t have a timeout that is the same length (or shorter) than the timeout of your scope transaction.
• Most important: in your send port which holds the reference to the web service, make sure you are not using any retries (unless, of course, you explicitly need to). If the web service fails with an error such as “There is an error in XML document (5, 10)” and you have retries enabled you will only see warnings in your event viewer (which means the exception block won’t be activated until the SOAP timeout).

Mapping: Inline XSLT Scenario

by community-syndication | Dec 19, 2005 | BizTalk Community Blogs via Syndication

Many of the blog entries that I write about are based on questions that customers ask me.  One of the areas that customers tend to ask most about are advanced mapping topics.  I thought it would be a good idea to take many of these questions and post them as blog entries. 

The first in this series is an example of when custom inline XSLT is required.

In this example I received a schema where cities were listed with their name and country as elements.  The transformation that needed to take place was to group the output by country and within each country to list all of the cities in that country

The source schema looked like this:

And the destination schema needed to look like this:

And finally the sample input looked like this:

<ns0:Root xmlns:ns0=”http://Testing.SchemaSrc”>
    <cities>
        <city>
            <name>Paris</name>
            <country>France</country>
        </city>
        <city>
            <name>Roma</name>
            <country>Italia</country>
        </city>
        <city>
            <name>Nice</name>
            <country>France</country>
        </city>
        <city>
            <name>Madrid</name>
            <country>Espana</country>
        </city>
        <city>
            <name>Milano</name>
            <country>Italia</country>
        </city>
        <city>
            <name>Firenze</name>
            <country>Italia</country>
        </city>
        <city>
            <name>Napoli</name>
            <country>Italia</country>
        </city>
        <city>
            <name>Lyon</name>
            <country>France</country>
        </city>
        <city>
            <name>Barcelona</name>
            <country>Espana</country>
        </city>
    </cities>
</ns0:Root>

In order to make this transformation you need to use inline XSLT.  There is really no easy way to do this with the functoids provided by BizTalk.

The first thing we need to do is to drag a Script functoid onto the Grid.  Then drag a line to the element that we want to create output for.  With the inline XSLT functionality we are required to produce output for all of the elements underneath the records our functoid is attached.  So, in our example we are responsible for creating all of the content under the countries record.

The map will look like this:

The following XSLT can be cut and paste into the Inline Script Buffer section of the script functoid after the Script Type drop down as been set to Inline XSLT.

To create the output in the format that we want we will start with the following XSLT. 

<xsl:variable name=”unique-countries”

select=”//cities/city[not(@country=preceding-sibling::city/@country)]/@country” />

This creates a variable named unique-countries and populates with a list of unique country nodes.  Now we need to start the output of our nodes and then loop through the countries.  The following code will do that.

1.  <countries>

2.          <xsl:for-each select=”$unique-countries”>

3.                  <country name=”{.}”>

4.                          <xsl:for-each select=”//city[@country=current()]”>

5.                                <city><xsl:value-of select=”@name” /></city>

6.                        </xsl:for-each>

7.                </country>

8.        </xsl:for-each>

9.  </countries>

Lets walk through this code.

Lines 1 and 9 create the countries node output and close out the node. 

Line 2 creates the loop that will loop through all of the unique countries. 

Lines 3 and 7 create the Country node output and close out the node – notice the name attribute.  This syntax takes the current node name and writes out the value. 

Lines 4 and 6 create the country node output and close out the node.  Also line 4 uses the current() function which takes the country of the current node and with the code in line 5 puts the value that is in the source document in the name element and writes it out in the city node.

This code will loop through as many unique countries as are contained in the input document.  Then for each unique country that is found the code will loop through all of the city/name nodes outputting their values.  When the next country node is current then the code loops though looking for all of the city/name nodes that match

The output looks like this:

<ns0:Root xmlns:ns0=”http://Testing.SchemaDst”>

        <countries>

                <country name=”France”>

                        <city>Paris</city>

                        <city>Nice</city>

                        <city>Lyon</city>

                </country>

                <country name=”Italia”>

                        <city>Roma</city>

                        <city>Milano</city>

                        <city>Firenze</city>

                        <city>Napoli</city>

                </country>

                <country name=”Espana”>

                        <city>Madrid</city>

                        <city>Barcelona</city>

                </country>

        </countries>

</ns0:Root>

ManageMessageRefCountLog SQL Agent Job failing in BETA2

by community-syndication | Dec 19, 2005 | BizTalk Community Blogs via Syndication

Since I have seen enough questions on this, I will officially post here about it (and we might have a KB Article on it … not sure how that works for BETAs though). There is a new sql agent job you will find on the messagebox database server called ManageMessageRefCountLog_<messagebox>. In the BETA 2 bits you will find this job fails occassionally on both SQL 2000 and 2005. This is a known issue which we are working to fix in RTM. The failure is ignorable. The error stems from this job attempting to start a secondary sql agent job. If that job is already running we get an error. The approach we were using to determine if the job was already running had a race in it, but we did not get to fix this for BETA2. The job is scheduled to restart itself every minute so even if it fails, it will just restart again. This job is part of our cleanup process for deleting messages. The expectation is that this job will run forever. The only negative side effect the failure of this job can have is if you are running in VERY high performance testing, a one minute down time of this job could cause enough spool growth to trigger our throttling mechanism. In this case, you would see throttling kick in due to spool growth and you could see this. We are of course working to fix this for RTM for this reason and for the fact that customers don’t seem to like the red X that shows up when it fails. 🙂 🙂 If you don’t know how to monitor throttling counters, I recommend reading through our documentation on this and then checking out the MessageAgent perf counters which are now accessible via perfmon. Significant enhancements have been made here to help maintain a healthy system so you should read up on it.

Thx

Lee

BizTalk SQL Scripts for Troubleshooting Deployments

by community-syndication | Dec 18, 2005 | BizTalk Community Blogs via Syndication

Here’s a sql script I use against the BizTalkMgmtDb when I’m debugging issues that I think are caused by botched deployments:

SELECT nvcName AS AssemblyName, nvcVersion AS Version, DATEADD(hh, DATEDIFF(hh, GetUTCDate(), GETDATE()), dtDateModified) AS DateModified, nvcModifiedBy AS ModifiedBy
FROM bts_assembly
WHERE (nSystemAssembly = 0)
ORDER BY AssemblyName

It returns the assembly name, version, the date it was modified (deployed) and who deployed it.  The DATEADD stuff is to put the dates into the appropriate time zone.

Here’s a similiar script but it focuses on orchestrations:

SELECT nvcFullName as OrchestrationName, DATEADD(hh, DATEDIFF(hh, GetUTCDate(), GETDATE()), dtModified) as DateModified
FROM dbo.bts_orchestration

MSF 4.0 and Microsoft Team Services

by community-syndication | Dec 16, 2005 | BizTalk Community Blogs via Syndication

The forthcoming release next year of Visual Studio Foundation Server will include two methodology templates for the next version of MSF (the Microsoft Solution Framework).  These are MSF for Agile Software Development (MSF Agile) and MSF for CMMI Process Improvement (MSF4CMMI).   Here is an amended version of a report I recently wrote.

http://geekswithblogs.net/cyoung/articles/63354.aspx