Changing a VPC Computer Name with BizTalk

by community-syndication | Apr 26, 2005 | BizTalk Community Blogs via Syndication

Im looking at using Virtual PC with a team of BizTalk developers. The idea is to install the dev env (Server 2003, SQL Server, Visual Studio, BizTalk etc) on a virtual PC image, and then each team member uses a copy of the image to develop with.

We have network problems if two or more images with the same computer name are started with network access (to SourceSafe). So I attempted to change the computer name in an image. After a while I managed to do it like this:

Export any information from the BizTalk databases (business rules etc.)
Run ConfigFramework /u
Delete BizTalk jobs in SQL Server Agent
Delete BizTalk logons in SQL Server Security
Delete BizTalk databases
Change computer name
Re-start computer
Change SQL Server Name (with sp_dropserver, sp_addserver)
Run ConfigFramework
Change BizTalkMgmtDB connection in Visual Studio
Change rules engine DB connection in Business Rules Policy Editor
Re-enable BackupBizTalk server DBs job
Add any Hosts and host instances using BizTalk Server Administration
Import information to the BizTalk databases (business rules etc.)

I’m wondering if anyone else is using VPC with a BizTalk team, and has come across this issue, or has any tips. (I’ll post an article here and include any feedback).

Is there a workaround for having multiple VPCs with the same name? And is there an ‘easy’ way to change the name of a BizTalk box?

Update
Another Stockholm BizTalk guy called Ali pointed me to this. Looks like it’s easy to avoid the “world of pain” of re-naming the VM PC, thanks to Dunk (wonder why that never got in the Bloggers Guide??). I’d still be keen to know if the re-naming procedure could be optimised…

Schema Design Patterns: Salami Slice

by community-syndication | Apr 25, 2005 | BizTalk Community Blogs via Syndication

This is the second of five entries talking about schema design patters.  The previous entry discussed the Russian Doll approach.

In the Salami Slice approach all elements are defined globally but the type definitions are defined locally.  This way other schemas may reuse the elements.  With this approach, a global element with its locally defined type provide a complete description of the elements content.  This information ‘slice’ is declared individually and then aggregated back together and may also be pieced together to construct other schemas.

<?xml version=”1.0″ encoding=”UTF-8″?>
<xs:schema xmlns:xs=”http://www.w3.org/2001/XMLSchema” elementFormDefault=”qualified” attributeFormDefault=”unqualified”>
    <xs:element name=”BookInformation”>
        <xs:complexType>
            <xs:sequence>
                <xs:element ref=”Title”/>
                <xs:element ref=”ISBN”/>
                <xs:element ref=”PeopleInvolved”>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
    <xs:element name=”Title”/>
    <xs:element name=”ISBN”/>
    <xs:element name=”PeopleInvolved”>
        <xs:complexType>
            <xs:sequence>
                <xs:element ref=”Author”/>
                <xs:element ref=”Publisher”/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
    <xs:element name=”Author”/>
    <xs:element name=”Publisher”>
</xs:schema>

The advantage is that the schema is reusable since the elements are declared globally.

The disadvantages are: the schema is verbose since each element is declared globally and then referenced to describe the data which leads to larger schema size.  This approach also is not self contained and is coupled.  The elements defined may be contained in other schemas and because of this the schema is coupled to other schema and thus changes to one schema will impact other schemas.

This type of approach is commonly used since it is easy to understand and create reusable components.  It would be an appropriate design to promote reusability and data standardization across differing applications.  This approach is not, however, recommended when modifications to the standard elements will be necessary.  If the length, data types, restrictions or other modifications of the elements need to be changed then this will cause added work as well as a larger impact to other systems.

Back on Track with the Guide

by community-syndication | Apr 24, 2005 | BizTalk Community Blogs via Syndication

There’s a new Bloggers Guide out today. I missed the March release, (moving apartments, painting, laying floors etc.). There will be a May release in a couple of weeks, and I will add the new blogs that have been submitted (sorry I could not get them in this release guys). The 5MB size restriction of GotDotNet may cause problems in the future, (the current ZIP file is 5.26MB, so I’m not sure when (or if) the restriction will kick in).

Get it here.

Implementation of Message Transformation Normaliser Pattern in the BizTalk 2004

by community-syndication | Apr 21, 2005 | BizTalk Community Blogs via Syndication

This is a simple explanation of how to join together two schemas with a 1:1 relationship between them in the Biztalk Mapper. I came across this problem when I had 2 legacy csv files which needed to be merged (using a unique identifier field in each file) before further processing could occur in a Biztalk Orchestration.

I will show by example. The example is a bit convoluted, but stay with me 😉 different departments are ordering the same parts but are paying different prices we want to see the difference in price for the parts. There are also fields missing from each field which we want to add to the finished schema.

1) So first create 2 schemas one for each department, note the common PartID which is the unique identifier and price, but also not that one file comes with a Part name and another with a manufacturer:

2) Create the Output Schema

3) Now create an orchestration which will call the Map to join the two schemas by taking them both in and using a transform shape to map the two then it will output them

4) Set the transform properties as below create a new map set the destination to the output map

5) Which will create a map

6) Finally the biz … ok on the map join the common fields from the first schema (but not the second) with fields in the output schema now add a script functoid and set it up like this. Using an XSLT template it will search the second message for the values (simple xpath) where the PartID in the first message matches the PartID in the second message. It will then return the value of the price attribute in the second message. From this you can see how powerful using inline xslt is.

7) Add another script block to get the manufacturer field set the inline script buffer to:

Managing Flat Files in BizTalk 2004

by community-syndication | Apr 21, 2005 | BizTalk Community Blogs via Syndication

I have done a heap of work with Flat Files (mostly csv) in Biztalk 2004 I collected a lot of articles from other Blogs on this which were fab I though I would centralise this info for you, again thanks goes to the fellow bloggers for posting this stuff.

BizTalk 2004 Flat File Schema Tutorial 1

BizTalk 2004 Flat File Schema Tutorial 2

BizTalk Flat File Parsing Annotations

The flat file strikes back: BizTalk 2004 parsing positional records

Not sure where I got this from but it’s a great way of removing the first line from a file if it contains the column headers without having to write a new biztalk dissassembler pipeline component, so thanks to whoever worked this one out.

“To setup a CSV / TSV file, do the following
Setup a schedma as normal. It should be -> products -> product -> all your fields
In , select schema editor extensions, and add “Flat File Extension”
In Products, set the child order to postfix, the child delimiter type to “Hex” and the delimiter to “0x0D 0x0A” (CR/LF). Adjust as needed.
In Product, set the Child order to Infix, the type to Hex (or char) and the delimiter to 0x09 (tab), or , (comma). Adjust as needed.
To skip the first line of a file, during the mapping (eg, the file has a header).
Setup the mapping as normal
Add an Iteration functiod. Connect it up to the “product” level record (eg, product, not products).
Add a not equal (<>) functiod. Connect it to the Iteration functoid, and the target record. Set the second value on that to 1.
If you have trouble deploying a generated xml deployment file (eg, from BTSDeploy wizard) which contains a sql connector, you may find that the user that BizTalkServer is using, does not have access to the target database. Give it access :).
Send Ports are the best place to put maps. I never managed to get them to work in receive ports – the documents tended to disappear.
When in doubt, reboot. For example, if everything works on the development machine, but for some odd reason deploy on the test server and then doesn’t work – reboot the test server. Chances are, its managed to have something hang around in the GAC or something…. restarting BTS might work too.”

This next one is a good overview of property setting on a flat file schema it is taken from this blog post which is not accessable at the moment: http://weblogs.ilg.com/brumfieldb/archive/2004/08/09/440.aspx

There doesn’t seem to be a lot of good information on disassembling various types of flat-files in BizTalk.   There are a number of flat-file properties added for a schema set with the Flat-File Extensions, but it’s not always clear on how to use all these options to accomplish what you need.  Hopefully this post will help serve as an example of how to use some of the flat-file features in BizTalk to parse more than a straightforward comma delimited file.

For one of my projects, I needed to disassemble a file with a format like this:

HEADER    USER_1234

PURCHASE    PO_001

LINEITEM    Item32    33

LINEITEM    Item63    45

PURCHASE    PO_002

LINEITEM    Item454    12

The file contains a header record for each file, one or more purchase orders each with one or more line items.  Each line contains a tag that indicates its usage (e.g. LINEITEM).  The elements for each line are separated by a tab (ASCII hex code of 0x09).

We define a target schema to represent the xml-ized version of this file.

The schema root and each element group must be appropriately configured to correctly parse the flat file.  The configuration for each root and group node are defined as follows:

Schema Root

The settings in the schema root define the default usage in sub-groups throughout the schema.  Here we’ve defined it to be CR LF and the Default Child Order defines that the data will precede the delimiters.  These settings can be overridden at each group level if necessary.

Orders

Header

The header group allows us to specify a tag identifier as well as note that the elements on this line are tab (0x09) delimited.

Order

OrderHeader

LineItems

LineItems is a logical grouping of LineItems.  Since LineItems in the flat file are separated by a CR LF, the Delimiter type is set to the default child delimiter and that the delimiter will appear after the child item.

LineItem

Notes: 

• Most of the group nodes do not have any real values in the flat-file, but instead are logical groupings of real elements.  For instance, LineItems has no real representation in the flat file, but is the logical grouping of LineItem elements from the flat file.  These groups must be setup to for the postfix child order.
  
• Tag Identifiers are used by BTS to recognize a line, but are not imported as data.
  
• With this specification, there needs to be a CR LF following the last line.

R. Addis

Schema Design Patterns: Russian Doll

by community-syndication | Apr 21, 2005 | BizTalk Community Blogs via Syndication

I am working on a BizTalk solution which uses a schema that was hand created using XML Spy.  The schema was created in such a way in which everything was globally available and when I brought it into the BizTalk editor the schema looked to have 266 root nodes.  Needless to say this spurred many conversations about schema design and the art of creating reusable types.

So this is the first of 5 blog entries talking about schema design patterns.  The last one in the series will be about creating schemas based on the patterns with the BizTalk Editor. 

What I have found is that there are four schools of thought on this subject.  These are the Russian Doll, the Salami Slice, the Venetian Blind and the Garden of Eden approach. 

The main characteristic that differentiates these approaches are whether the elements and types are globally defined.  Elements and types that are global are direct children of the <schema> node.  Elements and types are considered local when they are nested within other elements and types.  If elements or types are global then they are available for reuse by other schemas using either import or include statements.  Locally defined elements and types are not available for reuse.

In the Russian Doll approach, the schema has one single global element – the root element.  All other elements and types are nested progressively deeper giving it the name due to each type fitting into the one above it.  Since the elements in this design are declared locally they will not be reusable through the import or include statements.  This will not change if the elements are namespace qualified or namespace unqualified.

<?xml version=”1.0″ encoding=”UTF-8″?>
<xs:schema xmlns:xs=”http://www.w3.org/2001/XMLSchema” elementFormDefault=”qualified” attributeFormDefault=”unqualified”>
    <xs:element name=”BookInformation”>
        <xs:complexType>
            <xs:sequence>
                <xs:element name=”Title”/>
                <xs:element name=”ISBN”/>
                <xs:element name=”PeopleInvolved”>
                    <xs:complexType>
                        <xs:sequence>
                            <xs:element name=”Author”/>
                            <xs:element name=”Publisher”>
                                <xs:complexType>
                                    <xs:sequence>
                                        <xs:element name=”CompanyName”/>
                                        <xs:element name=”ContactPerson”/>
                                    </xs:sequence>
                                </xs:complexType>
                            </xs:element>
                        </xs:sequence>
                    </xs:complexType>
                </xs:element>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema> 

The advantages of the Russian Doll approach are:  The schema is self contained as it has all of its parts in the schema and does not interact with other schemas.  In as much as it is self contained it is also decoupled.  Since the content of the schema is not visible to other schemas, changes to the schema are decoupled from other schema components.

The disadvantage is that it is not reusable.

This type of approach would be appropriate for use within a single application or for migration of data from legacy systems.

Memory Growth in BizTalk Messaging

by community-syndication | Apr 16, 2005 | BizTalk Community Blogs via Syndication

By Kartik Paramasivam and Raied Malhas

In this document we list various reasons that might lead BizTalk to get into an Out Of Memory situation and then suggest mitigations/solutions for such conditions.

1.  Introduction to BizTalk Hosts:
Before discussing Out of Memory conditions, let us talk about hosts and how adapters are mapped to hosts.

Adapters (receive and send) and orchestrations run under the BizTalk NT Service.   An instance of the BizTalk NT Service is called a host instance. There can also be receive adapters (HTTP/SOAP) which can run under in any other process (e.g. IIS) instead of the BizTalk NT Service. Such adapters are called isolated adapters.
Hosts in BizTalk have 1 or more host instances (process). The default BizTalk installation has 2 hosts.
1) Default host
2) Isolated Host.

Note: You can only have 1 host instance per host on a given server.

The following is the mapping between end points (receive or send port) and hosts:
Send Port -> Send Handler -> Send Host
Receive location -> Receive Handler-> Receive Host

Typically you should create separate receive hosts for receive adapters and send hosts for all send adapters.  Doing this gives each adapter a separate process to live in. This guarantees that one adapter will not adversely affect another. Also if your BizTalk host/process goes Out of Memory, you will know which components are running in that process.

2. The Obvious Suspects for Out of Memory conditions:

When the process goes Out Of Memory (OOM), you need to 1st find out what components are running in that process.  If possible,  follow the recommendations listed above for separating out receive/send adapters and orchestrations into different hosts.  Once you have done that then you can check if one of the following 3 conditions exists in the process that is going OOM.

1) You are executing transforms/maps on relatively large messages in receive/send port or XLANG. The point here is that XSL transforms load the whole message in memory to transform them. 

Solution 1: Decrease the number of messages that your process operates on concurrently (section on Send Host below might give you some idea).
Solution 2: Decrease the size of your XML message that you are trying to transform.

2) You are executing XML receive/send pipeline on a document that contains one or more items such as the following:
– Large attribute values
– Large element values
– Large attribute or element tags.

If one of the above applies to your XML document then that item is fully loaded in memory.

Solution 1: Try to limit the size of the above entities.
Solution 2: If you can’t limit the size, then make sure your process doesn’t concurrently operate on multiple documents. (See sections below on limiting concurrency)

3) You have a custom pipeline component or adapter, which loads the whole document in memory.  Most of the components shipped with BizTalk (except transforms) support streaming as opposed to loading the whole document in memory, and hence have a low memory foot print. However, we have observed that custom pipeline components written by customers may or may not support streaming.  

3. Out of Memory in Send Host:

The Send Host of BizTalk Adapters can run Out of Memory when it is processing a large number of messages (i.e. when the system is under high stress loads). This can cause the Send Host memory utilization to rise rapidly causing the system to get into an Out Of Memory condition. Usually, the larger the messages being processed the faster the memory utilization of BizTalk’s SendHost process will be.

Calculating the Maximum number of messages that will be loaded in-memory by BizTalk’s Send Host process:
Under high stress loads, the BizTalk send host process  will load as many instances of messages into memory as it can until the number of messages exceeds the “HighWatermark”.  By default in BizTalk Server 2004, SP1, this value is 200 for messaging.  Below you’ll learn how to configure this value, more details on Watermark settings are available in the “BizTalk Server Performance Characteristics” document at: http://www.gotdotnet.com/team/wsservers/). So, if you have a Dual processor server that is processing a large number of messages, the BizTalk host will load a maximum of:
[HighWatermark Value] * NumberOfProcessors
= 200 * 2
= 400 messages in-memory

For a Hyper-threaded server:
If you have a hyper-threaded server, then the number of processors perceived by the BizTalk send host doubles the actual number of processors. So if you had a dual processor BizTalk server and if we assume that the default value for the maximum number of message instances processed at once is 200 (so, ”HighWatermark” value is equal to 200), then the BizTalk process hosting the Send Host will load a maximum of:
[HighWatermark Value]*[NumberOfProcessors for a Hyper-threaded machine]
= 200 * [NumberOfProcessors * 2]
= 200 * [2 * 2]
= 200 * 4
= 800 messages in-memory

So, in a case where you have a dual processor hyper-threaded BizTalk server with 2GigaBytes of RAM that’s under high stress, there is a possibility that SendHost will get into an Out of Memory state. The process in this case will be loading 800 messages (as calculated above) in memory. The following graph shows an example of such a case where the memory usage of the send host process grew very rapidly to 1.5 Giga Bytes in less than 15 minutes:

Note: The memory footprints of BizTalk hosts can be monitored using Performance Monitor to view “Private Bytes” counter for the required host (receive, send, etc) instance under the “Process” object.

Note: On a 32 bit box, the process can only grow to around 1.5 GB Max (2 GB in some rare cases) even if you have more physical memory to spare (i.e. 8 GB RAM for e.g.)

How to avoid Out of Memory in the SendHost:
Fortunately, the Out of Memory condition can be avoided by configuring the setting that controls the maximum number of messages that can be loaded into memory concurrently.

To configure the maximum number of message instances, in BizTalk’s Management Database (default name is: “BizTalkMgmtDB”), open the “adm_ServiceClass” table. You’ll see for “Messaging InProcess” row, by default, the value for LowWatermark is set to 100 and for HighWatermark to 200. The HighWatermark value is what dictates the maximum number of message instances that can be loaded in-memory of the Send Host process. Change the Low and High Watermark values to lower numbers, an example of lowered values are:
LowWatermark = 25
HighWatermark = 50

If those values (25-50) still show OOM conditions, you should decrease those values more until you stop hitting the Out of Memory condition.

The following graph shows an example of the same test case that the graph above shows. In the graph below, the Low and HighWatermark values were changed to 25-50, respectively. Please note how the memory remained constant without growing to get into an Out of Memory:

Graph2: An example of a test that avoided hitting an Out of Memory in a SendHost containing FILE Adapter

Note: Decreasing the Watermark values can decrease Performance. So, only decrease those values if you run into an Out of Memory condition.

4. Out of Memory in Receive Hosts:

It is fairly rare to see a host containing a receive adapter to go out of memory. If that happens it is typically due to one of the cases listed in section 2.

If none of the suggested recommendations in Section 2 can be performed, then reducing the number of messages processed concurrently can help protect the receive host from getting into an out of memory condition.  

Note: Remember however that by doing so, we decrease the concurrency and hence the throughput/performance will decrease.

1) Reduce the messaging engine thread pool size:  A user can control the number of threads used by the messaging engine to publish messages into the message box.  By reducing the number of threads used by the messaging engine we reduce the rate at which the receive adapter will receive messages into the message box.  Remember this setting only needs to be done for the host corresponding to the receive handler for the adapter. 

By default the messaging engine creates 10 threads/cpu for
publishing messages.

To specify the messaging thread pool size
1. Click Start, and then click Run.
2. In the Run dialog box, in the Open box, type regedit, and then click OK.
3. In Registry Editor, expand HKEY_LOCAL_MACHINE, expand SYSTEM, expand CurrentControlSet, expand Services, right-click BTSSvc* (you would first need to determine which of the BTSSvc* regkeys correspond to the host corresponding to the receive handler), point to New, click DWORD Value, type MessagingThreadPoolSize, and then press ENTER.
4. In Registry Editor, double-click MessagingThreadPoolSize.
In the Edit DWORD Value dialog box, in the Value data box, type a number between 1 and 30, and then click OK.

5. Note for Custom Adapters:

If you have verified all of the above conditions and if you think your custom receive/send adapter is causing the OOM conditions, then you need to look at the source code of your adapter and verify the following:

The following only applies to Adapters written in Managed code:
The adapters get an object of type IBTTransportBatch using the GetBatch() API on the TransportProxy object. Once the adapter is done using the IBTTransportBatch object the adapter needs to do a Marshal.ReleaseComObject( batch) in a loop to release the object.

Basically the .net GarbageCollector does not kick in in time to release the unmanaged memory.  Hence failure to call ReleaseComObject() will result in what looks like a memory leak. 

Low-latency scenarios that use two-way (solicit-response) HTTP send ports.

by community-syndication | Apr 11, 2005 | BizTalk Community Blogs via Syndication

HttpOutCompleteSize

This setting controls the size of the batch of messages that is returned from the HTTP send adapter.  The default value for this is 5.  If the buffer is not full and there are outstanding responses then the adapter will wait for 1 second until it commits the batch.  For low-latency scenarios this should be set to 1 which will allow the adapter to send response messages immediately to the message box for processing.  This will have the greatest effect during times of low-throughput activity with varied response times from backend systems.

Actually this setting controls the number of messages being returned to BizTalk (EPM) from the HTTP adapter regardless of whether the port is one-way or two-way.  A message that is returned in the batch includes a request to DeleteMessage, MoveToNextTransport, MoveToSuspendedQ, and, most interestingly, SubmitResponseMessage.  Although this setting will improve response times in low-latency scenarios it does increase the amount of chattiness between the adapter and the message box.

It is the send side equivalent to the HttpBatchSize on an HTTP receive adapter.  In low-latency scenarios this is typically set to 1 to ensure that the messages are getting processed as quickly as they are received.

To set this value you need to add HttpOutCompleteSize as a DWORD value to the registry under:

HKLM\SYSTEM\CurrentControlSet\Services\BTSSvc{guid}\

where GUID is the ID of the host for HTTP send handler.

This information is specific to BizTalk Server 2004 and there is no guarantee that this setting will behave the same way in the next release of BizTalk Server.

Disclaimer: This is an undocumented setting and was not formally tested since it was not considered to be publicly available.  You must do your own testing to ensure that the behavior and performance of your system are as expected.