Move along, nothing to see

I suppose my first entry in this blog should be some insightful post about the BizTalk Rules Engine (BRE). Sorry, move along please, nothing to see.


 


You see, I want to record my journey through a whole new technology. The BRE is that technology. I’m lucky – I’ve been given the opportunity to specialise in one facet of a BizTalk, and I want to share that experience – even the good, bad and ugly times.

Receiving RawStrings…

(See update below)

If you have attempted to send email from BizTalk, you’ve gotten pretty comfortable
with this documentation
topic.

It discusses how to use a sample class (Microsoft.XLANGs.CustomFormattersSDK.RawString)
and associated custom formatter to deal with messages that contain raw string content. 
As the docs say, “Sending a message of type System.String will not work, because
the string gets formatted as an XML document in the message, which is not your desired
solution.”  (In other words, whether you send or receive, using System.String
as a message type means you are dealing with a wrapping tag.)

However, though the docs cover sending RawStrings, it is less clear how you
can use the RawString class when you want to receive messages as such.  Just
typing the port operation as RawString will yield a UnexpectedMessageTypeException,
because none of the default receive pipelines will set the MessageType context property
on inbound messages to be RawString – and this type matching is what orchestrations
expect.

We need a custom receive pipeline that will set the MessageType context property equal
to the fully qualified type name for RawString.  Jon’s discussion of receiving
binary messages
provides some background here, and Jon’s ContextAdder pipeline
component can make the MessageType context property assignment very easy.

Here are the steps required to receive RawStrings:

  1. Download & build Jon Flanders’ ContextAdder pipeline
    component, and make sure the build outputs end up in your Microsoft BizTalk Server
    2004\Pipeline Components directory.
  2. Build this
    code
    (the RawString class & custom formatter) into a separate C# library (with
    a strong name) and deploy it to the GAC (if you haven’t already when dealing
    with email.)
  3. Use the RawString type on the orchestration port operation that you are trying to
    receive RawStrings on…

  4. Create a receive pipeline (ReceiveRawString), and add the ContextAdder pipeline component
    in the Validate stage.  (You will only ever need one such pipeline around –
    put it in a common library.)  Configure the properties collection with the properties
    for RawString, as shown below (“Value” property will be the full five-part
    assembly-qualified type name):

  5. Specify the ReceiveRawString pipeline for the associated physical port.
  6. Now you will have full access to the raw string content, and can do whatever you need
    to do…You might want to declare an orchestration variable of type RawString,
    and assign it to the RawString message so you can access properties/methods like ToString(),
    which returns the underlying string content.

    >

    Update:

    Greg Van de Wiele has pointed
    out the other way to go about this (and Jon has talked
    about this approach in his recent posts as well.)  It can be less work
    in that no custom pipeline is required (a one time task) – though perhaps a little
    harder to explain to the “next guy“…you can decide!

    It goes like this:

    1. You will need to do step 2 from above.
    2. Use the XmlDocument type on the orchestration port operation that you are trying to
      receive raw string content on, and use a PassThruReceive pipeline.
    3. Create a multipart message type, with a single part that is of type RawString. 
      Also, declare an instance of this type, say, tempRawStringMsg.
    4. Receive the message from the port in step 1 into an XmlDocument-typed message (say,
      xmlDocIn)
    5. Drag out a message assignment shape.  In the expression, do: tempRawStringMsg.rawString
      = xmlDocIn;
    6. As before, you might want to declare an orchestration variable (not a message) of
      type RawString, and assign it to tempRawStringMsg.rawString so you can access properties/methods
      like ToString(), which returns the underlying string content.

    Note that not wrapping the RawString in a multipart message doesn’t work
    – you will get a compiler error indicating that assignments can’t be made between
    XmlDocument and RawString types.  Also note that introducing an intermediate
    message has a performance cost.

Receiving RawStrings…

(See update below)

If you have attempted to send email from BizTalk, you’ve gotten pretty comfortable
with this documentation
topic.

It discusses how to use a sample class (Microsoft.XLANGs.CustomFormattersSDK.RawString)
and associated custom formatter to deal with messages that contain raw string content. 
As the docs say, “Sending a message of type System.String will not work, because
the string gets formatted as an XML document in the message, which is not your desired
solution.”  (In other words, whether you send or receive, using System.String
as a message type means you are dealing with a wrapping tag.)

However, though the docs cover sending RawStrings, it is less clear how you
can use the RawString class when you want to receive messages as such.  Just
typing the port operation as RawString will yield a UnexpectedMessageTypeException,
because none of the default receive pipelines will set the MessageType context property
on inbound messages to be RawString – and this type matching is what orchestrations
expect.

We need a custom receive pipeline that will set the MessageType context property equal
to the fully qualified type name for RawString.  Jon’s discussion of receiving
binary messages
provides some background here, and Jon’s ContextAdder pipeline
component can make the MessageType context property assignment very easy.

Here are the steps required to receive RawStrings:

  1. Download & build Jon Flanders’ ContextAdder pipeline
    component, and make sure the build outputs end up in your Microsoft BizTalk Server
    2004\Pipeline Components directory.
  2. Build this
    code
    (the RawString class & custom formatter) into a separate C# library (with
    a strong name) and deploy it to the GAC (if you haven’t already when dealing
    with email.)
  3. Use the RawString type on the orchestration port operation that you are trying to
    receive RawStrings on…

  4. Create a receive pipeline (ReceiveRawString), and add the ContextAdder pipeline component
    in the Validate stage.  (You will only ever need one such pipeline around –
    put it in a common library.)  Configure the properties collection with the properties
    for RawString, as shown below (“Value” property will be the full five-part
    assembly-qualified type name):

  5. Specify the ReceiveRawString pipeline for the associated physical port.
  6. Now you will have full access to the raw string content, and can do whatever you need
    to do…You might want to declare an orchestration variable of type RawString,
    and assign it to the RawString message so you can access properties/methods like ToString(),
    which returns the underlying string content.

    >

    Update:

    Greg Van de Wiele has pointed
    out the other way to go about this (and Jon has talked
    about this approach in his recent posts as well.)  It can be less work
    in that no custom pipeline is required (a one time task) – though perhaps a little
    harder to explain to the “next guy“…you can decide!

    It goes like this:

    1. You will need to do step 2 from above.
    2. Use the XmlDocument type on the orchestration port operation that you are trying to
      receive raw string content on, and use a PassThruReceive pipeline.
    3. Create a multipart message type, with a single part that is of type RawString. 
      Also, declare an instance of this type, say, tempRawStringMsg.
    4. Receive the message from the port in step 1 into an XmlDocument-typed message (say,
      xmlDocIn)
    5. Drag out a message assignment shape.  In the expression, do: tempRawStringMsg.rawString
      = xmlDocIn;
    6. As before, you might want to declare an orchestration variable (not a message) of
      type RawString, and assign it to tempRawStringMsg.rawString so you can access properties/methods
      like ToString(), which returns the underlying string content.

    Note that not wrapping the RawString in a multipart message doesn’t work
    – you will get a compiler error indicating that assignments can’t be made between
    XmlDocument and RawString types.  Also note that introducing an intermediate
    message has a performance cost.

Question: Secure the access or secure the content?

Abstract: this seems an easy question, but I have not found a proper answer yet feel free to give your answer and a little justification.


The complete question is:
What should I secure? the access to the resources, or the content of the resources?


Securing the access means controlling who sees the resources, in terms of who can read files, databases, etc.
Securing the content means encrypting the file content or the data inside databases, so everybody can read but only a few can understand.


Meanwhile you think a response,
– There is an application, let’s say, and ASP.NET application.
– Then there is IIS authentication, let’s say, Windows Integrated Authentication.
– Then there is .NET Framework Code Access Security settings, let’s say, system administrators configuring execution permissions to the assembly.
– Then there is more Code Access Security (developer’s), so the .NET assembly asks declaratively for a read permission to a resource file.
– Then there is the resource file, encrypted, of course, to store a Connection String.
– Then there is a connection to a database engine, with user/password challenge for authentication.
– Then there is database authorization, give access to a concrete database object.
– Then there is a query, that returns an encrypted column (lovely Yukon).
– Then there is a database-level encryption user key, using a password or passphrase provided by the user, so the executing assembly can read the column data.
– So the column data goes in clear to the assembly, which returns the information to the IIS to be returned to the human-being at the other side of the network. Of course IIS uses a HTTPS connection, beware of hackers


back to que question… Encrypt files or protect them from being read? or both? most important: why?

Knowing which BTSNTSvc.exe to attach to…

Andy Morrison wrote
about a technique for identifying the correct host instance to attach to when
debugging components associated with orchestrations.

A potentially easier solution is to keep perfmon running with the counter shown below
(BizTalk:Messaging/ID Process – all instances.)  The counter values will update
every time a host instance recycles.  The Debug-Processes dialog in Visual Studio
will show you (and allow you to sort by) process IDs to make this easy. 

Then put PerfMon in “report view” to easily see process IDs by host instance:

Knowing which BTSNTSvc.exe to attach to…

Andy Morrison wrote
about a technique for identifying the correct host instance to attach to when
debugging components associated with orchestrations.

A potentially easier solution is to keep perfmon running with the counter shown below
(BizTalk:Messaging/ID Process – all instances.)  The counter values will update
every time a host instance recycles.  The Debug-Processes dialog in Visual Studio
will show you (and allow you to sort by) process IDs to make this easy. 

Then put PerfMon in “report view” to easily see process IDs by host instance:

Messages from Scratch with Embedded Resources

There have been several folks who have discussed how to create messages “from scratch”
within an Orchestration context – you can read Matt’s
thoughts and check out a BizTalk documentation
excerpt
. This is a common question, and I had an additional technique I thought
I would share… 

Background: When you have a schema that has many promoted properties (or distinguished
fields), or many elements that can be set via xpath expressions easily, it can be
useful to simply start with a “template” document instance and populate the element
content that you are interested in. 

In this situation, you will often have a “Message Assignment” shape that looks something
like this:

xmlDoc.LoadXml("<ns0:BizTalkSampleS3 
      xmlns:ns0="http://BizTalkSample.Schemas.BizTalkSampleS3">
      <SomeElement></SomeElement></ns0:BizTalkSampleS3>");
someMsg = xmlDoc;
someMsg.SomeElement = "some content";
// (or xpath(someMsg,someElementXPath) = "some content" if we don't have a 
// distinguished field.)
   

One disadvantage of loading up “template” xml documents from either expression shapes
or code (via XmlDocument.LoadXml) is that those xml fragments can get easily “lost”,
and are hard to update early in the development cycle when schemas may still be in
flux.  Loading the template files from the file system is problematic because
the question arises “where should I store these files, so that I can find them in
any environment I deploy to?” (Solvable, but a hassle.)

Instead, why not embed the template xml documents as assembly resources?  For
those unfamiliar with that process, I have a short tutorial here (& a helper class.) 

  1. You will need a C# project as part of your overall BizTalk solution.  Place your
    template xml file(s) in the directory corresponding to this project, and add them
    as an “existing item” to the project. 
  2. Select this file within the Solution Explorer, and within the Properties window, select
    “Embedded Resource” as the “Build Action” as shown here:

     

  3. Place this
    class (text here)
    within the same C# project that houses the resources you have added.
  4. To construct a message, drag out a “Message Assignment” shape, and within the associated
    expression write some code like the following.  Simply pass the file name of
    the template document as an argument to GetXmlDocResource (or GetStringResource.)

    sampleUsingTemplate = 
       BizTalkSample.Components.EmbeddedResourceCache.
          GetXmlDocResource("BizTalkSampleS3_output.xml");
       // Populate the "rest" of the message with distinguished fields, promoted 
       // properties, xpath expressions, etc.
       sampleUsingTemplate.SomeElement = "foo";
                

The class I have supplied will cache the loaded resources in a hashtable for performance
sake, and allow you to load resources as both strings and XmlDocuments.

A last thought: Many people ask, “Why can’t I just create a message using a new operator
or a default constructor of some sort?” Well, because few XSD schemas sufficiently
constrain the set of valid instance documents enough for that to be useful – what
form would a “default message” take? (Would it have the optional elements you need?
Some elements that you don’t want?)

Enjoy – feedback appreciated!

BizTalk Server 2004 and SQL Server 2000 SP4

I found a post on the BizTalk Discussion Groups about SQL Server 2000 SP4 in combination with BizTalk Server 2004
Here his post :

Installed SQL Server 2000 SP4 on two development servers over the weekend
with no problems and on two production server on Monday with only slight
problems during the Analysis Server SP4 installation. The problem (bug) has
been reported to MS and they’re issuing a KB article. It’s related to the
MDAC update on servers running MOM2005.

I’m still here…..

So, I’m looking at my blog today, and I see I haven’t posted anything in 2 months and a day.  Shameful!!!  Anyway, I’m climbing back in the saddle.


In what little free time I had before, I was working on a whitepaper for MSDN for the BizTalk Adapter for SQL Server.  I’m glad to say the paper is complete, and should be on the MSDN site very shortly.  I will post some examples from the paper here very shortly.


The question came up at work today about debatching in Orchestrations.  This is a very common scenario in business processing, or at least the issue of dealing with batch style messages is.  Stephen Thomas has done a great job in writing up some examples of how to deal with this.  So good, that I won’t bother to recap.  Anyway, there are a couple of important things to keep in mind when considering your options.



  1. Is it important that “good“ messages must be processesed, and “bad“ messages be dealt with?

  2. Are the file sizes significant?

  3. Does the service level agreement have specific time constraints?

  4. Does the trailer and header (if flat file messages) of the message have to be dealt with?

Many times you can’t let one single bad record cause an entire file not to be processed.  If this is the case, you will have to deal with your messages in another fashion.  One option that is common is to debatch, or “shred“ your messages in the orchestration using XPATH.  You will notice from Stephen’s examples that when using XPATH to debatch messages inside your orchestration, peformance starts to degrade as the file size grows.  I suspect what is happening is that as the XPATH iteration gets deeper in the nest, it is still starting at the beginning of the file, so it slows over time.


If you have larger files and have a time constraint, there are still a few options available to you.  Since XPATH seems to be slow with large files due to it’s apparent scanning of the entire message each time, you may want to try Streaming XPATH.  This option has all the benefits of XPATH, whithout the negative side effects.  Look at this information to see if it might work for you.


There are still a couple of other options that may be available to you.  I will start to post some more thoughts and examples on this topic in the next several weeks.


Cheers,


Todd

DBNETLIB Errors (Network Errors) after adding Multiple Publication MessageBoxes BizTalk Server 2004 SP1 / Microsoft SQL Server 2000 SP3

At one of our customers we installed a complete new BizTalk infrastructure to improve the performance (throughput).


The following machines are involved:


 1 x SQL Server dedicated to master message box processing wit publication disabled, single default instance. 
      Software
      Microsoft SQL Server 2000 with SP3a and BizTalk SQL Hotfix SQL2000-KB810185-8.00.0878-ENU.exe
      Microsoft Windows Server 2003 SP1
      Database Instance BizTalkMsgBoxDb


 2 x SQL Servers dedicated to message box publication, each 1 named instance.
      Software
      2 x Microsoft SQL Server 2000 with SP3a and BizTalk SQL Hotfix SQL2000-KB810185-8.00.0878-NU.exe
      Microsoft Windows Server 2003 SP1
      Database Instance BizTalkMsgBoxPubDb1 , BizTalkMngmtDb , EDIDb , SSO , etc
      Database Instance BizTalkMsgBoxPubDb2 , BizTalkDTADb


 1 x BizTalk 2004 Server Machine DL380 3 GB of RAM


      Software
      Microsoft BizTalk Enterprise Edition 2004 SP1
      Microsoft Windows Server 2003 SP1
      MDAC 2.8
      MSXML / SQLXML


When BizTalk uses one singe MessageBox on the Master Message Box Machine there are no errors generated, BizTalk works fine. When we enable the one or two extra Publication MessageBoxes (via the BizTalk Server Administration Console), the BizTalk Engine starts with Network Errors. 



“An attempt to connect to “BizTalkMsgBoxPubDb2“ SQL Server Database on Server “<SERVERNAME>“ failed with error: “[DBNETLIB][ConnectionWrite(send())]. General network error. Check your network documentation.“.


This has something to do with ACK’s and FIN’s I read in some discussion groups.


What I saw on the SQL Server where the Publication Message Box is configured were many SQL Server Locks.
BizTalk Server 2004 Engine starts up the connection with his own message boxes one minute later.


Any idea how we can fix this network problem or anyone got an idea ?


Many thanks for your input.