Another BizTalk 2004 Positional Flat File Schema Example

Another BizTalk 2004 Positional Flat File Schema Example

I got got an email from Chirag in Nigeria with the following problem. He had a positional flat file which he needed to parse which looked like this (the ` marks the carrage return and like feed, EOF is the end of file marker remove these to test the file):


0120200300 01                                      THORNTON                                                           `
0120200400 01                                      OGIHARA                                                            `
0120200500 01                                      WILSON                                                             `
0080300101 JENNIFERMRS                                                        `
0080300201 BENJAMINMSTR                                                       EOF


There are two different schemas or ‘Sections’ of data here and they are identified by the first 3 digits at the start of each line 012 and 008. Cirag had written a parser in C# but he was concerned it was too slow and this was not the “BizTalk” way to do things.


1) Create a schema that looks like the one shown below:


 



2) Set the properties on the following nodes. The key properties which make this solution work are the Parser Optimisation, Lookahead Depth for performance the Lookahead Depth must be set to allow the parser to identify the first 3 characters of each record i.e. 012 or 008. This works in turn with the Tag Identifier property which matches different records with different areas in the schema i.e. the 012 records should be parsed into the K20 parent node area and the 008 into the K30 parent node area, this is quite a powerful piece of functionality. The most common use for the Tag Identifier property I have seen is with correctly parsing header and footer records into a schema.


Note you will need the BizTalk 2004 SP1 installed to set the Parser Optimisation, Lookahead Depth and Allow Early Termination properties in the schema properties dialog or you will need to open the schema and set these using notepad or some other editor.


Node                   Property Name                        Property Value
Schema                 Schema Editor Extension              Flat File Extension 
                       Parser Optimisation                  Speed
                       Lookahead Depth                      3
                       Allow Early Termination              No


 


AirlineMessage         Structure                            Delimited
                       Child Delimiter Type                 Hexadecimal
                       Child Delimiter                      0x0D 0x0A
                       Child Order                          Infix


 


K20                    Structure                            Positional
                       Tag Identifier                       012
                       Tag Offset                           0
                       Max Occurs                           *
                       Min Occurs                           0


PLen                   Positional Length                    4


Key2                   Positional Length                    2


Key3                   Positional Length                    2


ChangedFlag            Positional Length                    1


PartyNo                Positional Length                    2


Remark                 Positional Length                    32


SplName                Positional Length                    6


SurName                Positional Length                    67


 


K30                    Structure                            Positional
                       Tag Identifier                       008
                       Tag Offset                           0
                       Max Occurs                           *
                       Min Occurs                           0


PLen                   Positional Length                    4


Key2                   Positional Length                    2


Key3                   Positional Length                    2


ChangedFlag            Positional Length                    1


Name                   Positional Length                    67


 


3) Now you’ve got the schema lets test it there are two easy ways to test the schema either:
a) Right click the schema in the solution explorer choose properties set the entries in the dialog to the following


Now Right click the schema in the solution explorer again and select validate instance. If all goes ok the the output window should display something like the following:


Invoking component…
Validation generated XML output <file:///C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\_SchemaData\TestElement_output.xml
>.
Validate Instance succeeded for schema TestElement.xsd, file: <file:///C:\Project\Artifacts\Biztalk
Projects\UnuniformPositionalFlatFile\Test.pos>.
Component invocation succeeded.
 
 
Double click on the Validation generated XML output link and the xml output result of the file will be  displayed. Then
click in the pane in which the XML is displayed hold down the Ctrl key and press K and D this will format the result xml correctly so you can view it.


b) Call the the Flat File Dissasembly utility FFDASM.exe. the easiest way to do this is to create a bat file with the contents like the following:


c:
cd C:\Program Files\Microsoft BizTalk Server 2004\SDK\Utilities\PipelineTools
ffdasm.exe “C:\path\inputFlatFileName.pos” -bs “C:\path\schemaName.xsd” -c -v
pause..


The output should look like:



 <AirlineMessage xmlns=”http://UnuniformPositionalFlatFile.Test>



   <K20 PLen=”0120 Key2=”20 Key3=”02 ChangedFlag=”0 PartyNo=”0 Remark=”01 SplName=”” SurName=”THORNTON xmlns=”” />


   <K20 PLen=”0120 Key2=”20 Key3=”04 ChangedFlag=”0 PartyNo=”0 Remark=”01 SplName=”” SurName=”OGIHARA xmlns=”” />


   <K20 PLen=”0120 Key2=”20 Key3=”05 ChangedFlag=”0 PartyNo=”0 Remark=”01 SplName=”” SurName=”WILSON xmlns=”” />


   <K30 PLen=”0080 Key2=”30 Key3=”01 ChangedFlag=”0 Name=”JENNIFERMRS xmlns=”” />


   <K30 PLen=”0080 Key2=”30 Key3=”02 ChangedFlag=”0 Name=”BENJAMINMSTR xmlns=”” />

 <AirlineMessage>

 

 R. Addis

 

 

NAck Message Processing Orchestration for BizTalk Server 2004

NAcks or Negative Acknowledgments are small SOAP messages that exist in BizTalk when a messaging error occurs.  This happens when adapters fail, pipelines fail, messages do not match subscriptions, ect.



These messages can be subscribed to and a business process can be written to react to all of these messages or specific error codes. 



This sample Orchestration shows how simple it is to catch these messages.  To subscribe to a NAck, just create a subscription to: BTS.AckType = NACK.  It’s that easy!



Download: NAck Catching Sample



Setup: Download and extract the solution.  Create a receive port that causes a routing failure.  Drop a message and watch for the output in the Out folder.



For a routing failure NAck, the following properties are available in the message context and could be accessed inside the Orchestration:


AckDescription


AckFailure Category


AckFailure Code


AckID


AckInboundTransportLocation


AckOwnerID


AckReceivePortID


AckReceivePortName


AckType



With this information, logic could be included inside the Orchestration to extract / reprocess a message or log the error to an additional notification facility (other then the event log since it would already be logged there).

The Vocabulary: after thought for API?

The Vocabulary: after thought for API?


The more I work with the Rules API the more I’m convinced that the Vocabulary element was only created for the BRC. The most generous I can be is that someone came up with a good idea for the BRC and “retrofitted” it for the API. There are two reasons for this;



  1. The API is unsupported and undocumented.
  2. The BRL (Business Rules Language) can function happily without it – in fact a rule can reference directly the document element / .NET method etc.

If you take a look at the following two bites from the exported rule store, UI think you’ll see why I come to this conclusion;


The following is in effect a definition of a string literal in the export vocabulary


  <vocabularydefinition id=”54e58c56-e279-4729-b0d4-2e7bfbef989e” name=”TransferControlAccount” description=””>


      <literaldefinition type=”string”>


        <string>222222</string>


      </literaldefinition>


      <formatstring language=”en-US” string=” Transfer Control Account” />


   </vocabularydefinition>









The following is taken from an extract of an action. This interesting thing to note is that the literal value “222222” is also specified, along side the link to the vocab entry specifying it;



        <function>


          <vocabularylink uri=”2e2bb376-b4ac-4d42-9573-cdf42088f681″


element=”453cb556-e96e-4e10-b456-96bf6d1313c6″ />


          <classmember classref=”Accounting” member=”AddAccountingEntries” sideeffects=”true”>


            <argument>


              <constant>


                <vocabularylink uri=”2e2bb376-b4ac-4d42-9573-cdf42088f681″


element=”54e58c56-e279-4729-b0d4-2e7bfbef989e” />


                <string>222222</string>


              </constant>


            </argument>


          </classmember>


        </function>






So why would this redundant data be here? Performance? I doubt it. The whole “immutability” of Rules and Vocabularies makes sense from a version management perspective until you actually start using them. Imagine the scenario; You have defined your Vocabulary before starting your rules (This is required because you can’t reference a Vocabulary from your rules until that Vocabulary has been published – this in itself is predicated on knowing all the terms up front – big assumption!). You define 25+ rules, using multiple references to your vocabulary. Fine. You now publish the rule policy. At this point the world is still in order – version 1.0 of both rules and vocabulary are in a lotus type harmony. But what happens when you want to extend or amend the vocabulary? You create a new version of the Vocabulary (cut & paste!),make your amendments and publish it. Of course NONE of your rules will reference this new vocabulary (1.1) as all links reference 1.0 of the vocabulary. The only way to reference the new vocabulary is to create a new version of your rules (cut & paste!), and edit EACH AND EVERY reference to the vocabulary to repoint to the new version.


This isn’t new of course, and tools do exist to deal with it.


The BRC has been criticised for this “cut & paste” mentality, but in fact some of the problem goes back to the design of the rules system itself.

Symbolic Debugging for Orchestrations…

Symbolic Debugging for Orchestrations…

As has been noted before in the lore of BizTalk, it sure would be useful to use a
real debugger with orchestrations – at least occasionally.

There are times when an expression shape winds up getting a bit sticky (not that I
would know…) and a debugger would be just the ticket.  Other times, the exception
you are getting from the orchestration engine isn’t at all clear.

Still other times, seeing the actual contents of messages or context as you step through
would be interesting.

Jon has posted
on debugging orchestrations (in IL) with an ILDASM/ILASM
loop
, and had also discussed building your orchestration assemblies manually (using
xsharpp.exe).  (He correctly noted these approaches wouldn’t generate anything
supportable, but they work for spelunking.) 

I wanted to suggest something a bit different – I wanted to go back instead to the
venerable BTS File Dump utility that Charles
released
in May of 2004 (before the GenerateCSFiles registry key was
discovered
…) and propose a different technique for symbolic debugging.

First, install Charles’ BizTalk File Dump utility – you can download it from here
Fire it up, and change the output path in the utility to something easy, like c:\temp\BTSFileDump.

Here is the Edit/Debug cycle…(Start by making sure the file dump utility is running,
and click the “Start dumping files” button.)

  1. Build your orchestration project (or whole solution if need be.)

  2. Do a Ctrl-Shift-F (Find in Files) in Visual Studio, and change the “Look in:”
    folder to c:\temp\BTSFileDump

  3. Search for something in one of your expression shapes, say, “MyClass.Execute”. 
    The correct generated file (that the file dump utility grabbed) will appear in your
    Find Results – open it up, and set a breakpoint.  (Not on the xml designer portion
    – on the actual code!)

  4. From the Debug menu, choose Processes and attach to BTSNTSvc.exe.  (Have more
    than one
    ?)  Choose CLR debugging only – not native.  The symbols should
    be loaded automatically – no need to copy PDBs to the GAC.

  5. Trigger your orchestration however you normally would.  Bask in a picture like
    this:  (Puts a Petzold-style WindowProc to shame…) 


    Debugview
  6. Use QuickWatch to examine message contents/context, if you like – you’ll be interested
    in expansions like
    this one
    (where sampleRequest is a message in the orchestration.)

  7. Find your problem, Debug-Detach All, and fix the problem in the orchestration.

  8. Click “Delete all files in output path” in the file dump utility (to avoid duplicates)
    and rebuild.  Repeat the process if you need to.  (To save time, consider
    an external tool that just re-GACs the orchestration assembly and resets the host
    process.  See here.)

Note that this technique will work in a production setting.  You could copy the
PDBs and sources to the production server, and use DbgClr.exe (in the Framework
SDK) or cordbg.exe to attach to the appropriate
host process
.

Happy debugging…

Symbolic Debugging for Orchestrations…

Symbolic Debugging for Orchestrations…

As has been noted before in the lore of BizTalk, it sure would be useful to use a
real debugger with orchestrations – at least occasionally.

There are times when an expression shape winds up getting a bit sticky (not that I
would know…) and a debugger would be just the ticket.  Other times, the exception
you are getting from the orchestration engine isn’t at all clear.

Still other times, seeing the actual contents of messages or context as you step through
would be interesting.

Jon has posted
on debugging orchestrations (in IL) with an ILDASM/ILASM
loop
, and had also discussed building your orchestration assemblies manually (using
xsharpp.exe).  (He correctly noted these approaches wouldn’t generate anything
supportable, but they work for spelunking.) 

I wanted to suggest something a bit different – I wanted to go back instead to the
venerable BTS File Dump utility that Charles
released
in May of 2004 (before the GenerateCSFiles registry key was
discovered
…) and propose a different technique for symbolic debugging.

First, install Charles’ BizTalk File Dump utility – you can download it from here
Fire it up, and change the output path in the utility to something easy, like c:\temp\BTSFileDump.

Here is the Edit/Debug cycle…(Start by making sure the file dump utility is running,
and click the “Start dumping files” button.)

  1. Build your orchestration project (or whole solution if need be.)

  2. Do a Ctrl-Shift-F (Find in Files) in Visual Studio, and change the “Look in:”
    folder to c:\temp\BTSFileDump

  3. Search for something in one of your expression shapes, say, “MyClass.Execute”. 
    The correct generated file (that the file dump utility grabbed) will appear in your
    Find Results – open it up, and set a breakpoint.  (Not on the xml designer portion
    – on the actual code!)

  4. From the Debug menu, choose Processes and attach to BTSNTSvc.exe.  (Have more
    than one
    ?)  Choose CLR debugging only – not native.  The symbols should
    be loaded automatically – no need to copy PDBs to the GAC.

  5. Trigger your orchestration however you normally would.  Bask in a picture like
    this:  (Puts a Petzold-style WindowProc to shame…) 


    Debugview
  6. Use QuickWatch to examine message contents/context, if you like – you’ll be interested
    in expansions like
    this one
    (where sampleRequest is a message in the orchestration.)

  7. Find your problem, Debug-Detach All, and fix the problem in the orchestration.

  8. Click “Delete all files in output path” in the file dump utility (to avoid duplicates)
    and rebuild.  Repeat the process if you need to.  (To save time, consider
    an external tool that just re-GACs the orchestration assembly and resets the host
    process.  See here.)

Note that this technique will work in a production setting.  You could copy the
PDBs and sources to the production server, and use DbgClr.exe (in the Framework
SDK) or cordbg.exe to attach to the appropriate
host process
.

Happy debugging…

Tech-Ed Pattern Presentation

Brennan and I are still planning to present on patterns at tech-ed, titled: %u00e2%u0080%u009cImplementing Real World Patterns with BizTalk 2004.%u00e2%u0080%u009d Stay tuned, as we will post a general overview of what we will be discussing, as well as links that will be relevant to our discussion. Once Tech-Ed is complete, we will be posting the full code and PowerPoint presentation(s). We are looking forward to seeing you there….

BizTalk native adapters – soon to be better than ever!

I am so glad to see the updates to the native adapters in BizTalk 2006. Scott Woodgate posted a document on the changes here. (you might need to join the BizTalkServerStuff group if this link doesn’t work) I was lucky enough to spend some time with the adapter framework over the past few months, and the behavior of the native ones with 2004 left me scratching my head a few times. Anyhow, I really like the Adapter Framework despite not liking the examples built on it that are included with BizTalk. I’ve also been surprised that the community hasn’t strucken out and developed a bit more. There’s the potential for great extensibility, and it’s not too complicated once one takes a couple weeks to get oriented. The SDK sample adapter with 2004 is way overbuilt in my opinion. Plus it uses the SystemMessageContext class which is on the list of unsupported classes. I honestly had enough with the SDK samples once I found that guy being used, so I don’t know if there are any more or not. Anyhow, kudos to the BTS team for spending some time on these! I think they’ll make a bigger splash than most people anticipate….

Chapter 10 – Improving Web Services Performance (Patterns and Practices) New

Chapter 10 – Improving Web Services Performance (Patterns and Practices) New

Once you’ve got your SOA all sorted – IIS can run out of available connections pretty quick due to things like Web Services calling other WebServices on the same box using up valuable connections.


Here’s a great MSDN Article that gives you the tech. details so your WebServices will hum.


http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/scalenetchapt10.asp

SQL Adapter Wizard Query twice

SQL Adapter Wizard Query twice

When u want to receive XML data from your stored procedure which containts a SELECT and UPDATE (processed records) statement, you can have some problems to Generate the SQL Adapter XSD.


When u press the last Window in the Wizard you could get an error message Failed to execute queury.


What we saw in the SQL Profiler is that the Generate SQL Adapter Wizard Executes twice the stored procedure Oops. The first one from Application name SQLClient provider and the second one from Visual Studio .NET 2003.


The first query execute returns the XMLDATA (XML Schema) , but also the stored procedure update. The second run executes the stored procedure once again without your XMLData schema.


So disable your update in your stored procedure and generate your XML schema after that enable the update and delete the XMLData option.


Good to know that the Wizard runs the stored procedure twice……

IRuleSetTrackingInterceptor: tracing rules execution

IRuleSetTrackingInterceptor: tracing rules execution

I don’t know about you but I find the best way to understand how something is working is from reading the code and then stepping through its execution, with VS’s trusty local and immediate windows. A friend of mine calls this  “developers documentation”, and whilst I certainly wouldn’t go as far as saying it’s a substitute for good docs (and by that Chris, I mean “living models”!), I always end up doing it during any maintenance code cycles (unintended consequences and all that).


 


Unfortunately the execution of a BizTalk rule policy isn’t quite as straight forward. It’s not a set of simple sequential steps that you can step through. If it was it would probably be far less useful as a rules engine – and certainly far less performant.. Instead it implements a RETE algorithm for small rulesets and some propriety one for large rules sets according to the man himself. Anyway the long and the short of this is that rules processing is done behind closed doors, which makes for a tough time in understanding what happened during the policy being executed. I’m sure a good understanding of the RETE process would help – Dummies guide to RETE anyone?


 


The Microsoft Business Rules Composer (BRC) does allow rule “testing” by implementing the Tracking Interceptor DebugTrackingInterceptor. The following is a sample output produced by the BRC for the Loans Processing policy (from the Loans Sample in the SDK. 


 


RULE ENGINE TRACE for RULESET: LoanProcessing 5/19/2005 12:46:13 PM


 


FACT ACTIVITY 5/19/2005 12:46:13 PM


Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e


Ruleset Name: LoanProcessing


Operation: Assert


Object Type: DataConnection:Northwind:CustInfo


Object Instance Identifier: 782


 


FACT ACTIVITY 5/19/2005 12:46:13 PM


Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e


Ruleset Name: LoanProcessing


Operation: Assert


Object Type: TypedXmlDocument:Microsoft.Samples.BizTalk.LoansProcessor.Case


Object Instance Identifier: 778


 


FACT ACTIVITY 5/19/2005 12:46:13 PM


Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e


Ruleset Name: LoanProcessing


Operation: Assert


Object Type: TypedXmlDocument:Microsoft.Samples.BizTalk.LoansProcessor.Case:Root


Object Instance Identifier: 777


 


CONDITION EVALUATION TEST (MATCH) 5/19/2005 12:46:13 PM


Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e


Ruleset Name: LoanProcessing


Test Expression: NOT(TypedXmlDocument:Microsoft.Samples.BizTalk.LoansProcessor.Case:Root.Income/BasicSalary > 0)


Left Operand Value: 12


Right Operand Value: 0


Test Result: False


 


CONDITION EVALUATION TEST (MATCH) 5/19/2005 12:46:13 PM


Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e


Ruleset Name: LoanProcessing


Test Expression: NOT(TypedXmlDocument:Microsoft.Samples.BizTalk.LoansProcessor.Case:Root.Income/OtherIncome > 0)


Left Operand Value: 10


Right Operand Value: 0


Test Result: False


 


CONDITION EVALUATION TEST (MATCH) 5/19/2005 12:46:13 PM


Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e


Ruleset Name: LoanProcessing


Test Expression: TypedXmlDocument:Microsoft.Samples.BizTalk.LoansProcessor.Case:Root.PlaceOfResidence/TimeInMonths >= 3


Left Operand Value: 15


Right Operand Value: 3


Test Result: True


[.. cut for brevity ..]


 


This implements the IRuleSetTrackingInterceptor interface which you can use to produce you own custom debugging/tracing of rule processing.


 


Most of the information is there (although I’m still troubled that the tracing of the process doesn’t allow you to examine the facts i.e., the data rows, xml docs etc), but it’s not exactly easy to see what’s happened. Understanding the match-conflict resolution-action helps considerably.


 


If you are executing your policy outside of BizTalk, or in a component consumed within BizTalk you can specify an alternative IRuleSetTrackingInterceptor. This has the advantage of allowing you to step through the rule processing if you wish, and also allows you to view fact details (through the facts you pass to the policy). The following code demonstrates how to invoke your MyInterceptorClass().


 


xmlDocument = IncomingXMLMessage.XMLCase;


typedXmlDocument = new Microsoft.RuleEngine.TypedXmlDocument(“Microsoft.Samples.BizTalk.LoansProcessor.Case”,xmlDocument);


policy = new Microsoft.RuleEngine.Policy(“LoanProcessing”);


policy.Execute(typedXmlDocument,new MyInterceptorClass());


OutgoingXMLMessage.XMLCase = xmlDocument;


policy.Dispose();


 


Once you understand the various stages to the rule engine, you soon yearn for a better way to visualise a complex processing of rules. To this end I started thinking about how I’d like to see what was happening. In the past I’ve used UML sequence diagrams to demonstrate how messages are sent between object and also object lifetime.


 


Initially , I though that each fact would “constructed” by assertion and “destructed” by retraction. The same would happen with the rules placed into and out of the Agenda. This is sort of how it would look;


 


 


The problem with this approach is that it makes a very complex picture when many facts are asserted and many rules get passed onto the agenda. In the end I settled for a “swim lanes” type approach, separating each phase (facts asserting/retracting, condition matching, adding to agenda and firing actions) in a lane and then just processing sequentially. I think it works quite well, but you can see the loans processing sample output here.


 


 


The TrackingInterceptor also outputs to XML. If anyone’s interested in a copy of the source/component you can drop me a note here.


 


After my experiences I have the following request for MS


 


Request 1: Allow the BRC to use a difference Tracking Interceptor


Request 2: Allow the Tracking Interceptor to get hold of the full facts – i.e., let me dump out the xmldoc.


Request 3: Allow the “Call Rules” shape to specify a tracking interceptor