by community-syndication | May 30, 2005 | BizTalk Community Blogs via Syndication
I got got an email from Chirag in Nigeria with the following problem. He had a positional flat file which he needed to parse which looked like this (the ` marks the carrage return and like feed, EOF is the end of file marker remove these to test the file):
0120200300 01 THORNTON `
0120200400 01 OGIHARA `
0120200500 01 WILSON `
0080300101 JENNIFERMRS `
0080300201 BENJAMINMSTR EOF
There are two different schemas or ‘Sections’ of data here and they are identified by the first 3 digits at the start of each line 012 and 008. Cirag had written a parser in C# but he was concerned it was too slow and this was not the “BizTalk” way to do things.
1) Create a schema that looks like the one shown below:
2) Set the properties on the following nodes. The key properties which make this solution work are the Parser Optimisation, Lookahead Depth for performance the Lookahead Depth must be set to allow the parser to identify the first 3 characters of each record i.e. 012 or 008. This works in turn with the Tag Identifier property which matches different records with different areas in the schema i.e. the 012 records should be parsed into the K20 parent node area and the 008 into the K30 parent node area, this is quite a powerful piece of functionality. The most common use for the Tag Identifier property I have seen is with correctly parsing header and footer records into a schema.
Note you will need the BizTalk 2004 SP1 installed to set the Parser Optimisation, Lookahead Depth and Allow Early Termination properties in the schema properties dialog or you will need to open the schema and set these using notepad or some other editor.
Node Property Name Property Value
Schema Schema Editor Extension Flat File Extension
Parser Optimisation Speed
Lookahead Depth 3
Allow Early Termination No
AirlineMessage Structure Delimited
Child Delimiter Type Hexadecimal
Child Delimiter 0x0D 0x0A
Child Order Infix
K20 Structure Positional
Tag Identifier 012
Tag Offset 0
Max Occurs *
Min Occurs 0
PLen Positional Length 4
Key2 Positional Length 2
Key3 Positional Length 2
ChangedFlag Positional Length 1
PartyNo Positional Length 2
Remark Positional Length 32
SplName Positional Length 6
SurName Positional Length 67
K30 Structure Positional
Tag Identifier 008
Tag Offset 0
Max Occurs *
Min Occurs 0
PLen Positional Length 4
Key2 Positional Length 2
Key3 Positional Length 2
ChangedFlag Positional Length 1
Name Positional Length 67
3) Now you’ve got the schema lets test it there are two easy ways to test the schema either:
a) Right click the schema in the solution explorer choose properties set the entries in the dialog to the following
Now Right click the schema in the solution explorer again and select validate instance. If all goes ok the the output window should display something like the following:
Invoking component…
Validation generated XML output <file:///C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\_SchemaData\TestElement_output.xml>.
Validate Instance succeeded for schema TestElement.xsd, file: <file:///C:\Project\Artifacts\Biztalk Projects\UnuniformPositionalFlatFile\Test.pos>.
Component invocation succeeded.
Double click on the Validation generated XML output link and the xml output result of the file will be displayed. Then click in the pane in which the XML is displayed hold down the Ctrl key and press K and D this will format the result xml correctly so you can view it.
b) Call the the Flat File Dissasembly utility FFDASM.exe. the easiest way to do this is to create a bat file with the contents like the following:
c:
cd C:\Program Files\Microsoft BizTalk Server 2004\SDK\Utilities\PipelineTools
ffdasm.exe “C:\path\inputFlatFileName.pos” -bs “C:\path\schemaName.xsd” -c -v
pause..
The output should look like:
<AirlineMessage xmlns=”http://UnuniformPositionalFlatFile.Test“>
<K20 PLen=”0120“ Key2=”20“ Key3=”02“ ChangedFlag=”0“ PartyNo=”0“ Remark=”01“ SplName=”” SurName=”THORNTON“ xmlns=”” />
<K20 PLen=”0120“ Key2=”20“ Key3=”04“ ChangedFlag=”0“ PartyNo=”0“ Remark=”01“ SplName=”” SurName=”OGIHARA“ xmlns=”” />
<K20 PLen=”0120“ Key2=”20“ Key3=”05“ ChangedFlag=”0“ PartyNo=”0“ Remark=”01“ SplName=”” SurName=”WILSON“ xmlns=”” />
<K30 PLen=”0080“ Key2=”30“ Key3=”01“ ChangedFlag=”0“ Name=”JENNIFERMRS“ xmlns=”” />
<K30 PLen=”0080“ Key2=”30“ Key3=”02“ ChangedFlag=”0“ Name=”BENJAMINMSTR“ xmlns=”” />
<AirlineMessage>
R. Addis
by community-syndication | May 25, 2005 | BizTalk Community Blogs via Syndication
The more I work with the Rules API the more I’m convinced that the Vocabulary element was only created for the BRC. The most generous I can be is that someone came up with a good idea for the BRC and “retrofitted” it for the API. There are two reasons for this;
- The API is unsupported and undocumented.
- The BRL (Business Rules Language) can function happily without it – in fact a rule can reference directly the document element / .NET method etc.
If you take a look at the following two bites from the exported rule store, UI think you’ll see why I come to this conclusion;
The following is in effect a definition of a string literal in the export vocabulary
<vocabularydefinition id=”54e58c56-e279-4729-b0d4-2e7bfbef989e” name=”TransferControlAccount” description=””>
<literaldefinition type=”string”>
<string>222222</string>
</literaldefinition>
<formatstring language=”en-US” string=” Transfer Control Account” />
</vocabularydefinition>
The following is taken from an extract of an action. This interesting thing to note is that the literal value “222222” is also specified, along side the link to the vocab entry specifying it;
<function>
<vocabularylink uri=”2e2bb376-b4ac-4d42-9573-cdf42088f681″
element=”453cb556-e96e-4e10-b456-96bf6d1313c6″ />
<classmember classref=”Accounting” member=”AddAccountingEntries” sideeffects=”true”>
<argument>
<constant>
<vocabularylink uri=”2e2bb376-b4ac-4d42-9573-cdf42088f681″
element=”54e58c56-e279-4729-b0d4-2e7bfbef989e” />
<string>222222</string>
</constant>
</argument>
</classmember>
</function>
So why would this redundant data be here? Performance? I doubt it. The whole “immutability” of Rules and Vocabularies makes sense from a version management perspective until you actually start using them. Imagine the scenario; You have defined your Vocabulary before starting your rules (This is required because you can’t reference a Vocabulary from your rules until that Vocabulary has been published – this in itself is predicated on knowing all the terms up front – big assumption!). You define 25+ rules, using multiple references to your vocabulary. Fine. You now publish the rule policy. At this point the world is still in order – version 1.0 of both rules and vocabulary are in a lotus type harmony. But what happens when you want to extend or amend the vocabulary? You create a new version of the Vocabulary (cut & paste!),make your amendments and publish it. Of course NONE of your rules will reference this new vocabulary (1.1) as all links reference 1.0 of the vocabulary. The only way to reference the new vocabulary is to create a new version of your rules (cut & paste!), and edit EACH AND EVERY reference to the vocabulary to repoint to the new version.
This isn’t new of course, and tools do exist to deal with it.
The BRC has been criticised for this “cut & paste” mentality, but in fact some of the problem goes back to the design of the rules system itself.
by community-syndication | May 24, 2005 | BizTalk Community Blogs via Syndication
As has been noted before in the lore of BizTalk, it sure would be useful to use a
real debugger with orchestrations – at least occasionally.
There are times when an expression shape winds up getting a bit sticky (not that I
would know…) and a debugger would be just the ticket. Other times, the exception
you are getting from the orchestration engine isn’t at all clear.
Still other times, seeing the actual contents of messages or context as you step through
would be interesting.
Jon has posted
on debugging orchestrations (in IL) with an ILDASM/ILASM
loop, and had also discussed building your orchestration assemblies manually (using
xsharpp.exe). (He correctly noted these approaches wouldn’t generate anything
supportable, but they work for spelunking.)
I wanted to suggest something a bit different – I wanted to go back instead to the
venerable BTS File Dump utility that Charles
released in May of 2004 (before the GenerateCSFiles registry key was
discovered…) and propose a different technique for symbolic debugging.
First, install Charles’ BizTalk File Dump utility – you can download it from here.
Fire it up, and change the output path in the utility to something easy, like c:\temp\BTSFileDump.
Here is the Edit/Debug cycle…(Start by making sure the file dump utility is running,
and click the “Start dumping files” button.)
-
Build your orchestration project (or whole solution if need be.)
-
Do a Ctrl-Shift-F (Find in Files) in Visual Studio, and change the “Look in:”
folder to c:\temp\BTSFileDump
-
Search for something in one of your expression shapes, say, “MyClass.Execute”.
The correct generated file (that the file dump utility grabbed) will appear in your
Find Results – open it up, and set a breakpoint. (Not on the xml designer portion
– on the actual code!)
-
From the Debug menu, choose Processes and attach to BTSNTSvc.exe. (Have more
than one?) Choose CLR debugging only – not native. The symbols should
be loaded automatically – no need to copy PDBs to the GAC.
-
Trigger your orchestration however you normally would. Bask in a picture like
this: (Puts a Petzold-style WindowProc to shame…)
-
Use QuickWatch to examine message contents/context, if you like – you’ll be interested
in expansions like
this one (where sampleRequest is a message in the orchestration.)
-
Find your problem, Debug-Detach All, and fix the problem in the orchestration.
-
Click “Delete all files in output path” in the file dump utility (to avoid duplicates)
and rebuild. Repeat the process if you need to. (To save time, consider
an external tool that just re-GACs the orchestration assembly and resets the host
process. See here.)
Note that this technique will work in a production setting. You could copy the
PDBs and sources to the production server, and use DbgClr.exe (in the Framework
SDK) or cordbg.exe to attach to the appropriate
host process.
Happy debugging…
by community-syndication | May 24, 2005 | BizTalk Community Blogs via Syndication
As has been noted before in the lore of BizTalk, it sure would be useful to use a
real debugger with orchestrations – at least occasionally.
There are times when an expression shape winds up getting a bit sticky (not that I
would know…) and a debugger would be just the ticket. Other times, the exception
you are getting from the orchestration engine isn’t at all clear.
Still other times, seeing the actual contents of messages or context as you step through
would be interesting.
Jon has posted
on debugging orchestrations (in IL) with an ILDASM/ILASM
loop, and had also discussed building your orchestration assemblies manually (using
xsharpp.exe). (He correctly noted these approaches wouldn’t generate anything
supportable, but they work for spelunking.)
I wanted to suggest something a bit different – I wanted to go back instead to the
venerable BTS File Dump utility that Charles
released in May of 2004 (before the GenerateCSFiles registry key was
discovered…) and propose a different technique for symbolic debugging.
First, install Charles’ BizTalk File Dump utility – you can download it from here.
Fire it up, and change the output path in the utility to something easy, like c:\temp\BTSFileDump.
Here is the Edit/Debug cycle…(Start by making sure the file dump utility is running,
and click the “Start dumping files” button.)
-
Build your orchestration project (or whole solution if need be.)
-
Do a Ctrl-Shift-F (Find in Files) in Visual Studio, and change the “Look in:”
folder to c:\temp\BTSFileDump
-
Search for something in one of your expression shapes, say, “MyClass.Execute”.
The correct generated file (that the file dump utility grabbed) will appear in your
Find Results – open it up, and set a breakpoint. (Not on the xml designer portion
– on the actual code!)
-
From the Debug menu, choose Processes and attach to BTSNTSvc.exe. (Have more
than one?) Choose CLR debugging only – not native. The symbols should
be loaded automatically – no need to copy PDBs to the GAC.
-
Trigger your orchestration however you normally would. Bask in a picture like
this: (Puts a Petzold-style WindowProc to shame…)
-
Use QuickWatch to examine message contents/context, if you like – you’ll be interested
in expansions like
this one (where sampleRequest is a message in the orchestration.)
-
Find your problem, Debug-Detach All, and fix the problem in the orchestration.
-
Click “Delete all files in output path” in the file dump utility (to avoid duplicates)
and rebuild. Repeat the process if you need to. (To save time, consider
an external tool that just re-GACs the orchestration assembly and resets the host
process. See here.)
Note that this technique will work in a production setting. You could copy the
PDBs and sources to the production server, and use DbgClr.exe (in the Framework
SDK) or cordbg.exe to attach to the appropriate
host process.
Happy debugging…
by community-syndication | May 19, 2005 | BizTalk Community Blogs via Syndication
I am so glad to see the updates to the native adapters in BizTalk 2006. Scott Woodgate posted a document on the changes here. (you might need to join the BizTalkServerStuff group if this link doesn’t work) I was lucky enough to spend some time with the adapter framework over the past few months, and the behavior of the native ones with 2004 left me scratching my head a few times. Anyhow, I really like the Adapter Framework despite not liking the examples built on it that are included with BizTalk. I’ve also been surprised that the community hasn’t strucken out and developed a bit more. There’s the potential for great extensibility, and it’s not too complicated once one takes a couple weeks to get oriented. The SDK sample adapter with 2004 is way overbuilt in my opinion. Plus it uses the SystemMessageContext class which is on the list of unsupported classes. I honestly had enough with the SDK samples once I found that guy being used, so I don’t know if there are any more or not. Anyhow, kudos to the BTS team for spending some time on these! I think they’ll make a bigger splash than most people anticipate….
by community-syndication | May 19, 2005 | BizTalk Community Blogs via Syndication
When u want to receive XML data from your stored procedure which containts a SELECT and UPDATE (processed records) statement, you can have some problems to Generate the SQL Adapter XSD.
When u press the last Window in the Wizard you could get an error message Failed to execute queury.
What we saw in the SQL Profiler is that the Generate SQL Adapter Wizard Executes twice the stored procedure Oops. The first one from Application name SQLClient provider and the second one from Visual Studio .NET 2003.
The first query execute returns the XMLDATA (XML Schema) , but also the stored procedure update. The second run executes the stored procedure once again without your XMLData schema.
So disable your update in your stored procedure and generate your XML schema after that enable the update and delete the XMLData option.
Good to know that the Wizard runs the stored procedure twice……
by community-syndication | May 19, 2005 | BizTalk Community Blogs via Syndication
I don’t know about you but I find the best way to understand how something is working is from reading the code and then stepping through its execution, with VS’s trusty local and immediate windows. A friend of mine calls this “developers documentation”, and whilst I certainly wouldn’t go as far as saying it’s a substitute for good docs (and by that Chris, I mean “living models”!), I always end up doing it during any maintenance code cycles (unintended consequences and all that).
Unfortunately the execution of a BizTalk rule policy isn’t quite as straight forward. It’s not a set of simple sequential steps that you can step through. If it was it would probably be far less useful as a rules engine – and certainly far less performant.. Instead it implements a RETE algorithm for small rulesets and some propriety one for large rules sets according to the man himself. Anyway the long and the short of this is that rules processing is done behind closed doors, which makes for a tough time in understanding what happened during the policy being executed. I’m sure a good understanding of the RETE process would help – Dummies guide to RETE anyone?
The Microsoft Business Rules Composer (BRC) does allow rule “testing” by implementing the Tracking Interceptor DebugTrackingInterceptor. The following is a sample output produced by the BRC for the Loans Processing policy (from the Loans Sample in the SDK.
RULE ENGINE TRACE for RULESET: LoanProcessing 5/19/2005 12:46:13 PM
FACT ACTIVITY 5/19/2005 12:46:13 PM
Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e
Ruleset Name: LoanProcessing
Operation: Assert
Object Type: DataConnection:Northwind:CustInfo
Object Instance Identifier: 782
FACT ACTIVITY 5/19/2005 12:46:13 PM
Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e
Ruleset Name: LoanProcessing
Operation: Assert
Object Type: TypedXmlDocument:Microsoft.Samples.BizTalk.LoansProcessor.Case
Object Instance Identifier: 778
FACT ACTIVITY 5/19/2005 12:46:13 PM
Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e
Ruleset Name: LoanProcessing
Operation: Assert
Object Type: TypedXmlDocument:Microsoft.Samples.BizTalk.LoansProcessor.Case:Root
Object Instance Identifier: 777
CONDITION EVALUATION TEST (MATCH) 5/19/2005 12:46:13 PM
Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e
Ruleset Name: LoanProcessing
Test Expression: NOT(TypedXmlDocument:Microsoft.Samples.BizTalk.LoansProcessor.Case:Root.Income/BasicSalary > 0)
Left Operand Value: 12
Right Operand Value: 0
Test Result: False
CONDITION EVALUATION TEST (MATCH) 5/19/2005 12:46:13 PM
Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e
Ruleset Name: LoanProcessing
Test Expression: NOT(TypedXmlDocument:Microsoft.Samples.BizTalk.LoansProcessor.Case:Root.Income/OtherIncome > 0)
Left Operand Value: 10
Right Operand Value: 0
Test Result: False
CONDITION EVALUATION TEST (MATCH) 5/19/2005 12:46:13 PM
Rule Engine Instance Identifier: fb330399-15f0-4dc7-9137-4463a32f580e
Ruleset Name: LoanProcessing
Test Expression: TypedXmlDocument:Microsoft.Samples.BizTalk.LoansProcessor.Case:Root.PlaceOfResidence/TimeInMonths >= 3
Left Operand Value: 15
Right Operand Value: 3
Test Result: True
[.. cut for brevity ..]
This implements the IRuleSetTrackingInterceptor interface which you can use to produce you own custom debugging/tracing of rule processing.
Most of the information is there (although I’m still troubled that the tracing of the process doesn’t allow you to examine the facts i.e., the data rows, xml docs etc), but it’s not exactly easy to see what’s happened. Understanding the match-conflict resolution-action helps considerably.
If you are executing your policy outside of BizTalk, or in a component consumed within BizTalk you can specify an alternative IRuleSetTrackingInterceptor. This has the advantage of allowing you to step through the rule processing if you wish, and also allows you to view fact details (through the facts you pass to the policy). The following code demonstrates how to invoke your MyInterceptorClass().
xmlDocument = IncomingXMLMessage.XMLCase;
typedXmlDocument = new Microsoft.RuleEngine.TypedXmlDocument(“Microsoft.Samples.BizTalk.LoansProcessor.Case”,xmlDocument);
policy = new Microsoft.RuleEngine.Policy(“LoanProcessing”);
policy.Execute(typedXmlDocument,new MyInterceptorClass());
OutgoingXMLMessage.XMLCase = xmlDocument;
policy.Dispose();
Once you understand the various stages to the rule engine, you soon yearn for a better way to visualise a complex processing of rules. To this end I started thinking about how I’d like to see what was happening. In the past I’ve used UML sequence diagrams to demonstrate how messages are sent between object and also object lifetime.
Initially , I though that each fact would “constructed” by assertion and “destructed” by retraction. The same would happen with the rules placed into and out of the Agenda. This is sort of how it would look;
The problem with this approach is that it makes a very complex picture when many facts are asserted and many rules get passed onto the agenda. In the end I settled for a “swim lanes” type approach, separating each phase (facts asserting/retracting, condition matching, adding to agenda and firing actions) in a lane and then just processing sequentially. I think it works quite well, but you can see the loans processing sample output here.
The TrackingInterceptor also outputs to XML. If anyone’s interested in a copy of the source/component you can drop me a note here.
After my experiences I have the following request for MS
Request 1: Allow the BRC to use a difference Tracking Interceptor
Request 2: Allow the Tracking Interceptor to get hold of the full facts – i.e., let me dump out the xmldoc.
Request 3: Allow the “Call Rules” shape to specify a tracking interceptor
