Error Fixed In Flat File Disassembler Sample

Today I was working on some content for Tech Ed when I found a major error in a sample I posted about 4 months ago. 



The error caused the CR-LF’s between records to be included in the positional character count during debatching and actually into the messages themselves.  This through off the file layout but did not give any errors.  In fact, the expected result of three files were received.  They just did not have the right data in them. 



I have seen similar problems in the past were the Flat File Disassembler does not behave in any way as I would expect.



Below is before and after screen shots of the schema.


Before: After:         



The error is easily fixed by added an additional node into the schema that is Delimited, Post Fixed with CR-LF as the Hexadecimal delimiter.  This fix was required in both the Header and SingleRecord schemas.



The updated sample is available for download.



Download: Flat File Disassembler Sample
Watch the video: Flat File Disassemblier Output Options Video

Flat File Disassembler Output Options

The flat file disassembler can produce single record output or a single batch output. This sample shows how to accomplish each task when you need it. Make sure to read the blog content below for additional information.

This sample should work with BizTalk 2004 and BizTalk 2006.

Get more information from the original blog post on this topic: https://www.biztalkgurus.com/biztalk_server/biztalk_blogs/b/biztalk/archive/2004/12/17/flat-file-disassembler-output-types-in-biztalk-server-2004.aspx

 

Large Messages Can Cause Large Problems in BizTalk

Recently I have seen a lot of talk about large message support in BizTalk 2004.  The most interesting and depressing was on the BizTalk Core Engine WebLog.  It seems that large messages do not work as well in BizTalk 2004 as I would have hoped.



I have had success with some types of large messages using the file adapter and mapping.  I have gotten it to work with messages up to 500 MB but in a production environment this would not be a viable or supported solution. 



During some recent testing, I discovered the follow condition that rather surprised me.



Scenario


I was returning an unpredictable amount of data out of an Oracle database using .net code.  This data can be 1 MB or up to 250MB+ and needed to be processed in a single batch.  This data is then converted into Xml, cast to a message, and mapped with the results written to disk using a Send Shape.  Life was good until the extracted data returned was greater then 220 MB in size.


 


At this point, everything went downhill.  BizTalk Server began to have serious problems.  Receive functions were shutting down and I received the following Warning message in the Application Log:



A catastrophic failure occurred in the BizTalk service. The service will shutdown and auto-restart in 1 minute. If the database is still unavailable, this cycle will be repeated.


 


Error message: Exception of type System.OutOfMemoryException was thrown.


Error source:


 


BizTalk host name: BizTalkServerApplication Windows service name: BTSSvc{E93E7C4A-9E8B-40F6-AD7C-4F53B85F36C4}


 


For more information, see Help and Support Center at



All appears to be well and good since BizTalk seems to be handling the situation as expected… except for when the service actually starts again.  Since the Orchestration that caused the fatal error is still Active (the Orchestration itself did not through any errors), BizTalk starts to process the Orchestration from the last persistence point.  Thus, repeating the cycle in an endless loop.



Now, for the interesting behavior…  If this is running in a multi server BizTalk environment at the point the 1st BizTalk Server fails the 2nd one picks up; like I would expect.  I end up getting a completed Orchestration with no errors.  Super!  But, I only get a zero byte file as output.  So, looking inside HAT everything would look like it completed successfully with no errors.  Not good…



Overall, I found that an Orchestration seems to be able to handle about 440 MB of “stuff” before it crashes.  Just make sure you know what makes a copy of the message (like mapping).  Of course, processes with this amount of data would not be supported nor do I recommend actually doing it.  But, it has been fun to play around with and see what fails.



The moral of this story is to be very mindful of your message size.

Log4net presentation

A short while ago, I did a presentation for the Twin Cities .NET user group on
log4net.  (You might recall an earlier
blog entry where I discussed using log4net with BizTalk 2004…) 

The presentation was not specific to BizTalk 2004 – instead, it attempts to describe
why I think the log4net library is so very well thought out.  You can grab it here,
if you like.

The last slide references Loren
Halvorsen’s comparison
of log4net and the new Microsoft
Enterprise Library Logging Block
, which I would recommend taking a look at (the
comparison, that is.)  Tom Hollander (among many others!) later weighed
in with this
piece
.

Lastly, 1.2.9 beta of log4net has
recently become available (release notes here)….However,
my extensions
to log4net have not yet been updated to reflect this new drop.  I’ll be sure
to post when I’m able to update.

Going to Tech Ed 2005?

Are you going to Tech Ed 2005? Brennan and I (Brandon) will be presenting a cabana session during the breakouts and available to answer questions at the BizTalk booth all week. Hope to see you there! Cabana Title: Implementing Real-World Integration Patterns with BizTalk Abstract: This session will review several BizTalk 2004 patterns, including accompanying implementation pattern examples. In addition, the session will illustrate how to successfully leverage patterns within common real-world technology and business scenarios….

Debatching Messages Into Smaller Batches In An Orchestration

It is a common request.  How can I debatch a message into smaller batches rather than single messages inside BizTalk 2004? 



I have tried many times to get this accomplished natively inside the pipeline.  One would think it would be as easy as setting the group property to 10 if you wanted a batch size of 10, but that does not seem to work.  The only way I have been able to break up a message into smaller batches has been inside custom code.



I wrote a .net component that can be called using an atomic scope shape from inside an Orchestration.  This component will return smaller batches of any size you specify inside the Orchestration and wrap the smaller batches inside a specified root node.  This .net component uses the XmlTextReader to stream the message along and debatch it. 



This .net component is called StreamHelper.  It illustrates the following:



  • Using XmlTextReader
  • Passing an Xlang Message into a .net component
  • Returning an Xlang Message from a .net component
  • Setting context properties on a message returned from a .net component


Download: Debatch Streaming Sample



Setup: Download and extract the solution.  Build and GAC the StreamHelper assembly then Build and Deploy the DebatchProperties and DebatchOrch assemblies.  I have included a basic sample file to test with.  For more detailed information on the sample see the comments throughout the solution.



A better way to do it? 


I’m sure there is.  I just put this together in a short amount of time.  I would guess the most efficient place for this type of work would be inside a custom pipeline component.  But, working with custom pipelines can be difficult.



How does this debatching approach perform compared to the others?


Performance was almost identical to that of the Xml Node List inside an atomic scope.  In this sample, I used the XmlTextReader and passed Xlang messages in and out of the component to try to increase performance.  That did not seem to accomplish anything.  As expected, as the total number of messages inside the atomic scope increased the performance decreased. 






























































































Type


XML Size (MB)


# Nodes


Batch Size


Time (Sec)


Msg/Sec


Msg Size (kb)


.net Stream


1.6


500


1


5


100.0


3.0


.net Stream


3.6


1100


1


18


61.1


3.0


.net Stream


7.2


2200


1


44


50.0


3.0


.net Stream


18.1


5500


1


224


24.6


3.0


.net Stream


128.6


38500


1



FAIL



.net Stream


1.6


500


10


1


500.0


3.0


.net Stream


3.6


1100


10


3


366.7


3.0


.net Stream


7.2


2200


10


9


244.4


3.0


.net Stream


18.1


5500


10


15


366.7


3.0


.net Stream


128.6


38500


10


159


242.1


3.0



For more debatching information please see my past post:
Debatching Options and Performance Considerations in BizTalk 2004


 

Integrating SSO/Configuration with Deployments

(Update: See the latest on the Deployment Framework here.)

Just when you thought automated deployments for BizTalk 2004 were safe, another update
to the “Deploy with NAnt” template comes along… 🙂 

This is a very worthwhile “upgrade” to consider taking advantage of.  One of
the big items is support for “configuration info within SSO”, but a complete list
of features/fixes, etc. is below. 

SSO Integration: If you are currently using the SettingsFileGenerator.xls spreadsheet
(discussed in previous entries) in conjunction with the XmlPreProcess tool
to handle “environment-relative” configuration – that is, to handle variations between
your dev/QA/production environments – you will be glad to know you can get more leverage
out of that process now…

To review, with this process, you use the names that are defined in the first column
of the SettingsFileGenerator spreadsheet (shown below) as “tokens” within your binding
file, as in: 

<!-- ifdef ${ xml preprocess} -->
<!-- <Address>${FileSendLocation}\%MessageID%.xml</Address> -->
<!-- else -->
<Address>C:\temp\BizTalkSample OutDir\%MessageID%.xml</Address>
<!-- endif -->

At install time, the value for FileSendLocation that is appropriate to the environment
you are deploying to
is “plugged in” by XmlPreProcess (but
for developers, the file remains legal XML.)  The spreadsheet might look as follows:

(click)

Now, however, you can also put in name-value pairs (whether they vary
by environment or not) into the spreadsheet that you need access to at run time (i.e.
general configuration information.

The name-value pairs will be deployed into a newly created affiliate application within
the SSO, and can be read using the SSOSettingsFileReader class (which serves as a
cache, too – and is provided in the download.)  The static methods on this
class can be used from an orchestration expression shape, or from assemblies that
orchestrations call – as in string test = SSOSettingsFileReader.ReadValue(“BizTalkSample”,”SomeAppConfigItem”); 
(By the way, the “nested names” shown in the spreadsheet above are just
meant to suggest a partitioning mechanism you might want to use, rather than one flat
“namespace” of config data.)

Of course, the deployment scripts handle all the interaction with the SSO database
for deployments/un-deployments.  This whole mechanism (SettingsFileGenerator.xls,
XmlPreProcess, and now SSOSettingsFileImport.exe) is worth using even if you don’t
use the rest of this script infrastructure.  Note that Jon Flanders had a great
article on SSO-based configuration using a strongly-typed approach, where as this
approach is loosely-typed, but leverages a mechanism you might already be using (i.e.
the spreadsheet concept.)  (Note: Jon’s code for creating/deleting affiliate
apps helped me complete the SSOSettingsFileImport.exe utility quickly…) 

Other features/fixes: 

  • Support for multiple schema assemblies, including the case where schema assemblies
    reference each other due to schema imports.  (Undeployments occur in reverse
    order of deployments.)
  • Support for multiple orchestration assemblies, including the case where orchestration
    assemblies reference each other due to the use of Call/Start Orchestration shapes. 
    (Undeployments occur in reverse order of deployments.) 
  • Support for multiple orchestration and port binding files (just list them as comma
    separated values in the orchBindings/portBindings properties.) 
  • Fixed bug that occurred when undeploying ISAPI extensions in Windows 2003 / IIS 6. 
  • Fixed bug in undeploying pipeline components (thanks to John Adams) 
  • Added /NOFORCE flag to IISRESET calls in accordance with http://support.microsoft.com/kb/286196 
  • New property “includeInstallUtilForComponents” was added, which can be set to ‘true’
    in your project-specific nant file.  This will cause your component assemblies
    to be called with “installutil.exe” (with the /u flag for undeployment.)  This
    is useful if you have .NET Installer-derived classes that need to get called for creating
    event sources, perf counters, etc.  (Yes, you could certainly argue these should
    be called by the WiX piece of the script infrastructure instead, but this method works
    and is more in keeping with the spirit of the entire process.)  Note there is
    no harm if a given component assembly listed in your components property does not have
    an installer class. 
  • Eliminated the need for the includeCustomTarget property, and provided additional
    ways to supply custom functionality.  Project-specific nant file can now supply
    any or all of customDeployTarget, customPostDeployTarget, customUndeployTarget, customPostUndeployTarget
    — and they will be called if they are present. 
  • Integrated the deployment of log4net-specific stuff, if the includelog4net property
    is set to true.  See the work on BizTalk/log4net
    integration for more information.  (Log4net sample usage is included in the
    new sample.)   
  • Now using NAnt version .85 – so be sure to grab the latest NAnt/NAntContrib since
    the nant scripts are using .85-specific syntax to avoid “deprecated” warnings that
    were occurring otherwise.  Note that even once you have installed NAnt .85, you
    will still be able to deploy BizTalk projects that are using old versions of the deployment
    scripts.  Note also that this is a pretty basic port to .85 – there are no doubt
    more elegant ways of doing lots of things that I haven’t investigated yet, now that
    nant has expression support. 

Other notes 

The scripts like to see BizTalk projects using Debug and Release for target names. 
A long while back I had suggested a file-level search/replace within btproj files
to change these, but two more elegant options are available: 

  • In project properties, change the output location for the “Development”
    configurations to “bin\debug” and output location of “Deployment”
    configurations to “bin\release”.
  • (Via Puneet Gupta): Change the template for BizTalk projects, found in BTSApp.btproj
    file in %BizTalkInstallDir%\Developer Tools\BizTalkWizards\BTSApp\Templates\1033,
    to reflect Debug and Release targets. 

Another note: Lots of folks have noticed that XmlPreProcess can be used for a lot
more than port binding files!  Log4net config files, orchestration binding files,
you name it.  See the full Word documentation in the download for an example
of how to do this within customDeployTarget.

How do I upgrade?

  1. With a safe copy of your BizTalk solution, expand the “Core” zip contents
    right on your project directory, such that the core scripts will overwrite what you
    have now (but your project-specific scripts will be preserved.)
  2. Manually copy the WiX-related updates to your *.WiXSetup directory, since they will
    get unzipped to BizTalkSample.WiXSetup
  3. Do a deployment on a developer workstation and a server to test things out…

Downloading

  • You can always use the right-hand links on this blog.
  • Full sample here,
    core scripts here
  • GotDotNet workspace here. 
    Because of issues with GotDotNet, I’m only going to be keeping the EXE utility
    source code up to date – not the entire sample & core tree.

 Enjoy – comments always welcome and appreciated.

Integrating SSO/Configuration with Deployments

(Update: See the latest on the Deployment Framework here.)

Just when you thought automated deployments for BizTalk 2004 were safe, another update
to the “Deploy with NAnt” template comes along… 🙂 

This is a very worthwhile “upgrade” to consider taking advantage of.  One of
the big items is support for “configuration info within SSO”, but a complete list
of features/fixes, etc. is below. 

SSO Integration: If you are currently using the SettingsFileGenerator.xls spreadsheet
(discussed in previous entries) in conjunction with the XmlPreProcess tool
to handle “environment-relative” configuration – that is, to handle variations between
your dev/QA/production environments – you will be glad to know you can get more leverage
out of that process now…

To review, with this process, you use the names that are defined in the first column
of the SettingsFileGenerator spreadsheet (shown below) as “tokens” within your binding
file, as in: 

<!-- ifdef ${ xml preprocess} -->
<!-- <Address>${FileSendLocation}\%MessageID%.xml</Address> -->
<!-- else -->
<Address>C:\temp\BizTalkSample OutDir\%MessageID%.xml</Address>
<!-- endif -->

At install time, the value for FileSendLocation that is appropriate to the environment
you are deploying to
is “plugged in” by XmlPreProcess (but
for developers, the file remains legal XML.)  The spreadsheet might look as follows:

(click)

Now, however, you can also put in name-value pairs (whether they vary
by environment or not) into the spreadsheet that you need access to at run time (i.e.
general configuration information.

The name-value pairs will be deployed into a newly created affiliate application within
the SSO, and can be read using the SSOSettingsFileReader class (which serves as a
cache, too – and is provided in the download.)  The static methods on this
class can be used from an orchestration expression shape, or from assemblies that
orchestrations call – as in string test = SSOSettingsFileReader.ReadValue(“BizTalkSample”,”SomeAppConfigItem”); 
(By the way, the “nested names” shown in the spreadsheet above are just
meant to suggest a partitioning mechanism you might want to use, rather than one flat
“namespace” of config data.)

Of course, the deployment scripts handle all the interaction with the SSO database
for deployments/un-deployments.  This whole mechanism (SettingsFileGenerator.xls,
XmlPreProcess, and now SSOSettingsFileImport.exe) is worth using even if you don’t
use the rest of this script infrastructure.  Note that Jon Flanders had a great
article on SSO-based configuration using a strongly-typed approach, where as this
approach is loosely-typed, but leverages a mechanism you might already be using (i.e.
the spreadsheet concept.)  (Note: Jon’s code for creating/deleting affiliate
apps helped me complete the SSOSettingsFileImport.exe utility quickly…) 

Other features/fixes: 

  • Support for multiple schema assemblies, including the case where schema assemblies
    reference each other due to schema imports.  (Undeployments occur in reverse
    order of deployments.)
  • Support for multiple orchestration assemblies, including the case where orchestration
    assemblies reference each other due to the use of Call/Start Orchestration shapes. 
    (Undeployments occur in reverse order of deployments.) 
  • Support for multiple orchestration and port binding files (just list them as comma
    separated values in the orchBindings/portBindings properties.) 
  • Fixed bug that occurred when undeploying ISAPI extensions in Windows 2003 / IIS 6. 
  • Fixed bug in undeploying pipeline components (thanks to John Adams) 
  • Added /NOFORCE flag to IISRESET calls in accordance with http://support.microsoft.com/kb/286196 
  • New property “includeInstallUtilForComponents” was added, which can be set to ‘true’
    in your project-specific nant file.  This will cause your component assemblies
    to be called with “installutil.exe” (with the /u flag for undeployment.)  This
    is useful if you have .NET Installer-derived classes that need to get called for creating
    event sources, perf counters, etc.  (Yes, you could certainly argue these should
    be called by the WiX piece of the script infrastructure instead, but this method works
    and is more in keeping with the spirit of the entire process.)  Note there is
    no harm if a given component assembly listed in your components property does not have
    an installer class. 
  • Eliminated the need for the includeCustomTarget property, and provided additional
    ways to supply custom functionality.  Project-specific nant file can now supply
    any or all of customDeployTarget, customPostDeployTarget, customUndeployTarget, customPostUndeployTarget
    — and they will be called if they are present. 
  • Integrated the deployment of log4net-specific stuff, if the includelog4net property
    is set to true.  See the work on BizTalk/log4net
    integration for more information.  (Log4net sample usage is included in the
    new sample.)   
  • Now using NAnt version .85 – so be sure to grab the latest NAnt/NAntContrib since
    the nant scripts are using .85-specific syntax to avoid “deprecated” warnings that
    were occurring otherwise.  Note that even once you have installed NAnt .85, you
    will still be able to deploy BizTalk projects that are using old versions of the deployment
    scripts.  Note also that this is a pretty basic port to .85 – there are no doubt
    more elegant ways of doing lots of things that I haven’t investigated yet, now that
    nant has expression support. 

Other notes 

The scripts like to see BizTalk projects using Debug and Release for target names. 
A long while back I had suggested a file-level search/replace within btproj files
to change these, but two more elegant options are available: 

  • In project properties, change the output location for the “Development”
    configurations to “bin\debug” and output location of “Deployment”
    configurations to “bin\release”.
  • (Via Puneet Gupta): Change the template for BizTalk projects, found in BTSApp.btproj
    file in %BizTalkInstallDir%\Developer Tools\BizTalkWizards\BTSApp\Templates\1033,
    to reflect Debug and Release targets. 

Another note: Lots of folks have noticed that XmlPreProcess can be used for a lot
more than port binding files!  Log4net config files, orchestration binding files,
you name it.  See the full Word documentation in the download for an example
of how to do this within customDeployTarget.

How do I upgrade?

  1. With a safe copy of your BizTalk solution, expand the “Core” zip contents
    right on your project directory, such that the core scripts will overwrite what you
    have now (but your project-specific scripts will be preserved.)
  2. Manually copy the WiX-related updates to your *.WiXSetup directory, since they will
    get unzipped to BizTalkSample.WiXSetup
  3. Do a deployment on a developer workstation and a server to test things out…

Downloading

  • You can always use the right-hand links on this blog.
  • Full sample here,
    core scripts here
  • GotDotNet workspace here. 
    Because of issues with GotDotNet, I’m only going to be keeping the EXE utility
    source code up to date – not the entire sample & core tree.

 Enjoy – comments always welcome and appreciated.

’Processing convoy workflow scenarios’ patent

For those who are wondering why I’m not blogging very regularly: It’s not that I’m lying on my back and sun-bathing on the beach or anything (which was never been an option since it has been snowing a lot), I’m just rather busy lately. I was doing a SP1 evaluation earlier, for which MS has granted me a prestigious award for ‘Most Valuable Debugger’. It’s a kind of a joke, but nevertheless I’m very proud and amused at the same time :-). I guess I will have to change my blogging style in order to make it up in the future. So expect some more ‘publish some ideas quickly’ posts soon…


Here is something very interesting – I’ve discovered it some time ago when I was searching for general background information about convoys. This explains why it’s so hard to find anything theoretical and not directly from MS. More precisely, the ‘Processing convoy workflow scenarios’ patent has been linked to 8 different people, amongst them 1 very famous blogger named Lee.