Recoverable Interchange Processing Inside a Pipeline

Biztalk 2006 introduces many new and exciting features. One of these new features is Recoverable Interchange Processing for inbound receive interchanges. Biztalk 2006 now allows the option to fail the whole interchange or process just the non-errored single messages. This sample shows how easy it is to take advantage for this new feature. Please see the ReadMe.txt for setup instructions.

This sample should work with BizTalk 2006 and BizTalk 2006 R2.

Get more information from the original blog post on this topic: https://www.biztalkgurus.com/biztalk_server/biztalk_blogs/b/biztalk/archive/2005/07/20/biztalk-2006-recoverable-interchange-processing.aspx

Watch the video related to this sample here: https://www.biztalkgurus.com/biztalk_server/biztalk_2006/m/biztalk_2006_samples/32417.aspx

BizTalk 2006 Recoverable Interchange Processing

Biztalk 2006 introduces many new and exciting features.  One of these new features is Recoverable Interchange Processing for inbound receive interchanges.



First off, what is an Interchange?  An interchange is simply an item received by Biztalk.  In Biztalk terms, an interchange is typically used to define a message that contains several messages inside it. Some typical examples are batch flat files and Xml envelopes.



A specific example of this is a flat file document that contains a Header and many Detail records like this:







If we were to debatch this message using the flat file disassembler it would produce three single records.  In this case, it is one interchange that will produce three messages into the message box.



In Biztalk 2004, if a single record inside the flat file contained bad data the whole interchange (i.e. all the single messages) would fail.  In some business scenarios this is the desired behavior. 



This should not be confused with Adapter Batching.  Adapter batching will pick up and process many different interchanges together inside a configurable (usually) batch.  The interchanges inside a batch can fail in Biztalk 2004, just not individual messages inside a specific interchange.  Got it?



Biztalk 2006 now allows the option to fail the whole interchange or process just the non-errored single messages.  This is called Recoverable Interchange Processing.



Setting this up is as simple as setting the RecoverableInterchangeProcessing flag to True inside the XmlDisassembler pipeline component.  This can be done at runtime inside the Biztalk Server Admin tool without having to recompile any code. 



As the single messages are passing through the pipeline, the pipeline sets a new message property called BTS.InterchangeSequenceNumber.  When a single message inside an interchange fails, the sequence number along with useful message information is written to the event log.



Enough of my rambling…  How about taking a look at this for yourself.


Note this is based off of CTP build of Biztalk Server 2006.



Watch the video: Recoverable Interchange Processing Video



Download the sample:
Recoverable Interchange Processing Sample



See the readme.txt for setup instructions for the sample.  The default install is not configured to support recoverable interchanges.  You will need to change the property.



One additional point, I have heard that recoverable interchange processing will only work on flat files that have tag identifiers defined on the records.  I have not verified this but it makes sense.

BizUnit 2.0 – Automated Testing for BizTalk Solutions

I’ve released BizUnit 2.0, in case you’ve not come across it before, BizUnit is a framework that I put together to enable the rapid development of automated functional testing for BizTalk solutions, the motivation was to provide a framework that would bring automated testing of BizTalk solutions to the masses!! BizTalk solutions are often mission critical, an automated testing approach helps to drive quality before go-live and is essential post go-live to ensure that patches to not cause regressions. BizUnit is not restricted to testing BizTalk solutions, but it is targeted at them.


 


BizUnit defines test cases as Xml documents, which, enables test cases to be rapidly written and also enables the same test cases to moved between different environments by fixing up the Url’s for the endpoints, for example the endpoint Url’s for a development environment will be different to a testing, and pre-production environments.


 


Also, by having the test cases defined as Xml, it allows them to be easily generated, some of the customers that I have worked with define their test matrixes in Excel, and then auto-generate the BizUnit test cases from Excel.


 


Test Case Format


A test case is made up of three stages, test setup, test execution and test cleanup, the cleanup stage is always executed and intended to leave the platform in the same state that it started.


 


Each stage may consist of zero or more test steps, test steps are in general autonomous, state can be flowed between them if required using the ’context’ object that is passed to each test step.


  


 


How does it Work?


BizUnit takes a black box approach to testing BizTalk solutions, if you look at the scenario below, a BizTalk solution receives a request-response message over HTTP, the message is routed to an Orchestration which, sends a message to MSMQ and another to a FILE drop, the Orchestration waits for a FILE to be received, after which the Orchestration sends the response back to the waiting HTTP client. The solution also uses BAM, writing business data to the BAM database.


 


In order to test this scenario, a BizUnit test case is defined that has 5 test steps:



  1. The HttpRequestResponseStep sends the request to the two-way receive port and waits for the response. This step is executed concurrently so that the other test steps may execute whilst it waiting for the response
  2. The MSMQReadStep waits for a message to appear on an MSMQ queue, when it reads the message it uses the XmlValidationStep to perform schema validation and also execute a number of XPath expression to ensure the message contains the correct data
  3. The FileValidateStep waits for a FILE to be written to a given directory, when it reads the FILE it validates the data using the RegExValidationStep validation step since the FILE picked up was a flat file format
  4. The FileCreateStep creates a new FILE in the specified directory containing the data that the backend system would typically create. This allows the Orchestration to complete and send the response back to the waiting HttpRequestResponseStep step
  5. Finally, DBQueryStep is used to check that all of the BAM data has been successfully written to the BAMPrimaryImportDB 

 


BizUnit 2.0 Enhancements


A number of enhancements have been added to BizUnit 2.0, firstly, there are now some 37 test steps, which is vastly improved from the previous version. A lot of people have contributed test steps which I’ve included in this version. Briefly, here’s some of the enhancements:




  • Concurrent test step execution option – allows multiple test steps to be executed at once
  • Context – allows state to be flowed between test steps, think of the scenario where a FILE is written the content of which specifies where the response should be written
  • Test group setup / tear down stages  – allows group wide setup and cleanup
  • Help – all of the test steps have help in a .chm (accessed from Program Files\Microsoft Services BizTalkApplicationFramework\BizUnit 2.0\Documentation. The help contains Xml configuration examples for each test step that can be copy/pasted into a test case, along which a description of each configuration option etc
  • FailOnError – flag to allow failed test cases to be ignored, this is useful during the cleanup stage 

Here’s a list of all of the test steps in BizUnit build 2.0.1062.0, the documentation provides more details around what these steps do and how to use them:


BAMDeploymentStep


CheckPop3MailStep


ContextManipulatorStep


CrossReferenceDataVerificationStep


CrossReferenceSeedClearStep


CrossReferenceSeedLoadStep


DatabaseDeleteStep


DatabaseRowCountStep


DBQueryStep


DelayStep


DotNetObjectInvokerStep


EventLogCheckStep


EventLogClearStep


ExecuteCommandStep


FileCreateStep


FileDeleteMultipleStep


FileDeleteStep


FileMoveStep


FileMultiValidateStep


FilesExistStep


FilesMoveStep


FileValidateStep


HttpPostStep


HttpRequestResponseStep


MSMQCreateQueueStep


MSMQDeleteQueueStep


MSMQQueuePurgeStep


MSMQReadStep


MSMQWriteStep


OrchestrationConductorStep


ReceiveLocationEnabledStep


ReceivePortConductorStep


RenameDirectoryStep


RuleEngineStep


SMTPReadStep


SOAPHTTPRequestResponseStep


WaitOnFileStep


MQSeriesClearQueueStep


MQSeriesGetStep


MQSeriesPutStep


OutlookReadStep



 


Validation Steps:


BinaryValidationStep


ContextValidationStep


RegExValidationStep


XmlValidationStep


  


Context Loader Steps:


RegExContextLoader


TextContextLoader


XmlContextLoader


 


Test Coverage:


So, you’ve developed all of your BVT’s (Build Verification Tests) for your BizTalk solution using BizUnit, how do you know whether you’ve done a good job, and how much test coverage you have? Enter Jason Births “Orchestration Profiler”. This is a great tool that you can use in conjunction with BizUnit to ensure that you have adequate test coverage, it produces a detailed graphical report of what the test coverage is like, I highly recommend using it with BizUnit.



 


 


Finally, I’d like to give my special thanks to a number of people that have contributed test steps, requirements and ideas for BizUnit 2.0, my apologies if I have missed off anyone, it’s not intentional just my disorganisation!, please let me know if that is the case:


 


Dave Regan


Tanveer Rashid


Daren Jefford


Kevin Purcell


Karina Apostolides


Jon Bonnick


Brian Milburn


Rahmatullah Khan


 


Future work:


A couple of the guys above have done some great work building a Visual Studio plugin for BizUnit and a command line driver. I didn’t have time to get it in this release, but it’ll be following soon. If you find any bugs, or have test steps you’d like to contribute please sned them along to me and I’ll get them into the next drop.


 


Enjoy!


 

BizUnit 2.0 – Automated Testing for BizTalk Solutions

I’ve released BizUnit 2.0, in case you’ve not come across it before, BizUnit is a framework that I put together to enable the rapid development of automated functional testing for BizTalk solutions, the motivation was to provide a framework that would bring automated testing of BizTalk solutions to the masses!! BizTalk solutions are often mission critical, an automated testing approach helps to drive quality before go-live and is essential post go-live to ensure that patches to not cause regressions. BizUnit is not restricted to testing BizTalk solutions, but it is targeted at them.


 


BizUnit defines test cases as Xml documents, which, enables test cases to be rapidly written and also enables the same test cases to moved between different environments by fixing up the Url’s for the endpoints, for example the endpoint Url’s for a development environment will be different to a testing, and pre-production environments.


 


Also, by having the test cases defined as Xml, it allows them to be easily generated, some of the customers that I have worked with define their test matrixes in Excel, and then auto-generate the BizUnit test cases from Excel.


 


Test Case Format


A test case is made up of three stages, test setup, test execution and test cleanup, the cleanup stage is always executed and intended to leave the platform in the same state that it started.


 


Each stage may consist of zero or more test steps, test steps are in general autonomous, state can be flowed between them if required using the ’context’ object that is passed to each test step.


  


 


How does it Work?


BizUnit takes a black box approach to testing BizTalk solutions, if you look at the scenario below, a BizTalk solution receives a request-response message over HTTP, the message is routed to an Orchestration which, sends a message to MSMQ and another to a FILE drop, the Orchestration waits for a FILE to be received, after which the Orchestration sends the response back to the waiting HTTP client. The solution also uses BAM, writing business data to the BAM database.


 


In order to test this scenario, a BizUnit test case is defined that has 5 test steps:



  1. The HttpRequestResponseStep sends the request to the two-way receive port and waits for the response. This step is executed concurrently so that the other test steps may execute whilst it waiting for the response
  2. The MSMQReadStep waits for a message to appear on an MSMQ queue, when it reads the message it uses the XmlValidationStep to perform schema validation and also execute a number of XPath expression to ensure the message contains the correct data
  3. The FileValidateStep waits for a FILE to be written to a given directory, when it reads the FILE it validates the data using the RegExValidationStep validation step since the FILE picked up was a flat file format
  4. The FileCreateStep creates a new FILE in the specified directory containing the data that the backend system would typically create. This allows the Orchestration to complete and send the response back to the waiting HttpRequestResponseStep step
  5. Finally, DBQueryStep is used to check that all of the BAM data has been successfully written to the BAMPrimaryImportDB 

 


BizUnit 2.0 Enhancements


A number of enhancements have been added to BizUnit 2.0, firstly, there are now some 37 test steps, which is vastly improved from the previous version. A lot of people have contributed test steps which I’ve included in this version. Briefly, here’s some of the enhancements:




  • Concurrent test step execution option – allows multiple test steps to be executed at once
  • Context – allows state to be flowed between test steps, think of the scenario where a FILE is written the content of which specifies where the response should be written
  • Test group setup / tear down stages  – allows group wide setup and cleanup
  • Help – all of the test steps have help in a .chm (accessed from Program Files\Microsoft Services BizTalkApplicationFramework\BizUnit 2.0\Documentation. The help contains Xml configuration examples for each test step that can be copy/pasted into a test case, along which a description of each configuration option etc
  • FailOnError – flag to allow failed test cases to be ignored, this is useful during the cleanup stage 

Here’s a list of all of the test steps in BizUnit build 2.0.1062.0, the documentation provides more details around what these steps do and how to use them:


BAMDeploymentStep


CheckPop3MailStep


ContextManipulatorStep


CrossReferenceDataVerificationStep


CrossReferenceSeedClearStep


CrossReferenceSeedLoadStep


DatabaseDeleteStep


DatabaseRowCountStep


DBQueryStep


DelayStep


DotNetObjectInvokerStep


EventLogCheckStep


EventLogClearStep


ExecuteCommandStep


FileCreateStep


FileDeleteMultipleStep


FileDeleteStep


FileMoveStep


FileMultiValidateStep


FilesExistStep


FilesMoveStep


FileValidateStep


HttpPostStep


HttpRequestResponseStep


MSMQCreateQueueStep


MSMQDeleteQueueStep


MSMQQueuePurgeStep


MSMQReadStep


MSMQWriteStep


OrchestrationConductorStep


ReceiveLocationEnabledStep


ReceivePortConductorStep


RenameDirectoryStep


RuleEngineStep


SMTPReadStep


SOAPHTTPRequestResponseStep


WaitOnFileStep


MQSeriesClearQueueStep


MQSeriesGetStep


MQSeriesPutStep


OutlookReadStep



 


Validation Steps:


BinaryValidationStep


ContextValidationStep


RegExValidationStep


XmlValidationStep


  


Context Loader Steps:


RegExContextLoader


TextContextLoader


XmlContextLoader


 


Test Coverage:


So, you’ve developed all of your BVT’s (Build Verification Tests) for your BizTalk solution using BizUnit, how do you know whether you’ve done a good job, and how much test coverage you have? Enter Jason Births “Orchestration Profiler”. This is a great tool that you can use in conjunction with BizUnit to ensure that you have adequate test coverage, it produces a detailed graphical report of what the test coverage is like, I highly recommend using it with BizUnit.



 


 


Finally, I’d like to give my special thanks to a number of people that have contributed test steps, requirements and ideas for BizUnit 2.0, my apologies if I have missed off anyone, it’s not intentional just my disorganisation!, please let me know if that is the case:


 


Dave Regan


Tanveer Rashid


Daren Jefford


Kevin Purcell


Karina Apostolides


Jon Bonnick


Brian Milburn


Rahmatullah Khan


 


Future work:


A couple of the guys above have done some great work building a Visual Studio plugin for BizUnit and a command line driver. I didn’t have time to get it in this release, but it’ll be following soon. If you find any bugs, or have test steps you’d like to contribute please sned them along to me and I’ll get them into the next drop.


 


Enjoy!


 

Get Connected to Free Product Support and Tremendous Online Collaboration


Have you ever wanted to speak to Microsoft developers of a specific feature of BizTalk Server? I am sure your answer was “Yes let me at them”, so the Business Process Integration Division is extending an invitation to all customers to join our key feature developers, program managers, and testers in the following newsgroups:





  • microsoft.public.biztalk.accelerator.forsuppliers



  • microsoft.public.biztalk.newuser



  • microsoft.public.biztalk.accelerator.rosettanet



  • microsoft.public.biztalk.admin



  • microsoft.public.biztalk.appintegration



  • microsoft.public.biztalk.framework



  • microsoft.public.biztalk.general



  • microsoft.public.biztalk.library



  • microsoft.public.biztalk.nonxml



  • microsoft.public.biztalk.orchestration



  • microsoft.public.biztalk.sdk



  • microsoft.public.biztalk.server



  • microsoft.public.biztalk.setup



  • microsoft.public.biztalk.tools



  • microsoft.public.biztalk.xlangs



  • microsoft.public.biztalk.xsharp

We’ve been working very hard over the past year to connect with folks just like you and want to include you in our community of Most Valuable Professionals (MVP), developers, information technology professionals, chief information officers, chief executive officers, or any other role within large, medium, and small companies that hang out in our online newsgroup communities. We want to have you join in this vibrant online community to ask those questions you always wanted to ask but did not know where to go. Well, now you know where to go, we want you to come on in and join us!



If you are new to BizTalk Server, try out the NewUser newsgroup, Microsoft.public.biztalk.newuser.



We’re offering two levels of interaction with Microsoft Corporation employees as follows:




  1. Managed Newsgroup Support
  2. Unmanaged Newsgroup Support

Managed Newsgroup Support


MSDN managed newsgroups are available in English to MSDN Universal, Enterprise, Professional and Operating Systems subscribers to receive free technical support on select Microsoft technologies as well as to share ideas with other subscribers. MSDN managed newsgroups provide:



  • Unlimited on-line technical support – keep your PSS incidents
  • A commitment to respond to your post within two business days
  • Over 200 newsgroups to choose from
  • Spam protection for your e-mail address when posting items

Go to the following URL to sign up: http://msdn.microsoft.com/newsgroups/managed .  These newsgroups are monitored by Microsoft support engineers and product group team member as described above.


Unmanaged Newsgroup Support


MSDN unmanaged newsgroups are available to all individuals.


Go to the following URL to participate: http://msdn.microsoft.com/newsgroups. These newsgroups are monitored by Microsoft product group members, other customers like you, most valuable professionals, and various other individuals.


Questions, suggestions, and direct feedback can be sent to me.



James Fort


BPI Community Lead


mailto:jfort@microsoft.com 

Flat File Disassembler Output Options Video

This 9:49 minute video covers the three different output options from the biztalk flat file Disassembler provided in Biztalk. The three options are single detail records without headers or trailers, all detail records together without header and trailers, or all details together with the header and trailer.

This video was made for BizTalk 2004 but should still be relevant for other versions of BizTalk.

Working with the BizTalk Flat File Disassembler Video


I have put together a short 10 minute biztalk training video covering some of the different output options available using the Flat File Disassembler in Biztalk Server 2004. 



This video covers three different output options from the flat file disassembler. The three options covered are single detail records without a header or trailer, all detail records together without a header or trailer, or all details together with the header and trailer.



This video walks through the Flat File Disassembler Sample from a past post. It might be helpful to have the sample downloaded before watching the video.



I have the video available for live play via shock wave or downloadable via Windows Media Player.



Watch the video: Flat File Disassembler Output Options Video



Comments and feedback are strongly recommended.


Envelope and XPath Debatching in an Orchestration Lab

This lab walks through two different types of message debatching in Biztalk. It shows how envelopes can be used with the Xml Disassembler to break up larger Xml messages into single messages. It also covers xpath in a loop inside an Orchestration.

Get more information from the original blog post on this topic: https://www.biztalkgurus.com/biztalk_server/biztalk_blogs/b/biztalk/archive/2005/07/12/xpath-debatching-and-envelope-debatching-biztalk-lab.aspx

XPath Debatching and Envelope Debatching Biztalk Lab

Envelopes can be used with the Xml Disassembler in Biztalk 2004 to break up larger Xml messages into single messages.  Another approach to message debatching is to use xpath in a loop inside an Orchestration.  Both these approaches are covered in this Message Debatching Options Lab for Biztalk 2004. 



This lab will take around 40 minutes to complete and is based on Microsoft’s hands-on-labs.



Download: Message Debatching Options Hands-on Lab



Setup is simple.  Unzip the download to your c:\ drive.  Make sure the root folder named “Microsoft Hands-On-Labs” is located at c:\.  The lab manual is located under \ILL009_Debatching.  The exercises are located in the \Source\Exercises folder.  The full sample solution is located in the \Source\Solution folder.



This lab is recommended for beginner and above biztalk users looking to learn more about message debatching in biztalk.