Debatching Into Smaller Batches Inside an Orchestration

Debatching into single messages is easy with BizTalk 2004. This sample shows how messages can be groups into batches of any size inside an Orchestration using an external .net component. The external .net component uses streams to read the message from an Orchestration. The message is passed into the component as an XLang message type.

This sample should work in BizTalk 2004 and BizTalk 2006.

Get more information from the original blog post on this topic:  https://www.biztalkgurus.com/biztalk_server/biztalk_blogs/b/biztalk/archive/2005/03/22/debatching-messages-into-smaller-batches-in-an-orchestration.aspx

BizTalk WMI programming: How to write cleaner WMI code a lot faster

I’ve seen numerous handy WMI scripts and code samples from fellow BizTalk bloggers and from the BizTalk help. Amazingly every one of these samples use late-bound objects calls to invoke methods and access properties from the BizTalk management objects. This makes the code rather verbose, it’s very inconvenient and also very error prone. There’s a better way to write your WMI code. Therefore you have to download first and then install the ‘Management (WMI) Extensions for Visual Studio .NET 2003 Server Explorer’. This tool makes WMI programming much easier and enjoyable (at least for me). It’s no rocket science or anything but it comes in very handy for making your first steps on the WMI moon. You don’t have to take my word on it, look for yourself:

 



After having installed the extension you can go into your Visual Studio’s Server Explorer and select the namespace you want to add management classes from (select ‘add classes’ from the context menu).

 


 

After selecting the desired classes from your namespace (‘root \MicrosoftBizTalkServer’ I assume) the Server Explorer window will contain a list of the selected management object-types. It also shows a list of all instances for a given class when you click it. When having selected an instance, all of its properties will be shown inside the property-window (you could modify them from there too, but I wouldn’t recommend doing it this way).

 


 

Also methods can be invoked on classes and instances; this is nice for testing purposes. *No big deal* you are thinking of course; now pay attention:

 

When selecting ‘Generate Managed Class’ from the menu, the extension will automatically generate a wrapper class and add it to your solution’s startup project (it targets your .net project language).

 


 

The generated classes have strongly typed properties and methods, so you have access to and can invoke by simply using their name (even enum-types are created). You benefit from good old Intellisense again. Behind the covers the generated wrapper classes still use late-bound objects and calls of course, but you don’t have to care at least about this, nor write ugly late binding code. Now you can write clean high-level WMI code a lot faster.

Take a look at this little WMI sample (where I make a call to ‘GetInstances’: this overloaded method is created automatically and it requires only the where-part of the WQL query to return a typed collection of your WMI object-type).

 


 

Try to rewrite the following method (please don’t focus on the language change; it’s purely for educational purposes. I’ve never executed a single line of this code 🙂

 


 

Into this:

 


 

Which one do you prefer?

Announcements

Okay, peridiodically the marketing team / other teams ask me to post things to make sure the word is getting out about events. My guess is that all of you are reading scott woodgate’s blog (see my links) and if you are not you should, so these will probably be duplicates, but please read them and in this case do the feedback so we can have a strong tech ed session this year (I will be there, so please stop by my talk and we can chat :). Also, I think I have wrangled the owner of the orchestration compiler and now one of the key developers on the engine also to put some info on the blog. That would be great stuff. He is the ultimate source of information related to compiler questions and how to express certain design patterns in your orchestration. Looking forward to hearing from Paul. Okay, the Tech-Ed announcements:

Industry leaders will conduct special in-depth workshops on Sunday, June 5, from 1 P.M. to 6 P.M. These concentrated “Deep Dive” sessions will last five hours and provide an in-depth look at important technology areas and solutions. 

The pre-con will start off with an overview session led by Rebecca Dias that gives an overview of SO, Microsoft solutions in the SO space and how they work together. It is then followed by a session by Yasser Shohoud on how to build web service on .Net. The session is followed by a webservices lab. This is then followed by a HIS session by Paul Larsen that shows how to extend the reach of SOA to mainframe via HIS. Following Paul’s session, the earlier web services lab is extended with HIS to expose legacy IBM systems as a web service. Mike Wood’s session picks up from there and shows cases how to orchestrate, expose and integrate with web services. Mike’s talk is followed by another hands-on-lab session where we extend the web services / HIS lab with orchestration in the middle orchestrating these services and maybe followed by exposing the resulting orchestration again as a web service.

Title: Building Simple and Complex Web Services using the .Net Framework, BizTalk Server and Host Integration Server

Are you convinced that Service Orientation is the way to go, but don’t yet have the technical skills to begin implementation or are confused by the plethora of choices Microsoft offers you to build Service Oriented systems? Do you want fundamentals on Microsoft’s integration technologies and guidance on trade offs if you use one solution vs. the other for different applicable scenarios? If so, this is a training you won’t want to miss. This training will provide you the guidance and information you need to make an informed decision on how to best construct your integration solution using Microsoft technologies that will ensure your integration solution is extensible, maintainable and inter-operable. Our experts will give you highlights of each technology by demonstrating how you can construct loosely-coupled, composite, and secure systems that bridge corporate firewalls based on core .NET Framework technologies, expose line of business applications in your IBM datacenters as web services using Host Integration Server, and orchestrate the activities of disparate web services into a composite application using Microsoft’s Business Process Management solution BizTalk Server. To solidify your learning, each session is followed by a lab that gives you hands on experience on the technologies just covered. These labs builds upon one another, so by the end of the day you will have the experience of building a real world Service Oriented solution using all the concepts and technologies you just learned. 

I am pretty sure they want you to register for the class and I think the tech ed site is at www.msteched.com

This year TechEd is hosting a Birds of a Feather for both the IT Pro and Developer audiences. Topics are then posted to the website where everyone can go in and vote on what their favorite sessions are.  On April 11th we’ll close down the voting, and the sessions with the most votes will be held at Tech Ed.  This tool was based on feedback we received at Tech Ed last year that attendees wanted to drive this more and be able to submit and choose what BOF topics were presented. 

TechEd BOF site: http://www.msteched.com/content/bof.aspx

 

Thanks

Lee

MSMQ Adapter Notes

Steve and
I wound up working on the same problem at the same time (probably for the same person…)

When working with the MSMQ adapter, keep in mind that you must reference the Microsoft.BizTalk.Adapter.MSMQ.MsmqAdapterProperties.dll
to have access to MSMQ-specific properties.  (The list of properties available
is in the adapter
documentation.)

Why? The intellisense in the expression shape (when using the parentheses syntax on
messages, ports, etc.) is looking for classes derived from Microsoft.XLANGs.BaseTypes.PropertyBase
to present in the drop down list.  Some of those classes are part of your “native”
BizTalk installation, some are provided by add-on adapters, and (of course) some are
provided by property schemas that you develop.  A reference to the containing
assembly is necessary to find them.

Other things to note relative to the MSMQ adapter:

  • For dynamic send ports, the syntax can look like: SomePort(Microsoft.XLANGs.BaseTypes.Address)
    = @”MSMQ://FORMATNAME:DIRECT=OS:SOMEMACHINE\PRIVATE$\SOMEQ”;
  • In the case of dynamic send ports, the runtime will use non-transactional sends by
    default.  If you want transactional sends, you will need to set the transactional
    property: outboundMsg(MSMQ.Transactional) = true;
  • Transactional messages which fail to deliver to the remote queue will be found in
    the local Transactional Dead Letter queue.  No error will be raised by the adapter. 
    This may mean that your design requires acknowledgement messages to achieve what you
    are looking for.

Hidden gem in SP1 … cleaning up the msgbox in a test environment

So now that Jean is posting, this blog should get pretty interesting. Jean is the man, behind the man, behind the man. 🙂 On the current BizTalk team he is about employee number 5 or so which makes him a true wealth of knowledge and I have been working with him for 5 1/2 years and he hasn’t killed me yet so he must have a lot of patience too. 🙂 Good to have him on board. 🙂 🙂

I have gotten many questions over time from people asking how to cleanup their messagebox when they are running in a *test* environment without having to reconfigure all of the databases. Sometimes you run a test and something fails and you get 1000s of suspended messages in the database that you don’t know what to do with (and HAT takes to long to terminate … which we are trying to fix). Well, we have had an internal tool for quite a while for doing this which I was finally given permission to include in SP1, but we hid it so here is my chance to let you know that it exists and how to use it (there is a KB article on it, but I haven’t found it … haven’t looked too hard 🙂 ).

In the <install dir>\schema directory you will find a file called msgbox_cleanup_logic.sql (it is installed with SP1). This file contains the definition for a stored procedure called bts_CleanupMsgbox. By default, this stored procedure exists in your msgbox except that the default implementation is empty so it does nothing. To use this script first run the .sql file against all msgboxes using Query Analyzer. This will simply create the stored procedure. It won’t actually do anything to your box as far as cleanup goes. Then:

1) Shut down all bts servers

2) If you are using HTTP or SOAP run IISRESET from the cmd prompt to recycle IIS and shutdown our out-or-process host instance

3) If you have any custom Isolated Adapters, make sure they are shutdown also.

4) run “exec bts_CleanupMsgbox” on all of your msgboxes

5) Restart everything and away you go

What will this do???

You can certainly read the stored procedure to see what we are doing (since you guys seem to read all the other ones I write :), but the summary is that this deletes all running instances and all information about those instances including state, messages, and subscriptions.It leaves all activation subscriptions so that you do not have to reenlist your orchestrations or sendports. Everything will just work and now you no longer have 50,000 instances sitting in a suspended state.

Couple of notes / gotchas:

1) If you install a hotfix onto your test system which runs msgboxlogic.sql, it will overwrite this with the empty stored procedure, so you will have to recreate it by rerunning the .sql file.

2) If you create a new msgbox it will by empty on the new msgbox and you will have to run it there also

3) THIS IS NOT SUPPORTED ON PRODUCTION SYSTEMS!!!! I have no idea why you would ever use this on a production box since this will delete all of your data but do not do it. Do not even run the cleanup_logic.sql file on your production system. Just don’t even think about it cause your data will disappear and I can’t imagine that you would be happy.

4) READ NUMBER 3. 🙂 🙂

5) This script does not actually delete all of the subscriptions. It marks them for deletion and then allows the subscripton cleanup job run by sql agent to take care of them. They will not be filled by the routing process since they have been “ghosted”, but if you have 100’s of thousands of subscriptons, this could be a little bit of overhead. Right now, for those situations, you just have to wait for the job to finish. Sorry. This is pretty rare scenario that has this happen. To make sure you are okay, after running the script, go to the management node in enterprise manager for your server and under jobs for sql agent, kick off the PurgeSubscriptionsJob_<msgboxname> job and wait for it to finish. Once it finishes, you are golden. This is very rare, but if you are worried about it, do this step to feel okay.

Besides that, you are golden. This is a very usefull script when rerunning tests after fixing an issue in your configuration. Do not abuse this script though and use it as a crutch for not fixing stress issues in your scenario. If you have issues which are causing the msgbox to overload and become unstable, they should be investigated because you do not want them to occur in your production system. Keep that in mind. Hopefully this script will make it easier for you to run tests efficiently.

 

Thx

Lee

Hidden gem in SP1 … cleaning up the msgbox in a test environment

So now that Jean is posting, this blog should get pretty interesting. Jean is the man, behind the man, behind the man. 🙂 On the current BizTalk team he is about employee number 5 or so which makes him a true wealth of knowledge and I have been working with him for 5 1/2 years and he hasn’t killed me yet so he must have a lot of patience too. 🙂 Good to have him on board. 🙂 🙂

I have gotten many questions over time from people asking how to cleanup their messagebox when they are running in a *test* environment without having to reconfigure all of the databases. Sometimes you run a test and something fails and you get 1000s of suspended messages in the database that you don’t know what to do with (and HAT takes to long to terminate … which we are trying to fix). Well, we have had an internal tool for quite a while for doing this which I was finally given permission to include in SP1, but we hid it so here is my chance to let you know that it exists and how to use it (there is a KB article on it, but I haven’t found it … haven’t looked too hard 🙂 ).

In the <install dir>\schema directory you will find a file called msgbox_cleanup_logic.sql (it is installed with SP1). This file contains the definition for a stored procedure called bts_CleanupMsgbox. By default, this stored procedure exists in your msgbox except that the default implementation is empty so it does nothing. To use this script first run the .sql file against all msgboxes using Query Analyzer. This will simply create the stored procedure. It won’t actually do anything to your box as far as cleanup goes. Then:

1) Shut down all bts servers

2) If you are using HTTP or SOAP run IISRESET from the cmd prompt to recycle IIS and shutdown our out-or-process host instance

3) If you have any custom Isolated Adapters, make sure they are shutdown also.

4) run “exec bts_CleanupMsgbox” on all of your msgboxes

5) Restart everything and away you go

What will this do???

You can certainly read the stored procedure to see what we are doing (since you guys seem to read all the other ones I write :), but the summary is that this deletes all running instances and all information about those instances including state, messages, and subscriptions.It leaves all activation subscriptions so that you do not have to reenlist your orchestrations or sendports. Everything will just work and now you no longer have 50,000 instances sitting in a suspended state.

Couple of notes / gotchas:

1) If you install a hotfix onto your test system which runs msgboxlogic.sql, it will overwrite this with the empty stored procedure, so you will have to recreate it by rerunning the .sql file.

2) If you create a new msgbox it will by empty on the new msgbox and you will have to run it there also

3) THIS IS NOT SUPPORTED ON PRODUCTION SYSTEMS!!!! I have no idea why you would ever use this on a production box since this will delete all of your data but do not do it. Do not even run the cleanup_logic.sql file on your production system. Just don’t even think about it cause your data will disappear and I can’t imagine that you would be happy.

4) READ NUMBER 3. 🙂 🙂

5) This script does not actually delete all of the subscriptions. It marks them for deletion and then allows the subscripton cleanup job run by sql agent to take care of them. They will not be filled by the routing process since they have been “ghosted”, but if you have 100’s of thousands of subscriptons, this could be a little bit of overhead. Right now, for those situations, you just have to wait for the job to finish. Sorry. This is pretty rare scenario that has this happen. To make sure you are okay, after running the script, go to the management node in enterprise manager for your server and under jobs for sql agent, kick off the PurgeSubscriptionsJob_<msgboxname> job and wait for it to finish. Once it finishes, you are golden. This is very rare, but if you are worried about it, do this step to feel okay.

Besides that, you are golden. This is a very usefull script when rerunning tests after fixing an issue in your configuration. Do not abuse this script though and use it as a crutch for not fixing stress issues in your scenario. If you have issues which are causing the msgbox to overload and become unstable, they should be investigated because you do not want them to occur in your production system. Keep that in mind. Hopefully this script will make it easier for you to run tests efficiently.

 

Thx

Lee

WMI Suspended Instances for BizTalk Server 2004

The help guide gives many excellent examples of using WMI to perform a large variety of useful tasks in BizTalk 2004. 



One of the little known features is the ability to have suspended messages saved to a file at the time of suspense.  I would assume this is govern by the same rules of viewing tracked messages in HAT, that is the stream must be seekable for the message to be tracked and in this case saved (see Christof’s Blog Post).



What makes this feature so nice is it saves both the message and the context to a file.  This can be great at tracking down routing failures, convoy problems, and general correlation issues. 



I have put together a simple tool that can be run anytime you want to view the suspended messages and context.  Just deploy the sample to your C:\ drive.  The suspended messages are written to C:\SampleWMI\OutputMessages.



Download: WMI Suspended Instance Sample


Our friend BTSSubscriptionViewer.exe…..

Anyone who has done anything with BizTalk has run into the “could not find a matching subscription error” at one time or another.  Keep in mind that under the covers, BizTalk operates under a Publish/Subscibe architecture.  What this message is telling us is that nothing has subscribed to our message with it’s current properties (namespace, root element, port, etc).  This is where the BTSSubscriptionViewer comes in.


This handy little, often forgot about tool can be found buried deep within the /Program Files/Microsoft BizTalk Server 2004/SDK/Utilities directory.  Once you launch the tool, select Load under the Main menu to load all the current subscriptions (remember, subscriptions can constantly change due to Convoy patterns, dehydrated orchestrations, etc).


The top of the window will display the current subscriptions, and bottom pane will display detailed information about the specific subscription.  So, how is this going to help us?  Well, if you find your orchestration in the top pane, chances are there is an issue with the subscriptions information.  For instance, perhaps your message has the wrong namespace, or root node, so it doesn’t match the subscription in the format of http://YourNameSpace#MessageType .  Keep this in mind when dealing with Multi-part messages.  I have seen people have the parts reversed, so when the subscription is created the body of the message type is pointing to the wrong body part.  Also, if you are using Correlation, this will be helpful to see what the correlation value has been set to (don’t forget to set your follow correlation set!!!).


So, remember, when your subscriptions are not found, launch the BTSSubscriptionViewer tool to find out what’s going on under the covers.