Goodbye DNJ

Goodbye DNJ

by | Sep 22, 2007 | BizTalk Community Blogs via Syndication

Well,after 3 years here its time to say goodbye to DNJ. Im moving to WordPress and the new link is http://santoshbenjamin.wordpress.com

I’ve had some great conversations here but im getting rather tied down by the poor editing facilities etc as i’ve said on many an occasion. I’ve also decided that its time for a change to befit my new status in life as i have explained on the wordpress site.

Anyway, im leaving the posts here as an archive (and also because i cant find any tools to move the content across. if you do know of any, do let me know). I will be visiting the aggregate site cos theres still some great content here.

Hope to see some of you at my wordpress site.

cheers,
benjy

Blogging Again

by | Sep 22, 2007 | BizTalk Community Blogs via Syndication

So its been a long absence from the blog world. Havent had much to write from a Biztalk perspective as i was only involved in the analysis and design phase for a new project that is now underway. But the other reason, definitely more important, is my change in status. Im now the proud dad […]

BizTalk Server 2006 – Enterprise Production Considerations – Part 3 – Clustering

by | Sep 22, 2007 | BizTalk Community Blogs via Syndication

I have had this recurring conversation with customers and partners.  Clustering is not a simple concept, and neither is BizTalk.  If you put them together you seem to have a perfect storm.  There’s lots of confusion, which I hope to be able to dispel with this post.

Also, I would like to have a link to send people, just so that I don’t have to have the conversation again 🙂

The Summary

  1. The SQL servers hosting the BizTalk databases can be clustered for high availability.
  2. The Enterprise Single-Sign On Master Secret Service (MSS) can be clustered for high availability.
  3. The BizTalk hosts can be clustered for the following reasons:
    1. They are hosting something (i.e. Receive Port) that has to be single instance, but still highly available.
    2. If you have BizTalk instances running on the same nodes that are running a clustered instance of the MSS.

FAQ

  1. What parts should I cluster? 
    You should cluster the SQL instance(s) holding the biztalk databases, the ENTSSO Master Secret service, and the BizTalk Hosts that require on a single instance running at a time, but still needs to be highly available.
  2. Is a non clustered-BizTalk host still Highly Available?
    If you have at least two host instances running in the same group for this host, then YES.  It behaves more like an NLB cluster than an MSCS service.

The Easy Part

I say that the SQL clustering is the “Easy Part” not because it is simple, but because I don’t feel compelled to explain the nitty gritty details in this post. I will give some of the BizTalk specifics here, and dedicate the next post to SQL cluster configuration. For now, as a rule, go with SQL 2005 Active/Active/Passive.

  1. Separate the MessageBox and Tracking Database among separate LUNs and separate SQL instances.
  2. The “Other” databases can often be place on the same node running the tracking DB.  If you are heavy on BAM or BAS you might want to break off the respective databases to a separate instance.
  3. All the databases should have at least 2 LUNS (data log), often they have more, but that gets into mount points and that is a tanget.

The Somewhat Easy Part

Clustering the ENTSSO MSS is pretty simple and the procedure is posted here. Basically, this is just a running service of the ENTSSO that also takes on the responsibility of distributing the Master Secret Key. This is the key that all the ENTSSO services running on other machines need to read the encrypted information in the SSO Database.

  1. This should be clustered.  Unless you don’t care about high availabilty. This can really mess up BizTalk if it goes down.
  2. This is not really a resource intensive service (typically) so it often piggy-backs on the SQL server nodes runing the clustered instances of SQL Server.
  3. You don’t want this clustered on the BizTalk boxes as a rule of thumb. (reasons covered later in this post)

Ok, now the meat.

BizTalk hosts generally have instances on more than one box, as such they are already highly available. This means that clustering a BizTalk host is not generally necessary to ensure high availability. If one goes down the others will pick up the work, out-of-the-box standard config.  The problem comes in when you REALLY don’t want to have mutliple instances of a host running at the same time.  The classic example of this is a receive port for an adapter that can not gracefully handle two threads reading it at the same time (i.e. FTP). It also happens when you need to be certain that all of the messages are picked up and processed sequentially and you don’t want to code around the problem.

So you want an FTP port receive, but you can not affort duplicate message reads.  That means you can have only one instance of the host, but that kills the out-of-the-box high availability.  Now you need to cluster that host instance, that is the only way to ensure that exactly one instance of the host will be running (so long as at least one of the BizTalk app servers is running)

The Very Wierd Part

This is the one that people scratch their heads over.  It does not happen much, and its not really all that complex.  But it is not intutitive, so it messes with people’s heads. 

For this to really make sense, I have to back up a step and explain how the ENTSSO service generally works in a mutli-box BizTalk installation.

  1. All of the BizTalk App servers have an instance of the ENTSSO service running on the machine. It is this service that obtains the secret key from the ENTSSO Master Secret Server and then uses it to independently access the info in the SSO Database.
  2. One of the ENTSSO servers is setup as the Master Secret Server. This means that it will distribute the secret key to the other ENTSSO service instances, in addition to being the standard ENTSSO service for the local box.  This is the service instances that needs to be configured for High Availablity because all of the other ENTSSO service instances depend on it.
  3. The BizTalk host instances have a dependency on the ENTSSO Service running on the same box as them. This is because they need desperately need it to give them info from the SSO DB. This information is typically configurations for their ports that is stored in the SSO DB for safe keeping.

Figure 1 – Standard Multi-Box ENTSSO Strategy

Lets say that you have only 2 production servers. They will be running both SQL and BizTalk.  Like a good boy/girl you decide to cluster SQL and the ENTSSO MSS.

Now you try to configure BizTalk host instances, and you find that they don’t want to work quite right. That is because they have a dependency on the ENTSSO service, they always need it.  Now that the ENTSSO service is clustered, only the host instances running on the active (ENTSSO) node actually have access to the service they need. All the other instances will fail based on dependencies.

Figure 2 – BizTalk Hosts with Broken Dependency

An an extra bonus you have messed up the High availability of the MSS as well. The clustering service detects that the host instances are depending on the ENTSSO MSS service and probably will not allow it to failover to other nodes.

…NICE…

Option 1 is to not cluster ENTSSO MSS in a 2 box scenario

Bottom line is that, if you have only budget for two prod boxes, you probably don’t want to cluster the ENTSSO MSS. But this is really the only time I can think of that clustering that service is bad.  If it is not clustered you better be a “Johnny on the spot” with MOM and implement a way to have MOM do a poor man’s failover when it detects that the ENTSSO MSS server is down.

Option 2 is to also cluster all BizTalk host instances that depend on a clustered ENTSSO service.

If you really do want to cluster the ENTSSO MSS, then you also have to cluster all of the BizTalk hosts as well. This ensures that as the ENTSSO Service “flips” from one box to the next, the Microsoft Clustering Service can ensure that the BizTalk host instances flip at the same time and don’t come crashing down.

BizTalk Server 2006 – Enterprise Production Considerations – Part 3 – Clustering

by | Sep 21, 2007 | BizTalk Community Blogs via Syndication

I have had this recurring conversation with customers and partners.  Clustering is not a simple concept, and neither is BizTalk.  If you put them together you seem to have a perfect storm.  There’s lots of confusion, which I hope to be able to dispel with this post.

Also, I would like to have a link to send people, just so that I don’t have to have the conversation again 🙂

The Summary

  1. The SQL servers hosting the BizTalk databases can be clustered for high availability.
  2. The Enterprise Single-Sign On Master Secret Service (MSS) can be clustered for high availability.
  3. The BizTalk hosts can be clustered for the following reasons:
    1. They are hosting something (i.e. Receive Port) that has to be single instance, but still highly available.
    2. If you have BizTalk instances running on the same nodes that are running a clustered instance of the MSS.

FAQ

  1. What parts should I cluster? 
    You should cluster the SQL instance(s) holding the biztalk databases, the ENTSSO Master Secret service, and the BizTalk Hosts that require on a single instance running at a time, but still needs to be highly available.
  2. Is a non clustered-BizTalk host still Highly Available?
    If you have at least two host instances running in the same group for this host, then YES.  It behaves more like an NLB cluster than an MSCS service.

The Easy Part

I say that the SQL clustering is the “Easy Part” not because it is simple, but because I don’t feel compelled to explain the nitty gritty details in this post. I will give some of the BizTalk specifics here, and dedicate the next post to SQL cluster configuration. For now, as a rule, go with SQL 2005 Active/Active/Passive.

  1. Separate the MessageBox and Tracking Database among separate LUNs and separate SQL instances.
  2. The “Other” databases can often be place on the same node running the tracking DB.  If you are heavy on BAM or BAS you might want to break off the respective databases to a separate instance.
  3. All the databases should have at least 2 LUNS (data log), often they have more, but that gets into mount points and that is a tanget.

The Somewhat Easy Part

Clustering the ENTSSO MSS is pretty simple and the procedure is posted here. Basically, this is just a running service of the ENTSSO that also takes on the responsibility of distributing the Master Secret Key. This is the key that all the ENTSSO services running on other machines need to read the encrypted information in the SSO Database.

  1. This should be clustered.  Unless you don’t care about high availabilty. This can really mess up BizTalk if it goes down.
  2. This is not really a resource intensive service (typically) so it often piggy-backs on the SQL server nodes runing the clustered instances of SQL Server.
  3. You don’t want this clustered on the BizTalk boxes as a rule of thumb. (reasons covered later in this post)

Ok, now the meat.

BizTalk hosts generally have instances on more than one box, as such they are already highly available. This means that clustering a BizTalk host is not generally necessary to ensure high availability. If one goes down the others will pick up the work, out-of-the-box standard config.  The problem comes in when you REALLY don’t want to have mutliple instances of a host running at the same time.  The classic example of this is a receive port for an adapter that can not gracefully handle two threads reading it at the same time (i.e. FTP). It also happens when you need to be certain that all of the messages are picked up and processed sequentially and you don’t want to code around the problem.

So you want an FTP port receive, but you can not affort duplicate message reads.  That means you can have only one instance of the host, but that kills the out-of-the-box high availability.  Now you need to cluster that host instance, that is the only way to ensure that exactly one instance of the host will be running (so long as at least one of the BizTalk app servers is running)

The Very Wierd Part

This is the one that people scratch their heads over.  It does not happen much, and its not really all that complex.  But it is not intutitive, so it messes with people’s heads. 

For this to really make sense, I have to back up a step and explain how the ENTSSO service generally works in a mutli-box BizTalk installation.

  1. All of the BizTalk App servers have an instance of the ENTSSO service running on the machine. It is this service that obtains the secret key from the ENTSSO Master Secret Server and then uses it to independently access the info in the SSO Database.
  2. One of the ENTSSO servers is setup as the Master Secret Server. This means that it will distribute the secret key to the other ENTSSO service instances, in addition to being the standard ENTSSO service for the local box.  This is the service instances that needs to be configured for High Availablity because all of the other ENTSSO service instances depend on it.
  3. The BizTalk host instances have a dependency on the ENTSSO Service running on the same box as them. This is because they need desperately need it to give them info from the SSO DB. This information is typically configurations for their ports that is stored in the SSO DB for safe keeping.

Figure 1 – Standard Multi-Box ENTSSO Strategy

Lets say that you have only 2 production servers. They will be running both SQL and BizTalk.  Like a good boy/girl you decide to cluster SQL and the ENTSSO MSS.

Now you try to configure BizTalk host instances, and you find that they don’t want to work quite right. That is because they have a dependency on the ENTSSO service, they always need it.  Now that the ENTSSO service is clustered, only the host instances running on the active (ENTSSO) node actually have access to the service they need. All the other instances will fail based on dependencies.

Figure 2 – BizTalk Hosts with Broken Dependency

An an extra bonus you have messed up the High availability of the MSS as well. The clustering service detects that the host instances are depending on the ENTSSO MSS service and probably will not allow it to failover to other nodes.

…NICE…

Option 1 is to not cluster ENTSSO MSS in a 2 box scenario

Bottom line is that, if you have only budget for two prod boxes, you probably don’t want to cluster the ENTSSO MSS. But this is really the only time I can think of that clustering that service is bad.  If it is not clustered you better be a “Johnny on the spot” with MOM and implement a way to have MOM do a poor man’s failover when it detects that the ENTSSO MSS server is down.

Option 2 is to also cluster all BizTalk host instances that depend on a clustered ENTSSO service.

If you really do want to cluster the ENTSSO MSS, then you also have to cluster all of the BizTalk hosts as well. This ensures that as the ENTSSO Service “flips” from one box to the next, the Microsoft Clustering Service can ensure that the BizTalk host instances flip at the same time and don’t come crashing down.

How to manually recreate missing BizTalk performance counters

by | Sep 21, 2007 | BizTalk Community Blogs via Syndication

To identify performance issues in a BizTalk solution, you should rely on the performance counters that BizTalk Server uses. All the BizTalk performance counters should be  added to the registry when you run the configuration wizard. If the registry keys were overwritten somehow, those performance counters were gone.  You could use the configuration wizard to recreate those missing BizTalk performance counters, but it requires reconfiguring the features that use the performance counters. To do so you should unconfigure them first, which is something you might not want to do.  Even though it is undocumented, you can manually populate those resister keys for the missing performance counters.

There are two types of the performance libraries that BizTalk Server uses:

1. BTSPerfMonExt.dll for the performance objects that the unmanaged code of BizTalk Server uses.

2. netfxperf.dll for the performance objects that the managed code of BizTalk Server uses.

  • Note: netfxperf.dll is a .NET framework component. It is shared for all .NET performance counters. For more information about netfxperf.dll see .NET Performance Counters.

The following tables shows which performance library is used for each BizTalk performance object:

Object Performance Counter Library Installer
BizTalk:Message Box:* netfxperf.dll Microsoft.BizTalk.MsgBoxPerfCounters.dll

BizTalk:Message Agent
BizTalk:File Receive Adapter
BizTalk:File Send Adapter
BizTalk:HTTP Receive Adapter
BizTalk:HTTP Send Adapter
BizTalk:POP3 Receive Adapter
BizTalk:FTP Receive Adapter
BizTalk:FTP Send Adapter
BizTalk:MSMQ Receive Adapter
BizTalk:MSMQ Send Adapter
BizTalk:SOAP Receive Adapter
BizTalk:SOAP Send Adapter
BizTalk:SMTP Send Adapter
BizTalk:SQL Receive Adapter
BizTalk:SQL Send Adapter
BizTalk:Messaging
BizTalk:Messaging Latency

 BTSPerfMonExt.dll BTSPerfMonExt.ini
BTSPerfMonExt.h
BizTalk: BAS TPM Management Web Service
 netfxperf.dll Microsoft.BizTalk.KwTpm.TPMgmtWSPerf.dll
BizTalk: BAS TPM Publishing Web Service netfxperf.dll Microsoft.BizTalk.KwTpm.TPPubWSPerf.dll
BizTalk:Windows SharePoint Services Adapter netfxperf.dll Microsoft.BizTalk.KwTpm.WssAdapter.Runtime.dll
BizTalk:TDDS netfxperf.dll Microsoft.BizTalk.Bam.EventObservation.dll

 

To recreate the performance counters using netfxperf.dll, you can use InstallUtil.exe in the .NET Framework. If the assembly is installed in the GAC, you can run InstallUtil.exe as follows:

Installutil /i /assemblyname “<InstallerFileName>, Version=<VersionNumOfTheInstaller>, Culture=neutral,
PublicKeyToken=<PublicKeyOfTheInstaller>“

For example,

InstallUtil /i /assemblyname “Microsoft.BizTalk.MsgBoxPerfCounters, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35”

If it is not, you can run InstallUtil.exe as follows:

InstallUtil /i <Installer>

For example,

InstallUtil /i Microsoft.BizTalk.KwTpm.TPMgmtWS.dll

 

To recreate the performance counters using BTSPerfMonExt.dll,  you can use lodctr.exe and unlodctr.exe as follows:

1. Copy the BTSPerfMonExt.ini and BTSPerfMonExt.h files to the %windir%\system32 folder.
2. At a command prompt, go to the the %windir%\system32 folder, and then type the following command to uninstall the damaged performance counters.

unlodctr “BTSSvc.3.0”

3. Then, type the following command to reinstall the BizTalk performance counters:

lodctr BTSPerfMonExt.ini

Thanks,
Young

How to Promote Properties from an Orchestration

by | Sep 21, 2007 | BizTalk Community Blogs via Syndication

“The best reason to blog something is so that you can Google your own brain.”

— Poorly paraphrased Scott Hanselman, yet so very true.

In that spirit, today I’m posting a reminder of a very useful trick for promoting

properties out of Orchestrations. Kudos go to Saravana

Kumar and David

Hurtadofor posting about this long before I did, but I record this information

here to make sure I can find it when next I need it.

Normally when you assign a value to a message context property from within an Orchestration,

that property is not promoted, even if it was promoted prior to you assigning a value

to it. In order to promote a property within an orchestration we use an “elegant

hack” as detailed below:

  1. Create a new Correlation Set, and as such likely a new Correlation Type, which contains

    the property or properties which you wish to promote.

  2. When you use a “Send” shape in your orchestration, there is a property called “Initializing

    Correlation Sets”, check off your new Correlation Set from the drop down on this property.

  3. Rejoice, you have now promoted those properties.

Now before anyone asks, no this does not create orphaned subscriptions in the Message

Box, we are never creating a Receive using this Correlation Set so

we’re never creating a subscription. This can be very useful, in particular

I’ve used it just this week on a 2006 R2 project I’m working on to enable batching

from an Orchestration. That is a long story, but the short version is we’re creating

a new message in the Orchestration which we then intend to immediately queue for batching,

so we need to promote these properties without ever having passed through a pipeline

with this message.

So it begins

by | Sep 20, 2007 | BizTalk Community Blogs via Syndication

As the saying goes, there’s always a first time, and this blog entry is a first for me.
My name is Thiago Almeida, I work for one of New Zealand’s largest IT companies as one of their senior integration specialists using Microsoft technologies. I work most of the time with Microsoft BizTalk Server (Mainly 2006 and […]