BizTalk Health Monitor Dashboards Customization: Monitoring BizTalk Host Instances Status

BizTalk Health Monitor Dashboards Customization: Monitoring BizTalk Host Instances Status

Have you noticed that the default BizTalk Health Monitor Dashboard doesn’t monitor/report the status of the BizTalk Server Host Instances?

A few weeks ago, while delivering a BizTalk Server training course, I was presenting and playing around with the BizTalk Health Monitor with my “students” and explaining the importance of actively monitoring your BizTalk Server platform. Because my developer environment is a machine that doesn’t have many errors and is properly configured, like a production environment, it is difficult to show the BizTalk Health Monitor presenting problems, so I decided to stop some of the Host Instances! It is an easy way to “emulate” a problem:

However, when I ran the BizTalk Health Monitor, I realized the Host Instance tile was green!

Notice also that the information provided states that I have 10 Host Instances, and only 8 were running. I was surprised by that behavior. I confirmed with the Microsoft Engineering team, and this is not a bug. Actually, by default, the Host Instances dashboard tile is NOT supposed to report a warning if some Host Instances are stopped. this tile reports, in fact, what MSFT has in the category “BizTalk Host Instance” of the KPI view:

Each default Dashboard tile normally reports the content of one or more categories of different views (Warnings, Summary, Key indicators, Queries Output…); However, the choice of content of these tiles cannot be changed by us.

Now the main question is: Can we put the BizTalk Server Health Monitor “watching” the status of the Host Instances and raising some alerts?

Luckily for us, the answer is yes, and it is not that difficult. The tool allows us to:

  • Add custom rules that will allow us to query the environment.
  • And add our own custom tiles in our profile Dashboard view.

That also means that each profile can have there own monitoring customizations.

To create some custom rules to monitor the status of the Host Instances, assuming that you have the BizTalk Health monitor tool open, we need to:

  • Right-click over the Default profile, and select the Profile settings… option from the context menu.
  • From the Default profile – Profile settings window, select the Rules tab and then click on New rule.
  • From the New Rule – Select Query window, expand the Important target category, then select the BizTalk Host Instances sub-category, and click Ok.
  • On the New Rule (Query: BizTalk Host Instances) window, select the My Rule option from the left tree and:
    • On the Caption property, give a name to the rule: Stopped Host Instances.
    • On the Comment property, provide a small description: Monitoring BizTalk Server Host Instances status.
    • On the Trigger Actions panel, select the Each time a row validated all the Rule conditions option.
    • And click Commit changes.
  • On the New Rule (Query: BizTalk Host Instances) window, select the Condition 1 option from the left tree and:
    • On the Column to Check property, leave the default value: %GLOBALPROP_REPORTVALUE:Running%.
    • On the Operator property, from dropbox, select the option: IS DIFFERENT OF.
    • On the Comparison value property, type Yes.
    • And click Commit changes.
  • On the New Rule (Query: BizTalk Host Instances) window, on the Condition option from the left tree, right-click and select the option New Condition:
  • On the New Rule (Query: BizTalk Host Instances) window, select the Condition 2 option from the left tree and:
    • On the Column to Check property, leave the default value: %GLOBALPROP_REPORTVALUE:Running%.
    • On the Operator property, from dropbox, select the option: IS DIFFERENT OF.
    • On the Comparison value property, type Not Applicable.
    • And click Commit changes.
  • On the New Rule (Query: BizTalk Host Instances) window, select the Add Summary or Warning Entry option from the left tree under the Actions option and:
    • On the Category property, type: Host Instances.
    • On the Severity property dropbox, select the Red Warning option.
    • On the Caption property, type: Host Instances Status.
    • On the Value property, type: Is %GLOBALPROP_REPORTVALUE:Name% running: %GLOBALPROP_REPORTVALUE:Running%
    • And click Commit changes.
  • Finally, click Ok.

You can always try the custom rule by clicking Test.

This rule will gather information about Host Instances that are not running. Now we are going to create another rule to gather information about Host Instances that are running. To do that, we need to:

  • From the Default profile – Profile settings window, select the Rules tab and then click on New rule.
  • From the New Rule – Select Query window, expand the Important target category, then select the BizTalk Host Instances sub-category, and click Ok.
  • On the New Rule (Query: BizTalk Host Instances) window, select the My Rule option from the left tree and:
    • On the Caption property, give a name to the rule: Running Host Instances.
    • On the Comment property, provide a small description: Monitor Running Host Instances status.
    • On the Trigger Actions panel, select the Each time a row validated all the Rule conditions option.
    • And click Commit changes.
  • On the New Rule (Query: BizTalk Host Instances) window, select the Condition 1 option from the left tree and:
    • On the Column to Check property, leave the default value: %GLOBALPROP_REPORTVALUE:Running%.
    • On the Operator property, from dropbox, select the option: IS EQUAL TO.
    • On the Comparison value property, type Yes.
    • And click Commit changes.
  • On the New Rule (Query: BizTalk Host Instances) window, select the Add Summary or Warning Entry option from the left tree under the Actions option and:
    • On the Category property, type: Host Instances.
    • On the Severity property dropbox, select the Information option.
    • On the Caption property, type: Host Instances Status.
    • On the Value property, type: Is %GLOBALPROP_REPORTVALUE:Name% running: %GLOBALPROP_REPORTVALUE:Running%
    • And click Commit changes.
  • Finally, click Ok.

Make sure that the two custom rules are selected, and then perform another analysis of your platform.

Now, what we need to do is to create a custom tile to pin to our dashboard. A custom tile can be indeed created easily from any entry or category of the Warning view, Summary view, Key indicators view, or query output view. And to do that, we need to:

  • After analyzing our BizTalk Server environment, expand the report and then select the Summary option.
  • On the Summary report page, scroll down until you find the Host Instances summary, right-click on Host Instances and select the option Pin to the dashboard and then .
  • A new window will appear, saying that a new item was added to the dashboard. Click Ok.
  • If we now click on the Default profile, we will see that the Favorite tile was added to the dashboard.
  • We can customize the name of that tile by right-clicking and selecting the Edit option.
  • On the Favorite Tile – Favorite window:
    • On the Caption property, type: Host Instances Status.
    • On the Comment property, type: Host Instances Status.
    • And click Ok.

And finally, test it by doing another analysis of the environment.

How amazing is this!

Thanks to all that helped me document this feature. You know how you are!

Hope you find this useful! So, if you liked the content or found it useful and want to help me write more content, you can buy (or help buy) my son a Star Wars Lego! 

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.

He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.
View all posts by Sandro Pereira

BHM Load Error: Could not load assembly file ‘file:///C:Program files (x86)BizTalkHealthMonitorMYHCQueries_MBVQueries.dll’ or one of its dependencies

BHM Load Error: Could not load assembly file ‘file:///C:Program files (x86)BizTalkHealthMonitorMYHCQueries_MBVQueries.dll’ or one of its dependencies

This BHM load error regarding to MYHCQueries_MBVQueries.dll, that can may also occurs with MaintenanceRep.dll, is a very well-known problem/annoying issue that occurs after we install BizTalk Health Monitor or update it to a new version that will apply in all BizTalk versions (BizTalk Server 2016, 2013 R2, 2013 or 2010) and SO (Windows 10, Windows Server 2016, 2012 R2, 2012 and so on):

Could not load assembly file ‘file:///C:Program files (x86)BizTalkHealthMonitorMYHCQueries_MBVQueries.dll’ or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)

BHM Could not load assembly MYHCQueries_MBVQueries.dll

CAUSE

That is one of that times that BizTalk Server team did a great job. Not only throw the error but they also explain the problem and the solution to solve it.

This error when BHM tries to load the repository files downloaded from the Microsoft Download Center (DLC). BHM is using two repositories for this configuration, these repository files (MYHCQueries_MBVQueries.dll and MaintenanceRep.dll) can be blocked by OS when downloading from DLC and may cause this issue.

Solution

Again, the solution to this problem is also explained in the error window:

  • Close BHM
  • Go to BHM installation folder
    • The default path is C:Program files (x86)BizTalkHealthMonitor
  • Right-click in MYHCQueries_MBVQueries.dll and select “Properties” option

Select MYHCQueries_MBVQueries.dll Properties

  • You may see that the security warning at the bottom of the windows saying: “The file came from another computer and might be blocked to help protect this computer”
    • Check “Unblock” option and click “OK”

MYHCQueries_MBVQueries.dll Properties Security Warnings

  • Do the same steps for “MaintenanceRep.dll”

Once you have unblocked both files, you should no longer see the security warning when you go back into their properties and if you try to execute BHM again all the above-mentioned errors related to loading repositories should be resolved.

You may also find more details about this issue directly from BizTalk Health Monitor blog: BHM v3.2 – Errors While Loading Repositories.

Note:

  • If you have not yet installed BHM, you can also follow the above steps on the unzipped package prior to installing BHM.
Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community. View all posts by Sandro Pereira

Final Version of BizTalk Terminator Tool Released

I just wanted to announce that the final version of the standalone BizTalk Terminator tool – v2.5 – has just been released.

Terminator functionality is not going away.
All further development will continue via the maintenance node in BizTalk Health Monitor (BHM).

Below is a quick comparison of the two versions of Terminator:

Tool Version Standalone Terminator BHM Terminator
Available as a standalone tool maintenance node within BHM
Recommended for BizTalk 2004 – 2009 BizTalk 2010 and above
Supports BizTalk 2004 – 2013 R2 BizTalk 2010 and above
Future Tool Updates No
Version 2.5 is the final version.
Support for BTS2016 or above will not be added.
Yes
All tool updates happen here.
BTS2016 support just added in BHM v4
Future Task Repository Updates Yes
Via Update Task Repository menu item (see below).
Only repository updates for BTS2013R2 or below.
Yes
Via auto-update mechanism.
All repository updates will apply
Tool Expiration No more time bomb with v2.5 No

Update Task Repository Menu Item in Standalone Terminator

  • When you open Terminator, click the Help menu at the top left and you will see the Update Task Repository menu item.
  • Clicking this does the following:
    • Renames the existing MaintenanceRep.dll (located in the same folder as Terminator) to MaintenanceRep[DATETIME].dll
    • Downloads the current MaintenanceRep.dll from Azure
  • This feature requires external internet access as well as .NET 4.0 or above for Azure connectivity.  If you don’t have either of those on your BizTalk machine, you will need to run Terminator on a box that does and then just copy over the new MaintenanceRep.dll onto your BizTalk machine.  The box where you do this does not need to have access to BizTalk.
  • The Update Task Repository menu item is disabled once Terminator connects to a BizTalk environment.  You will need to close and re-open Terminator for it to be re-enabled.
  • See here for a list of repository updates released so far (only the Maintenance Repository is relevant to Terminator)
BizTalk Terminator STILL Not Cleaning Up Caching Items?

BizTalk Terminator STILL Not Cleaning Up Caching Items?

I originally published my BizTalk Terminator Not Cleaning Up Caching Items blog back in 2011 where I suggested to use the Terminate Multiple Instances (Hard Termination) task if the normal WMI-based Terminate Cache Instances task doesn’t clean up all the caching instances.

Since then, I’ve come across a caching cleanup scenario that even the hard termination task can’t handle.  Basically, if there are cache message refs (in <HOST>Q or <HOST>Q_Suspended tables) that are missing their associated caching instance (in the Instances table), then even the hard termination task won’t be able to clean them up.  The reason is that the Terminate Multiple Instances (Hard Termination) task finds the instances for termination by looking through the Instances table – so in this case it never sees the orphaned cache message refs.

If you run into this scenario, what you will see is that that once Hard Termination has successfully cleaned up all caching instances, BHM or MBV will continue to complain about cache messages and if you run the View Count of Cache Messages in All Host Queues task, it will show Cache Messages and Message Refs existing even though there are no Associated Cache Instances:

The workaround so far has been to to first run the DELETE Orphaned Messages In All Hosts task (to clean up the orphaned message refs) and then run the Terminate Multiple Instances (Hard Termination) task as mentioned in the previous blog (to clean up the cache instances).

I have also created a new cache cleanup task called Terminate Caching Items (Hard Termination) This task checks all the <HOST>Q, <HOST>Q_Suspended, and Instances tables for caching items associated with any host and in any state and cleans them all up.  The purpose of this new task is to handle all cache item cleanup scenarios with one step – so the user doesn’t have to dig into the state of the cache items in their environment to know how to clean them up.  So if BHM or MBV complain about cache items, simply running this task on the correct MessageBox should take care of it. 

This new task is available in BHM v3.1 and Terminator v2.2.

If you were not already aware that Terminator has been integrated into the BizTalk Health Monitor (BHM) tool, see here.

BizTalk Terminator – Now Available via BHM

BizTalk Terminator – Now Available via BHM

I just wanted to let everyone know that BizTalk Terminator is now also available as a part of BizTalk Health Monitor (BHM) – the new BizTalk tool that integrates MBV and Terminator functionality into an MMC Snap-In that you can run as a part of your BizTalk Admin Console.

Update:  Final version of standalone Terminator released.
More info here – including a comparison of the 2 versions of Terminator.

Terminator functionality can be found in BHM’s Maintenance node:

  • You must be on BHM v3 or above to get the Terminator functionality
  • Get BHM from https://www.microsoft.com/en-us/download/details.aspx?id=43716
  • See here for a walkthrough of BHM v3 functionality – including Terminator integration.
  • See the BHM blog at http://blogs.msdn.com/b/biztalkhealthmonitor for more information on BHM.
  • So far, I’ve included the most popular Terminator tasks in BHM and plan to add more over time
  • BHM officially supports BizTalk 2010, 2013, 2013R2, and 2016.
  • I’m going to continue updating standalone Terminator for now and it is still available here. 
  • At some point, I’ll probably end up creating a final build of the standalone version and move all new development to BHM.  I’ll update this blog once that decision is made.
  • Standalone Terminator tool is no longer being updatedThe final version of standalone Terminator is v2.5.  All new development (including BTS2016 support) is happening in BHM Terminator. 
  • MBV is no longer being updated and the last update to it was in July 2014.  To make sure you are using the latest MBV repository with the latest rules/queries, be sure to move to BHM.
BizTalk Terminator Not Cleaning Up Caching Items?

BizTalk Terminator Not Cleaning Up Caching Items?

I’ve been pinged a number of times on this so thought I should blog the workaround and an explanation. 

First, let’s say MBV shows you something like the following in the Warning and Summary Report:

Or you just notice in MBV that there’s a bunch of cache messages in one of the queue tables:

Well, according to my Using BizTalk Terminator to Resolve Issues article, you simply run the Terminate Caching Instances task: 

Issue Identified by MBV

Resolution Options

Terminator Resolution Task

Terminator View Task

Root Cause

Orphaned Cache Instances

MBV Integration or Manual Task Selection

Terminate Caching Instances

(in Delete task category)

View Count of Cache Messages in All Host Queues

View Count of Cache Instances in All Hosts

This is due to a known bug and there is a hotfix available.  See KBs 944426 & 936536 for details.

That should do it.  If it doesn’t, make sure you have stopped all the BizTalk hosts (that includes the IIS app pool hosting the BizTalk isolated host if the caching items are there) and try again.  (Hey, you shouldn’t be running Terminator without stopping all the BTS hosts anyway).

Now, that will definitely do it.

Well… like 99.9% of the time.

Let’s say you do all of the above and then run one of those View tasks (or MBV)  and notice that Terminator left behind some of those caching instances (and their associated caching messages).  This happens even though Terminator claims to have terminated all of them successfully.  And rerunning the task doesn’t help – Terminator will repeatedly say it successfully terminated those instances but either of the View tasks will show that they’re still there.

Ok, so now you’re most likely running into a very rare scenario that I’ve come across a few times. 

First, I should point out that the Terminate Caching Instances and Terminate Instances tasks use BizTalk’s WMI API to interact with the messagebox – and that’s key.  I had the opportunity to analyze some data from a customer environment that was running into this issue and it turns out that there are certain times when msgbox logic prevents the stored procs called by WMI from terminating “internal” instances – with caching being considered one of those “internal” types.  As far as when exactly the msgbox logic goes down this “rare” path, I’ve never had access to a repro environment where I could fully debug this so I don’t have a good answer for that.

So I was going to write code in Terminator’s WMI class to catch this scenario and warn the user that they need to use the workaround to clean up the remaining items but unfortunately the msgbox logic catches the failed call and doesn’t pass that info on to the WMI caller – so there’s no way for Terminator (or any WMI client) to know that some of the instances weren’t deleted.  As far as the WMI client is concerned, the stored proc call completed sucessfully so it assumes the instances it asked to be Terminated were actually terminated.

So what’s the workaround?  Well, don’t use WMI for this particular situation.  Instead, use Terminator’s Hard Termination tasks (Terminate Multiple Instances (Hard Termination) or Terminate Single Instance (Hard Termination).

While I was working on Terminator’s WMI class and having BizTalk engineers use Terminator on an internal-only basis within Microsoft, we noticed that on very rare occasions the termination tasks would not terminate something.  This would not be a limitation of Terminator and we could reproduce it with any WMI client (including the BTS Admin console).  I created the hard termination tasks specifically to handle those scenarios.  They bypass BizTalk’s normal termination API and use SQL calls to directly interact with the BizTalk msgbox.  They can terminate anything (well, so far) – even internal instances.  Originally, we just had the Terminate Single Instance (Hard Termination) task to help terminate a one-off instance that just wouldn’t terminate any other way.  That worked great for those one-off scenarios but I soon realized that sometimes there would be a larger number of instances that needed hard termination.  I left the single instance task to give users that functionality and wrote the Terminate Multiple Instances (Hard Termination) task.  That allows the user to choose the Host, Class, Status, and the Max number of instances to terminate and will do hard terminates on all items that fit the filter criteria. 

The only thing that is painful about the Terminate Multiple Instances (Hard Termination) task is that if you have instances across multiple hosts with various statuses, you will need to run the task for each permutation since it doesn’t have the ability to handle, in one execution, multiple hosts and statuses (or classes) like the WMI-based Terminate Instances and Terminate Caching Instances tasks.  Since Terminator v2 supports Powershell, I’m hoping at some point to create a powershell-based hard terminate task that can provide this functionality – just haven’t had the time to do that yet.

So, in short:

Problem:

The Terminate Caching Instances task is not cleaning up all cache items.

Solution:

Use the Terminate Multiple Instances (Hard Termination) task, choose Caching as the Class Parameter. 

You will need to re-run this task for each Host and Status that applies to the cache items you’re trying to terminate – MBV or the Terminator View Count of Cache tasks should give you some info in this regard. 

BTW, I’ve seen active, dehydrated, and suspended cache items so you may need to cycle through all of those Statuses.

In general, if you ever find that any of the termination tasks are not terminating what you want, the two Hard Termination tasks (Single and Multiple) are the workaround. 

Remember that Hard Termination tasks bypass the normal BTS APIs so should only be used if the normal Terminate Instances or Terminate Caching Instances task is not working – and as always, be careful with Terminator – especially when doing any of the deletion tasks.