I just wanted to announce that the final version of the standalone BizTalk Terminator tool – v2.5 – has just been released.
Terminator functionality is not going away.
All further development will continue via the maintenance node in BizTalk Health Monitor (BHM).
Below is a quick comparison of the two versions of Terminator:
||a standalone tool
||maintenance node within BHM
||BizTalk 2004 – 2009
||BizTalk 2010 and above
||BizTalk 2004 – 2013 R2
||BizTalk 2010 and above
|Future Tool Updates
Version 2.5 is the final version.
Support for BTS2016 or above will not be added.
All tool updates happen here.
BTS2016 support just added in BHM v4
|Future Task Repository Updates
Via Update Task Repository menu item (see below).
Only repository updates for BTS2013R2 or below.
Via auto-update mechanism.
All repository updates will apply
||No more time bomb with v2.5
Update Task Repository Menu Item in Standalone Terminator
- When you open Terminator, click the Help menu at the top left and you will see the Update Task Repository menu item.
- Clicking this does the following:
- Renames the existing MaintenanceRep.dll (located in the same folder as Terminator) to MaintenanceRep[DATETIME].dll
- Downloads the current MaintenanceRep.dll from Azure
- This feature requires external internet access as well as .NET 4.0 or above for Azure connectivity. If you don’t have either of those on your BizTalk machine, you will need to run Terminator on a box that does and then just copy over the new MaintenanceRep.dll onto your BizTalk machine. The box where you do this does not need to have access to BizTalk.
- The Update Task Repository menu item is disabled once Terminator connects to a BizTalk environment. You will need to close and re-open Terminator for it to be re-enabled.
- See here for a list of repository updates released so far (only the Maintenance Repository is relevant to Terminator)
I originally published my BizTalk Terminator Not Cleaning Up Caching Items blog back in 2011 where I suggested to use the Terminate Multiple Instances (Hard Termination) task if the normal WMI-based Terminate Cache Instances task doesn’t clean up all the caching instances.
Since then, I’ve come across a caching cleanup scenario that even the hard termination task can’t handle. Basically, if there are cache message refs (in <HOST>Q or <HOST>Q_Suspended tables) that are missing their associated caching instance (in the Instances table), then even the hard termination task won’t be able to clean them up. The reason is that the Terminate Multiple Instances (Hard Termination) task finds the instances for termination by looking through the Instances table – so in this case it never sees the orphaned cache message refs.
If you run into this scenario, what you will see is that that once Hard Termination has successfully cleaned up all caching instances, BHM or MBV will continue to complain about cache messages and if you run the View Count of Cache Messages in All Host Queues task, it will show Cache Messages and Message Refs existing even though there are no Associated Cache Instances:
The workaround so far has been to to first run the DELETE Orphaned Messages In All Hosts task (to clean up the orphaned message refs) and then run the Terminate Multiple Instances (Hard Termination) task as mentioned in the previous blog (to clean up the cache instances).
I have also created a new cache cleanup task called Terminate Caching Items (Hard Termination). This task checks all the <HOST>Q, <HOST>Q_Suspended, and Instances tables for caching items associated with any host and in any state and cleans them all up. The purpose of this new task is to handle all cache item cleanup scenarios with one step – so the user doesn’t have to dig into the state of the cache items in their environment to know how to clean them up. So if BHM or MBV complain about cache items, simply running this task on the correct MessageBox should take care of it.
This new task is available in BHM v3.1 and Terminator v2.2.
If you were not already aware that Terminator has been integrated into the BizTalk Health Monitor (BHM) tool, see here.
I just wanted to let everyone know that BizTalk Terminator is now also available as a part of BizTalk Health Monitor (BHM) – the new BizTalk tool that integrates MBV and Terminator functionality into an MMC Snap-In that you can run as a part of your BizTalk Admin Console.
Update: Final version of standalone Terminator released.
More info here – including a comparison of the 2 versions of Terminator.
Terminator functionality can be found in BHM’s Maintenance node:
- You must be on BHM v3 or above to get the Terminator functionality
- Get BHM from https://www.microsoft.com/en-us/download/details.aspx?id=43716
- See here for a walkthrough of BHM v3 functionality – including Terminator integration.
- See the BHM blog at http://blogs.msdn.com/b/biztalkhealthmonitor for more information on BHM.
- So far, I’ve included the most popular Terminator tasks in BHM and plan to add more over time
- BHM officially supports BizTalk 2010, 2013, 2013R2, and 2016.
I’m going to continue updating standalone Terminator for now and it is still available here.
At some point, I’ll probably end up creating a final build of the standalone version and move all new development to BHM. I’ll update this blog once that decision is made.
- Standalone Terminator tool is no longer being updated. The final version of standalone Terminator is v2.5. All new development (including BTS2016 support) is happening in BHM Terminator.
- MBV is no longer being updated and the last update to it was in July 2014. To make sure you are using the latest MBV repository with the latest rules/queries, be sure to move to BHM.
I’ve been pinged a number of times on this so thought I should blog the workaround and an explanation.
First, let’s say MBV shows you something like the following in the Warning and Summary Report:
Or you just notice in MBV that there’s a bunch of cache messages in one of the queue tables:
Well, according to my Using BizTalk Terminator to Resolve Issues article, you simply run the Terminate Caching Instances task:
Issue Identified by MBV
Terminator Resolution Task
Terminator View Task
Orphaned Cache Instances
MBV Integration or Manual Task Selection
Terminate Caching Instances
(in Delete task category)
View Count of Cache Messages in All Host Queues
View Count of Cache Instances in All Hosts
This is due to a known bug and there is a hotfix available. See KBs 944426 & 936536 for details.
That should do it. If it doesn’t, make sure you have stopped all the BizTalk hosts (that includes the IIS app pool hosting the BizTalk isolated host if the caching items are there) and try again. (Hey, you shouldn’t be running Terminator without stopping all the BTS hosts anyway).
Now, that will definitely do it.
Well… like 99.9% of the time.
Let’s say you do all of the above and then run one of those View tasks (or MBV) and notice that Terminator left behind some of those caching instances (and their associated caching messages). This happens even though Terminator claims to have terminated all of them successfully. And rerunning the task doesn’t help – Terminator will repeatedly say it successfully terminated those instances but either of the View tasks will show that they’re still there.
Ok, so now you’re most likely running into a very rare scenario that I’ve come across a few times.
First, I should point out that the Terminate Caching Instances and Terminate Instances tasks use BizTalk’s WMI API to interact with the messagebox – and that’s key. I had the opportunity to analyze some data from a customer environment that was running into this issue and it turns out that there are certain times when msgbox logic prevents the stored procs called by WMI from terminating “internal” instances – with caching being considered one of those “internal” types. As far as when exactly the msgbox logic goes down this “rare” path, I’ve never had access to a repro environment where I could fully debug this so I don’t have a good answer for that.
So I was going to write code in Terminator’s WMI class to catch this scenario and warn the user that they need to use the workaround to clean up the remaining items but unfortunately the msgbox logic catches the failed call and doesn’t pass that info on to the WMI caller – so there’s no way for Terminator (or any WMI client) to know that some of the instances weren’t deleted. As far as the WMI client is concerned, the stored proc call completed sucessfully so it assumes the instances it asked to be Terminated were actually terminated.
So what’s the workaround? Well, don’t use WMI for this particular situation. Instead, use Terminator’s Hard Termination tasks (Terminate Multiple Instances (Hard Termination) or Terminate Single Instance (Hard Termination).
While I was working on Terminator’s WMI class and having BizTalk engineers use Terminator on an internal-only basis within Microsoft, we noticed that on very rare occasions the termination tasks would not terminate something. This would not be a limitation of Terminator and we could reproduce it with any WMI client (including the BTS Admin console). I created the hard termination tasks specifically to handle those scenarios. They bypass BizTalk’s normal termination API and use SQL calls to directly interact with the BizTalk msgbox. They can terminate anything (well, so far) – even internal instances. Originally, we just had the Terminate Single Instance (Hard Termination) task to help terminate a one-off instance that just wouldn’t terminate any other way. That worked great for those one-off scenarios but I soon realized that sometimes there would be a larger number of instances that needed hard termination. I left the single instance task to give users that functionality and wrote the Terminate Multiple Instances (Hard Termination) task. That allows the user to choose the Host, Class, Status, and the Max number of instances to terminate and will do hard terminates on all items that fit the filter criteria.
The only thing that is painful about the Terminate Multiple Instances (Hard Termination) task is that if you have instances across multiple hosts with various statuses, you will need to run the task for each permutation since it doesn’t have the ability to handle, in one execution, multiple hosts and statuses (or classes) like the WMI-based Terminate Instances and Terminate Caching Instances tasks. Since Terminator v2 supports Powershell, I’m hoping at some point to create a powershell-based hard terminate task that can provide this functionality – just haven’t had the time to do that yet.
So, in short:
The Terminate Caching Instances task is not cleaning up all cache items.
Use the Terminate Multiple Instances (Hard Termination) task, choose Caching as the Class Parameter.
You will need to re-run this task for each Host and Status that applies to the cache items you’re trying to terminate – MBV or the Terminator View Count of Cache tasks should give you some info in this regard.
BTW, I’ve seen active, dehydrated, and suspended cache items so you may need to cycle through all of those Statuses.
In general, if you ever find that any of the termination tasks are not terminating what you want, the two Hard Termination tasks (Single and Multiple) are the workaround.
Remember that Hard Termination tasks bypass the normal BTS APIs so should only be used if the normal Terminate Instances or Terminate Caching Instances task is not working – and as always, be careful with Terminator – especially when doing any of the deletion tasks.