BizTalk Server WCF-SAP Adapter: System.ArgumentException: An item with the same key has already been added

BizTalk Server WCF-SAP Adapter: System.ArgumentException: An item with the same key has already been added

Following my last post and continuing on the pain points of BizTalk Server infrastructure tasks migration, I end up founding another interesting error: WCF-Custom Reason: System.ArgumentException: An item with the same key has already been added.

Again, we indeed did more than a simple migration, we take this opportunity to improve our solutions and/or BizTalk Server environment to update our applications to the new the new available features. In terms of WCF-SAP Adapter, we are no longer using traditional RFC and we will be starting using SAP .NET Connector (NCo). Despite WCF-SAP adapter will continue to support both the RFC SDK and the SAP .NET Connector, SAP has announced deprecation of its classic RFC SDK (no longer be supported after March 31, 2016) and because of that, it should be used anymore.

While trying to connect our BizTalk Server Receive Location to SAP in order to listen to incoming messages, we initially got this error message:

The Messaging Engine failed to add a receive location “WcfReceiveLocation_SAPBinding_IdocDELVRY05V3R700_Custom” with URL “sap://CLIENT=400;LANG=EN;@a/SERVERNAME/00?ListenerGwServ=sapgw00&ListenerGwHost=LISTENERHOST&ListenerProgramId=BTS_LIST&RfcSdkTrace=False&AbapDebug=False ” to the adapter “WCF-Custom”. Reason: “System.ArgumentException: An item with the same key has already been added.

at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)

at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)

at Microsoft.Adapters.SAP.NCo.RfcServerConnection.Open(RfcClientConnection conn)

at Microsoft.Adapters.SAP.SAPInboundContract.InitializeRfcServerConnectionNCo(String connectionArguments)

at Microsoft.Adapters.SAP.SAPInboundContract.StartListener(String[] listenerActions, TimeSpan timeout)

at Microsoft.ServiceModel.Channels.Common.Channels.AdapterChannelListener`1.OnOpen(TimeSpan timeout)

at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout)

at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)

at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint.Enable()

at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint..ctor(BizTalkEndpointContext endpointContext, IBTTransportProxy transportProxy, ControlledTermination control)

at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiver`2.AddReceiveEndpoint(String url, IPropertyBag adapterConfig, IPropertyBag bizTalkConfig)”.

BizTalk Server WCF-SAP Adapter: System.ArgumentException: An item with the same key has already been added

Cause

What I found out was that this error was somehow related to the error reported in my previous post: BizTalk Server WCF-SAP Adapter: SAP.Middleware.Connector.RfcCommunicationException: LOCATION CPIC (TCP/IP) on local host BTSSERVERNAME with Unicode, ERROR service ‘?’ unknown.

The reason why I’m saying that is that each time I restarted the BizTalk Server Host Instances, the error that I got the first time I try to enable the port was this service ‘?’ unknown error. Afterward, whenever I tried to enable this port, I would receive this new error:

  • System.ArgumentException: An item with the same key has already been added

Why this is happening for me is a mix of a small bug and consistency, let me explain better:

  • Why consistency? This because in fact, you cannot have two SAP Receive Locations listening to the same SAP endpoint/SAP Listener Program Id.
    • Actually, this should be when this error would normally occur.
  • Why a small bug? Because in fact there isn’t any Receive location enable for that specific SAP endpoint/SAP Listener Program Id. What is happening is that despite the error and the receive location became automatically disable it somehow stores in memory that SAP endpoint/SAP Listener Program Id

Solution

The solution is very simple and stupid:

  • Restart the host instance that is running that Receive Location

After that, this problem goes away, but make sure you solve all the problems, or this may happen again.

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.

Deploying BizTalk360 within your organization

Deploying BizTalk360 within your organization

In this blog post, we will explain how different stakeholders within your organization can be involved in operating the BizTalk environment and managing BizTalk interfaces, by smartly, but safely deploy BizTalk360.

The BizTalk Administrators – the primary users of BizTalk360

Primarily, BizTalk360 will mainly be used by BizTalk administrators. They will use the product in their day to day operations and will probably be the ones who firstly receive notifications of threshold violations and daily health check reports.

These administrators are responsible for the day to day operations of the BizTalk group(s) and will be the main group of users of BizTalk360. Therefore, they need all the authorizations so they will have Super User accounts in BizTalk360. They will also be responsible for creating and maintaining User Access Policies.

After installation of the product, they will set up alarms for the BizTalk platform and the BizTalk applications it contains and receive notifications which are generated by the alarms.

The way they configure monitoring of their BizTalk environments with BizTalk360, might evolve like this:

  • Basic Threshold/Health Check monitoring with email notification
    • BizTalk Platform – Monitor platform components like Host Instances, SQL Server jobs, NT Services, BizTalk Health Monitor
    • BizTalk Applications – Monitor application artifacts like Receive Locations, Orchestrations, Send Ports, Instance states
  • Advanced Environment Monitoring
    • Endpoints – Monitor web services, queues, file shares, FTP sites, Azure services
    • Data monitoring – Monitor interface processing, automate resuming and terminating of service instances
    • Enterprise notification channels – Receive notifications via HP Operations Manager, Slack, ServiceNow, Microsoft Teams, Webhook, PowerShell

In case of any issues, the (team of) BizTalk administrators will fix these issues themselves, or they contact other stakeholders to discuss how the issues need to be fixed.

Improve your business processes by extending the reach of BizTalk360

The power of BizTalk360 lies in the fact that it provides a rich user interface with many different dashboards for many different purposes, both from a technical and a functional perspective. In contrast with the tools which come out-of-the-box with BizTalk Server, all the capabilities in BizTalk360 are protected by User Access Policies and Auditing. This enables you to give any person exactly that set of permissions that they need to be able to do their job.

When the BizTalk administrators feel comfortable with the product and with BizTalk administration in general, or when the organization requires it, BizTalk360 can be deployed to other parts of the IT department or business departments.

Involve stakeholders by sending notifications and providing access to BizTalk360

By sending alerts to stakeholders directly, you improve information management as your BizTalk Administrators don’t need to send separate emails or contact these stakeholders based on issues that have occurred.
Although in many cases, email will be the primary means of sending notifications from BizTalk360, the product also contains the following Notification channels:

• Microsoft Teams
• ServiceNow
• Slack
• PowerShell
• Webhook

These channels can be configured on each BizTalk360 alarm and enable you to receive the notifications where it is most convenient for you. You can also use a simple to use SDK to develop your own custom Notification channel. Read more about Notification channels in our Documentation Portal.

But you can go even further, BizTalk360 allows you to give stakeholders secure and limited access to BizTalk360. This kind of access can reach from read-only access to particular parts of the BizTalk platform to the capability to act on certain issues.

Deployment of BizTalk360 to your organization can be done in the pace you and your organization feel comfortable with. By sending notifications to the stakeholders and provide access to BizTalk360, you can keep the stakeholders informed of any issues, improve the availability of your business processes and meanwhile spreading the workload between all the stakeholders.

Think of the following scenarios which could be achieved:

  • Sending alerts to the help desk – This enables the help desk engineers to analyze any issues at hand and take countermeasures
  • Informing your administrators directly of any issues – Think of your System Administrators or DBA’s who receive alerts about server or database issues
  • Automated creation of support tickets in your ticketing system – This takes away the need to have an administrator to do it manually
  • Inform business users of issues – Think of batches which are not processed (non-events) or faulty processing of their messages
  • Provide access to dashboards/portals – Give your stakeholders access to portals like the BAM portal, ESB portal, Business Rules Composer, (Data) Monitoring/Analytics dashboard, etc. etc.

Identifying stakeholders

When it comes to deploying BizTalk360 throughout an organization, you can identify the following roles:

  • IT Support personnel
  • SQL Server DBA’s, System Administrators and BizTalk developers
  • Business Users

Let’s have a look at the roles mentioned above and their potential responsibilities with regards to BizTalk Server/BizTalk360.

IT Support personnel

To have a good eye watching on your BizTalk environment, you could involve the IT Support staff of your organization. Depending on your organization, they might be available 24/7. So it makes sense to send notifications of unexpected behavior happening on the platform level, as they might be able to act before BizTalk Administrators can. Besides sending them notifications, you could give the IT Support staff read-only access to BizTalk, so they can explore any issues and maybe help you fix them, while you might be at home, having weekend.

SQL Server DBA’s, System Administrators and BizTalk developers

Since operating BizTalk involves more than just the BizTalk Server product itself, but also components like SQL Server databases and all kind of Windows Server components, you might consider involving System Administrators and SQL Server DBA’s in BizTalk operations by sending them notifications which might be relevant for them.

A few examples are:

  • SQL Server Administrators can receive notifications in case the BizTalk related SQL Server jobs fail
  • System Administrators can receive notifications when the BizTalk servers are running out of disk space
  • BizTalk Developers can receive notifications in case issues arise with upgraded or newly deployed BizTalk applications

As a next step, you might give these administrators and developers access to BizTalk360. This ranges from providing read-only access to particular features to giving them full operational access to the parts of their interest.

If you give any of the stakeholders access to BizTalk360, it is also helpful to use the BizTalk360 Knowledge Base. When you properly maintain the Knowledge Base, your support people will have the help they need at their fingertips and be able to solve known issues quickly.
The Knowledge Base associates Knowledge Base articles to Service Instances, EventLog entries, ESB Exceptions, and Throttling data. Read more about its capabilities in our Documentation portal.

You might consider providing System Administrators, SQL Server DBAs and BizTalk developers with the following authorizations:

  • System Administrators
    • Advanced Event Viewer
    • BizTalk Health Monitor
    • Topology
    • Host/Host Instances
    • BizTalk/SQL Servers
    • Manage BizTalk/SQL NT Services
    • Tracking Manager
    • Adapters
    • Backup/DR Visualizer
  • SQL Server DBA’s
    • Secure SQL Queries
    • Advanced Event Viewer
    • BizTalk Health Monitor
    • Message Boxes
    • SQL Servers
    • Manage SQL NT Services
    • SQL Server Instances
    • Manage SQL Jobs
    • Backup/DR Visualizer
  • BizTalk developers
    • MessageBox queries (with/without access to content/context)
    • Tracking Queries
    • Advanced Event Viewer
    • Tracking Manager
    • Secure SQL Queries
    • BAM portal
    • ESB Exception portal
    • EDI Reports
    • Messaging Patterns

Business Users

This category of users might exist both inside as outside your organization. Depending on that, it will differ how they are involved in managing the interfaces. Normally, they will not take part in managing the BizTalk platform itself.

Internal business users can be informed about the processing, by providing them with notifications of disruptions in the processing of their interfaces, i.e., inform them of suspended instances, transmission failures, and failing process monitoring.

When you want to give business users access to BizTalk360, you can think of the following features:

– Specific BizTalk applications
– Message Box (Queries)
– Graphical Flow (Tracking)
– Business Rules Composer
– EDI Reports, parties and agreements
– ESB Portal
– Business Activity Monitoring
– Messages Content/Context
– Secure SQL Queries

In case external business users are involved in certain interfaces, you might send them the same notifications as internal business users. As the external business users will be outside your organization, you normally will not give them access to the BizTalk360 User Interface.

Conclusion

We often see, that BizTalk Server is considered as a black box and deep BizTalk knowledge is needed to be able to find out what’s all happening inside that box. By using BizTalk360, we make it easy to gain that insight, even with little BizTalk expertise. Furthermore, by deploying BizTalk360 outside the BizTalk administrators team, you can give your middleware a face and achieve much more transparency about all the processing taking place in your BizTalk environment.

By using BizTalk360 outside the admin team, it is easier to notify other stakeholders by sending them notifications directly from BizTalk360. Even further, besides sending notifications to these stakeholders within (or outside) your organization, you can give people (limited) access to BizTalk360. This way they can view for themselves how all the processing is taking place or check the wellbeing of the environment, without the need of contacting the BizTalk Administrators team.

All in all, it must be clear, that by extending the use of BizTalk360 outside the admin team, you will have a better ROI of the product. If you would like to know more about how BizTalk360 can help your organization to manage your BizTalk Server middleware platform, feel free to contact us.

Author: Lex Hegt

Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360.

BizTalk Server WCF-SAP Adapter: SAP.Middleware.Connector.RfcCommunicationException: LOCATION CPIC (TCP/IP) on local host BTSSERVERNAME with Unicode, ERROR service ‘?’ unknown

BizTalk Server WCF-SAP Adapter: SAP.Middleware.Connector.RfcCommunicationException: LOCATION CPIC (TCP/IP) on local host BTSSERVERNAME with Unicode, ERROR service ‘?’ unknown

I’m currently migrating a BizTalk Server infrastructure from BizTalk Server 2013 R2 to BizTalk Server 2016, this is a simple task to do in terms of BizTalk Server Applications, or BizTalk Server solutions. However, migrating or in fact, creating a new environment always brings challenges in terms of connectivity and/or proper configurations. SAP.Middleware.Connector.RfcCommunicationException: LOCATION CPIC (TCP/IP) on local host BTSSERVERNAME with Unicode, ERROR service ‘?’ unknown error is one of this cases!

We indeed did more than a simple migration, we toked this opportunity to improve our solutions and/or BizTalk Server environment to update our applications to the new the new available features. In terms of WCF-SAP Adapter, we are no longer using traditional RFC and we will be starting using SAP .NET Connector (NCo). Despite WCF-SAP adapter will continue to support both the RFC SDK and the SAP .NET Connector, SAP has announced deprecation of its classic RFC SDK (no longer be supported after March 31, 2016) and because of that, it shouldn’t be used anymore.

While trying to connect our BizTalk Server Receive Location to SAP in order to listen to incoming messages, we initial got this error message:

The Messaging Engine failed to add a receive location “IN_WCF_SAP_LISTENER_BTS” with URL “sap://CLIENT=400;LANG=EN;@a/SERVERNAME/00?ListenerGwServ=sapgw00&ListenerGwHost=LISTENERHOST&ListenerProgramId=BTS_LIST&RfcSdkTrace=False&AbapDebug=False” to the adapter “WCF-Custom”. Reason: “SAP.Middleware.Connector.RfcCommunicationException:

LOCATION CPIC (TCP/IP) on local host BTSSERVERNAME with Unicode

ERROR service ‘?’ unknown

TIME Fri Nov 23 13:47:22 2018

RELEASE 721

COMPONENT NI (network interface)

VERSION 40

RC -3

DETAIL NiErrSet

COUNTER 3

at SAP.Middleware.Connector.RfcServer.DoStart(Boolean logErrors)

at SAP.Middleware.Connector.RfcServer.Start()

at Microsoft.Adapters.SAP.NCo.RfcServerConnection.Open(RfcClientConnection conn)

at Microsoft.Adapters.SAP.SAPInboundContract.InitializeRfcServerConnectionNCo(String connectionArguments)

at Microsoft.Adapters.SAP.SAPInboundContract.StartListener(String[] listenerActions, TimeSpan timeout)

at Microsoft.ServiceModel.Channels.Common.Channels.AdapterChannelListener`1.OnOpen(TimeSpan timeout)

at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout)

at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)

at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint.Enable()

at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint..ctor(BizTalkEndpointContext endpointContext, IBTTransportProxy transportProxy, ControlledTermination control)

at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiver`2.AddReceiveEndpoint(String url, IPropertyBag adapterConfig, IPropertyBag bizTalkConfig)”.

The receive location “IN_WCF_SAP_LISTENER_BTS_ DV” with URL “sap://CLIENT=400;LANG=EN;@a/SERVERNAME/00?ListenerGwServ=sapgw00&ListenerGwHost=LISTENERHOST&ListenerProgramId=BTS_LIST&RfcSdkTrace=False&AbapDebug=False” is shutting down. Details:”The Messaging Engine failed while notifying an adapter of its configuration. “.

SAP.Middleware.Connector.RfcCommunicationException: LOCATION CPIC (TCP/IP) on local host BTSSERVERNAME with Unicode, ERROR service '?' unknown

Cause

The first thing you need to know is that this error is not related to the fact that we are now using the SAP .NET Connector (NCo). This error will still occur if you change back to the classic RFC.

The basic steps if of course to see if we have connectivity and proper access to SAP, and because we were in DEV environment a good way to test this is to Add WCF-SAP Adapter Metadata to a BizTalk Project inside Visual Studio

  • In your Visual Studio BizTalk project, in Solution Explorer, right-click your project, click Add, and then click Add Generated Items.

You will have the same look and fill as creating a Receive Port on the Administrative Console with the advantage of being able to navigate the under the SAP object structure. This way you can test connectivity and access to the proper resources.

In our case, everything worked well. We were able to connect and generate the schemas.

However, all of this is not enough. For the WCF-SAP adapter to work properly in runtime you also need to configure, somewhere in the BizTalk Server machine the TCP port where the SAP adapter will be looking for these connections, because this property is not exposed through the BizTalk Server WCF-SAP Adapter GUI port (send or receive) configuration.

SAP.Middleware.Connector.RfcCommunicationException: LOCATION CPIC (TCP/IP) on local host BTSSERVERNAME with Unicode, ERROR service '?' unknown: WCF-SAP Port Configuration

In our case, the .NET Connector need to know the port of the message server. This is also very common to happen in load balancing situations.

Solution

In order to specify this TCP port, you need to modify the ‘services’ file in each BizTalk Server machine, normally in:

  • C:Windowssystem32driversetcservices

To include the following entry:

MSHOST 1234/tcp # SAP ENV System Message Server Port

Where:

  • MSHOST is your Gateway server, in our case sapgw00
  • And 1234 is your message server port, in our case: 3300

SAP.Middleware.Connector.RfcCommunicationException: LOCATION CPIC (TCP/IP) on local host BTSSERVERNAME with Unicode, ERROR service '?' unknown: solution

Just for curiosity, BizTalk isn’t the only that will use these values and this ‘services’ file. SAPGUI will also use these same values when connecting to SAP systems.

After we did this configuration, we were able to connect and start receiving messages from our SAP system.

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.

BizTalk Application Deployment Using Azure Pipeline with BizTalk360 API’s

BizTalk Application Deployment Using Azure Pipeline with BizTalk360 API’s

Introduction

The process of BizTalk Application deployment is a complex, time consuming and repetitive task. Microsoft BizTalk Server comes with capabilities to create deployment packages. However, the MSI packages you create with the out-of-the-box features of BizTalk Server have some shortcomings which lead to deployments which are error-prone.

BizTalk Deployment Framework

The BizTalk Deployment Framework (BTDF) covered these deployment issue.

  • Deploy complex BizTalk solutions containing BizTalk artifacts but also for example SSO settings, ESB itineraries and SQL Server scripts, with no human intervention
  • Consolidate all environment-specific configuration and runtime settings into one, easy-to-use Excel spreadsheet
    Maintain a single binding file that works for all deployment environments
  • Make automated deployment a native part of the BizTalk development cycle, then use the same script to deploy to BizTalk servers

To eliminate the manual process, organization’s start to adopt the DevOps practice. In this article, we will see how the BizTalk360 API’s are helpful in a BizTalk environment monitoring while deploying BizTalk Applications using Azure CI/CD Pipelines.

Deploy BizTalk Applications with Azure Pipelines

Microsoft has introduced automated deployment of BizTalk Applications in BizTalk Server 2016 Feature Packs. In BizTalk Server 2016 Feature Pack 1, automatic deployment and application lifecycle management (ALM) experience was introduced. Automatic deployment process has been improved with the release of BizTalk Server 2016 Feature Pack 2.

  1. Within Visual Studio for each project, set the Application Name of your BizTalk application
  2. In addition to using an agent-based deployment, you can also use deployment groups to deploy your BizTalk applications to multiple servers
  3. In the release task, you can install the BizTalk application, and enter the BizTalk management computer, and the path to the deployment package

Azure Pipelines (CICD)

With Automated build/release definitions you can set up the CI or CICD Pipelines based on environment (Development/QA/Staging/Production):

  1. Continuous Integration (CI) : Set up build task definitions rules to the VSTS Source code base
  2. Continuous Integration Continuous Deployment (CICD) : Configure both build task definition and release task definition to automate the build and test process

If you are new to Azure CI/CD Pipeline go through this article Configure automatic deployment with Visual Studio Team Services in BizTalk Server

Deployment Build Definitions

BizTalk Build Pipelines

In the BizTalk Application deployment process, you can create the following build/release definitions:

  1. Visual Studio Build

    The build and release definitions are Visual Studio Team Services tasks, and should be done by a VSTS admin. The build definition builds your project within your git or VSTS repository, and the release definitions deploys it to your BizTalk Server environment

  2. Publish Build Artifacts

    In Publish Build Artifacts stage configure the Path to publish (Application Project Folder)
    Publish Build Task

  3. Deploy BizTalk Server Application

    BizTalkApplicationDeployment
    A BizTalk Server application deployment build definition operation has three options available

    1. Create new BizTalk Application: Deploys a new application, if the application already exists, it uninstalls the current applications (full stop), and installs the new application. If continuous integration is enabled, it automatically redeploys the application when it is updated in the repository
    2. Update an existing BizTalk Application: Appends changes, such as schemas, to an already running application. It does not require a full redeploy of the application
    3. Install BizTalk Server Application: Install the application, enter the BizTalk management computer name and the deployment package path

BizTalk360 API’s in CI/CD

Customer Scenario: In Build pipeline stages the customer wants to utilize the BizTalk360 API to set the Stop Alert for Maintenance in a BizTalk environment. Once after the new build is deployed to BizTalk Server and then user wants to configure newly deployed artifacts to monitor in BizTalk360. As the last step revoke the Stop Alerts in BizTalk360 Application.

We can use the following steps in Build definitions to achieve the above scenario;
BizTalk Depolyment using BizTalk360 API

  1. Visual Studio Build (explained above)
  2. Set Stop Alert for Maintenance

    BizTalk360 customers can take advantage of using the BizTalk360 API’s to set the alert maintenance during the BizTalk Application deployment.
    Create the PowerShell Script stage to call the SetAlertMaintenance API method to set the BizTalk environment under the maintenance. Assign the PowerShell Script full path
    Set Alert Maintenance

    SetAlertMaintenance.ps1

    $Request = '{
      "context": {
        "callerReference": "SAMPLE_REFERENCE",
        "environmentSettings": {
          "id": "DC45E848-E485-41D7-92E5-7C9932D43CB8"
        }
      },
      "alertMaintenance": {
        "comment": "BizTalk Deploy",
        "maintenanceStartTime": "2018-11-22T15:26:37.000",
        "maintenanceTimeUnit": 0,
        "maintenanceTimeLength": 2,
        "isActive": false,
        "expiryDateTime": ""
      }
    }'
    $Request
    $body = ConvertFrom-Json -InputObject $Request
    $body
    $RepsonseSet = Invoke-RestMethod -Uri http://biztalk360webserver/Services.REST/AlertService.svc/SetAlertMaintenance -Method Post -ContentType "application/json" -Body $body -UseDefaultCredentials
    $RepsonseSet 
    
    

    Set Alert Maintenance API returns the alert maintenance object with maintenance Id (out parameter). You can refer this maintenance Id, when revoking the maintenance in the below step.

  3. Publish Build Artifacts (explained Above)
  4. Deploy BizTalk Server Application

    Once after BizTalk Application deployment is completed, next step you can configure the newly artifacts to monitor. For creating BizTalk360 alarms and BizTalk Application mappings during deployments with BTDF, you could consider to BT360Deploy command-line tool. Use this GitHub project to setup pipeline stage.

  5. Remove the Stop alert for Maintenance

    After the deployment of BizTalk Applications, the maintenance mode of BizTalk environment must be revoked in BizTalk360. To achieve this step, create a PowerShell Script stage and assign the PowerShell Script path(RemoveMaintenance.ps1)

    $RequestRem = '{"context":{"callerReference":"SAMPLE_REFERENCE","environmentSettings":{"id":"DC45E848-E485-41D7-92E5-7C9932D43CB8"}},"maintenanceId":"308d8b2c-b3e0-4024-b600-845113b06b80"}' 
    
    Invoke-RestMethod -Uri  http://biztalk360webserver/BizTalk360/Services.REST/AlertService.svc/RemoveAlertMaintenance -Method Post -ContentType "application/json" -Body $RequestRem -UseDefaultCredentials
    
    

    Remove Alert Maintenance

Summary

Automation of Build and/or Deployment will be helpful to reduce the turnaround time. Hopefully, this article has provided insight in how you can setup the BizTalk Application Deployment automation and how you can utilize BizTalk360 API’s to stop alert during deployment and how to set up monitoring of the newly deployed BizTalk Artifacts.

Get started with the free 30 days trial. For any queries/feedback please write to us support@biztalk360.com.

Author: Senthil Kumar Palanisamy

Senthil Kumar Palanisamy is the Technical Lead at BizTalk360 having 14 years of experience in Microsoft Technologies. Worked various products across domains like Health Care, Energy and Retail.

BizTalk Custom Functoid: Extension function parameters or return values which have Clr type Char are not supported

BizTalk Custom Functoid: Extension function parameters or return values which have Clr type Char are not supported

Back to Errors and Warnings, Causes and solutions, this time with a surprising error that I got while I was testing a custom Functoid that I had developer: Extension function parameters or return values which have Clr type ‘Char’ are not supported

The full error description was:

Invoking component…

C:DevelopmentBizTalk Server Project1BizTalk Server Project1Map1.btm: warning btm1028: The required field “VarValue” has no incoming link, constant value, or default value.

TestMap used the following file: <file:///C:DevelopmentBizTalk Server Project1Schema1_output.xml> as input to the map.

C:DevelopmentBizTalk Server Project1BizTalk Server Project1Map1.btm: error btm1050: XSL transform error: Unable to write output instance to the following <file:///C:UsersAdministratorAppDataLocalTemp2_MapDataBizTalkServerProject1Map1_output.xml>. Extension function parameters or return values which have Clr type ‘Char’ are not supported.

Test Map failure for map file <file:///C:DevelopmentBizTalk Server Project1BizTalk Server Project1Map1.btm>. The output is stored in the following file: <file:///C:UsersAdministratorAppDataLocalTemp2_MapDataBizTalkServerProject1Map1_output.xml>

Component invocation succeeded.

BizTalk-Custom-Functoid-type-Char-are-not-supported

Again, I catch this error while I was trying to test my custom functoid inside Visual Studio.

The Custom Functoid was invoking the following code:

private string GetCSharpBuffer()
{
    StringBuilder builder = new StringBuilder();
    builder.Append("public string SplitText(string element, int pos, char splitter)n");
    builder.Append("{n");
    builder.Append("tstring[] listValues = element.Split(splitter);n");
    builder.Append("tif (listValues != null)n");
    builder.Append("ttif (listValues.Length > pos)n");
    builder.Append("tttreturn listValues[pos];n");
    builder.Append("treturn "";n");
    builder.Append("}n");
    return builder.ToString();
}

Or, to be easier to read in a clean C# code that would be like this:

public string SplitText(string element, int pos, char splitter)
{
   string[] listValues = element.Split(splitter);

   if (listValues != null)
      if (listValues.Length > pos)
         return listValues[pos];
   return "";
}

Cause

Unfortunately, this is a BizTalk mapper engine limitation. The type Char is not supported at least on built-in code inside the XSLT, in other words by using the Functoid SetScriptBuffer functionality:

SetScriptBuffer(ScriptType.CSharp, this.GetCSharpBuffer());

I didn’t test but it may work if you replace this functionality with the SetExternalFunctionName functionality:

SetExternalFunctionName(GetType().Assembly.FullName, GetType().FullName, "SplitText");

Note: This means that the code will not be inside the XSLT but instead you will be calling an external assembly that needs to be on GAC.

Solution

The simple solution, or let’s call I workaround, is to use strings instead, either on inputs as outputs parameters of the Functoid. For that we just need to replace the above code to this:

private string GetCSharpBuffer()
{
    StringBuilder builder = new StringBuilder();
    builder.Append("public string SplitText(string element, int pos, string splitter)n");
    builder.Append("{n");
    builder.Append("tstring[] listValues = element.Split(Convert.ToChar(splitter));n");
    builder.Append("tif (listValues != null)n");
    builder.Append("ttif (listValues.Length > pos)n");
    builder.Append("tttreturn listValues[pos];n");
    builder.Append("treturn "";n");
    builder.Append("}n");
    return builder.ToString();
}

Or again, in a simple C# form:

public string SplitText(string element, int pos, string splitter)
{
    string[] listValues = element.Split(Convert.ToChar(splitter));

    if (listValues != null)
       if (listValues.Length > pos)
          return listValues[pos];
    return "";
}

By doing that you will be able to successfully test the Functoid.

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.

BizTalk360: How to monitor BizTalk Server BRE Policies Pending to be Deployed

BizTalk360: How to monitor BizTalk Server BRE Policies Pending to be Deployed

A few days ago, I helped a fellow “BizTalker” on a PowerShell script implementation that would monitor all your BizTalk Server BRE Policies, highest versions, that aren’t in the deployed state. You can know more about this on my blog: BizTalk DevOps: Monitor your BizTalk environment using PowerShell – Monitoring BRE Policies Pending to be Deployed.

However, my monitor script is, in fact, a combination of:

  • A piece of PowerShell that invokes a SQL Query and notifies by email if any non-compliance occurs
  • A SQL Server query that will have the ability to check what rules are in a non-compliance state; this is where the magic happens

I did that because out-of-the-box BizTalk360 doesn’t have the capability to monitor BRE Policies that aren’t in the deployed state. Despite that BizTalk360 allows you to create “Secure SQL Queries” that will enable us to:

  • Write optimized SQL queries and store them with friendly names in BizTalk360
  • Assign who will have permissions to run the queries
  • Run the queries, of course

This will allow us to quickly check if any policy is in a non-compliance state.

01-BizTalk360-Secure-SQL-Queries

You can see and download the SQL script here: BizTalk360 Secure SQL Server Query to check BRE that are not in a Deployed state.

Nevertheless, you cannot implement monitoring capabilities on these Secure SQL Queries. This feature was made for allowing support persons to safely execute custom queries without impacting the environment.

So, I wanted to go a little further, and to do that I asked my great friend Lex Hegt (Technical Lead at BizTalk360) to help me to implement a monitoring mechanism for this strategy, using the existing features in BizTalk360 and… we did it!

02-BizTalk360-Database-queries-to-monitor-BRE

How to monitor BRE using BizTalk360

The easy way for you is to download and import the BRE monitor alarm that I, along with Lex, developed here: BizTalk360 Monitor BRE Undeployed policies alarm.

To do that, after you download the zip file containing alarm information, you should:

  • Access to BizTalk360 and click ‘Settings‘ button on the upper right corner

01.3-BizTalk360-Import-Alarm-Settings

  • Select the ‘Import and Export‘ option from the left tree menu and select ‘Import

01.4-BizTalk360-Import-Alarm-import

  • Import operation is a 5-step process, as shown below:
    • Step 1 – Select configuration file (zip file containing alarm information): This is very straightforward, you just click on the ‘Select Files‘ button to choose the configuration file, or you can drag and drop the configuration file into the user interface as shown below.

01.5-BizTalk360-Import-Alarm-import-select-file

    • Step 2 – Choose the alarms you wanted to import: Once the configuration file is selected, the wizard will automatically move forward into step 2 and display all the alarms that are present in the configuration file. You can then either choose to import all of them or select only the specific alarms you wanted to import as shown below.
      • As you will see in the picture below, we also include the Secure SQL Queries, you should also select them;

01.6-BizTalk360-Import-Alarm-import-import-configurations

      • Note that by default all the alarms are kept in disabled status to avoid sending unnecessary notification after import, you may want to review all the configuration before enabling it
    • Step 3 – Map environment discrepancies: Once you have chosen the relevant alarms to import, the system will detect if there is any mapping required between the source system and destination system. The three common things that will require mapping are BizTalk Server, SQL Server, and SQL Server Instance names since they will, for sure, differ between environments. The third screen ‘Map Environment‘ will give the user the option to do this mapping as shown below.

01.7-BizTalk360-Import-Alarm-import-map-environment

    • Step 4 – Review the summary: On the fourth screen, based on the selection and mapping so far, will give a final summary report. It will also give bit more detail on each alarm, for example which artifacts are configured in each one of them (you need to expand the down arrow to view it). Once you are happy, you can just click on the ‘Import‘ button

01.8-BizTalk360-Import-Alarm-import-summary

    • Step 5 – Result summary and exception handling: Once the import button is clicked, all necessary backend calls will be made, and the result of the import process will be shown. You can view all the details like which artifacts are imported, what mapping applied etc. If there are any errors during the import process in specific alarms, those will be displayed as well
  • Once you close the import wizard, you can navigate to the ‘Manage Alarms‘ screen in the Monitoring section, and you can view all the imported alarms.

01.9-BizTalk360-Import-Alarm-manage-alarms

Of course, you can always create your alarm manually by:

  • Click ‘Monitoring‘ in the Navigation panel
  • Click ‘Manage Alarms‘ tab
  • Click ‘New Alarm
    • Enter a descriptive name for the Alarm, Email id (you can enter multiple email ids as comma separated values), and Alarm Description
  • Click ‘Next‘ to enter the Threshold Alarm page
    • Select the ‘Alert on threshold violation’ check box if you want to be alerted upon a threshold violation.
  • Click ‘Next‘ to enter the Health Monitoring Alert page
    • This step is optional, if you want to use the alarm for regular health/status check, you can select day(s)/time when you want to receive the status alert
  • Click ‘Next‘ to move to the screen to set up data monitoring alert.
    • Select the checkbox ‘Use this alarm for data Monitor Alerts’ if you wish to associate the current alarm with the data monitors.
    • Select the checkbox ‘Notify on Success as well’ to receive success email alerts for the configured data monitors. If you do not choose the second checkbox, you will not receive the success email alerts.
  • Click ‘Next‘ to move to the last section of adding the Advanced Settings information
    • All the settings in the Advanced settings page are optional. Set the phone number to receive the Notification SMS when the alert is triggered. As with the email ids, you can enter multiple phone numbers as comma separated values.
  • Click ‘OK‘ to create the alarm
    • The alarm will be created, and you will be redirected to the Manage Alarms page

But if you choose this approach you will them need to:

  • Select the ‘Manage Mapping‘ in the Navigation pane under ‘Monitoring‘ and then:
    • Select the alarm you just created above or that you already have in your environment
    • Select the tab option ‘Database Query
    • Finally, create two new queries, by selecting the button ‘New Query
      • One for monitoring BRE on Published state
      • Another for monitoring BRE on Saved state

01.10-BizTalk360-Import-Alarm-manage-mapping

    • After you press the ‘New Query‘ button you need:
      • Enter the Name for the Database Query
      • Add the SQL Instance name
      • And the SQL Database name
        • This need to be “BizTalkRuleEngineDb

01.11-BizTalk360-New-Alarm-New-Database-Query-Basic-Details

    • Click ‘Next‘ to add the Query and Threshold details
    • This will define when BizTalk360 must return an alert and notify us if any non-compliance occurs

01.12-BizTalk360-New-Alarm-New-Database-Query-Query-and-Threshold-details

01.13-BizTalk360-New-Alarm-BRE-Alarm

If you noticed, my queries have a red flag which means that I have non-compliance BRE and I will get an alert.

Now you have your favorite monitoring tool be able to monitor the state of your BRE!

Author: Sandro Pereira

Sandro Pereira is an Azure MVP and works as an Integration consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.

Microsoft Integration Weekly Update: November 19, 2018

Microsoft Integration Weekly Update: November 19, 2018

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

If you want to receive these updates weekly, then don’t forget to Subscribe!

Feedback

Hope this would be helpful. Please feel free to reach out to me with your feedback and questions.

Advertisements

Optimizing the Data Collection in BizTalk360

Optimizing the Data Collection in BizTalk360

Introduction

BizTalk360 is a tool for operating, monitoring and getting analytical information for BizTalk environment(s). BizTalk360 comes with productivity tools like the Advanced Event Viewer, the Throttling Analyzer, the Backup/DR visualizer, the BizTalk Health Monitor etc. These tools are also designed to help support people to perform their activities in a seamless way. BizTalk Server, being middle-ware, high volumes of data passes through the Message Box. Based on the business requirements, Tracking is enabled for important BizTalk Integration transactions.

In similar fashion, BizTalk360 collects important information to perform various activities in operations, monitoring and analytics section. In this article, we show BizTalk360’s data collection features and how you can configure the settings to optimize the data collection.

Advanced Event Viewer

The Event Log is an important component which is used by BizTalk Administrators to investigate, if any configuration or transaction errors have happened in BizTalk Server. The Advanced Event Viewer is a powerful feature in BizTalk360 which in fact is a central repository for collecting the event logs from multiple BizTalk servers and SQL servers in BizTalk environment. It has the capability to filter the event logs by various parameters like Event Id, Event Type, Sources etc.

The BizTalk360 Monitoring service collects the event log information based on Event Logs and Event Sources which are configured in the BizTalk360 Advanced Event viewer settings.

BizTalk360 Monitoring IOPS

BizTalk360 uses Windows Management Instrument (WMI) queries to collect the event log details from BizTalk Servers and SQL Servers in an environment. In a multi-server BizTalk environment, few customers raised concern about BizTalk360 IOPS (input/output operations per second). In this set-up, there is change of high volume of IOPS in BizTalk360 Box (Monitoring Service).

BizTalk360 Recommendation
To avoid large volume of data collection or to improve the performance of the BizTalk360 monitoring services, following event log configuration are suggested
1. Enable the necessary servers to collect Event Logs and Event Sources
• Primary BizTalk Servers in Active – Passive scenario
• Multiple SQL server, enable the event log data collection for important nodes.

2. BizTalk360 provides a default set of Event Logs and Event Sources which are widely used in BizTalk and SQL servers. Configure the necessary Event Sources that are required as per the business requirement:

• BizTalk Event Log Sources(Default):
Bam Event Provider, BamManagementUtility,BAMWebServices,BizTalk DW Reporting, BizTalk Server, BizTalk Server Deployment, BizTalk Server EDI, BizTalk360 Monitor, BusinessRulesEngine, ENTSSO,MSDTC,MSDTC Client, RuleEngineUpdateService, System.ServiceModel 4.0.0.0, System.ServiceModel.Install 3.0.0.0,W3SVC-WP,XLANG/s

Configure required BizTalk Sources as
“BizTalk Server,BizTalk Server Deployment,BizTalk Server EDI,ENTSSO,MSDTC,MSDTC Client,RuleEngineUpdateService”

• SQL Sources(Default):
MSSQLSERVER, MSSQLSERVER$AUDIT, MSSQLServerOLAPService, SQLSERVERAGENT

Configure required SQL Sources as
“MSSQLSERVER, SQLSERVERAGENT”
Note: This configuration will reduce the BizTalk360 Input/Output operations. You can read this article to know more details about Event-Log Data collection configuration.

Analytics Performance Data

In a data-driven business setup, there will be scenarios where critical executive decisions are made based on the performance reports of various components of the listed servers in a BizTalk environment. BizTalk360 aims to offer an out of the box tool with similar capabilities as the Performance Monitor tool in Windows servers.
BizTalk360 Analytics offers visual display of the most important performance counters that are consolidated and arranged on a single screen so that the information can be monitored in a glance. Custom reports can be built in minutes with the metrics that really count for your business, and they are easy to share with other users. In addition, dashboards can give you a summary of many reports on a single page using drag-and-drop widgets for fast and easy customization.

AdvancedEventViewerByServer
Performance metrics data collection can be optimized by selecting the necessary metrics (Windows, SQL, BizTalk and IIS) based on the server type.

• BizTalk Server: Enable Windows, BizTalk by default. Web Application is configured in the server enable IIS and similarly to SQL.
• SQL Server: Enable Windows and SQL performance metrics

Tracking Data

Initial design of the tracking data widgets was simple; it ran the necessary queries directly against the Tracking database and displayed the results in a graphical form on the widget. A larger database degraded the performance of these queries and it had a direct impact on the user experience of the Analytics dashboard, causing slower widget loading and occasional time outs. Since Analytics widgets offered users the option to view data up to 30 days within the widget, an efficient approach was required to improve the performance and user experience.
In BizTalk360 v8.3 to overcome these challenges, we decided to create a service that runs in the background. This service collects the tracking data periodically in small chunks and thereby avoid running expensive queries on the Tracking database. Here, only the resulting metrics are collected rather than pulling all the tracking data to BizTalk360 database. This improves the overall performance of the Tracking widgets as they are not querying the Tracking database directly.
BizTalk360 Analytics needs the tracking data to represent data in custom widgets

• Failure Rate by Port and Schema
• Message Performance by Host, Orchestration, Pipeline, Adapter, Port and Schema

BizTalk360 suggests enabling the data collection for categories and counters needed in custom widgets. If you want custom widgets for Failure Rate and Message Performance of Port and schema, enable only those counters in Tracking data collection.

AnalyticsPerformanceData

Data Purging

Collecting the necessary data is important to be able to represent the data in BizTalk360. However, you don’t want unlimited growth of your BizaTalk360 comes out of box with the “Data Purging” feature to able to manage the size of BizTalk360 database. If you want to know more about Data Purging please read this article.

DataPurgingConfiguration

Constantly manual monitoring the database size is a tedious process. You can utilize the Database Query monitoring when the database size is grown over the threshold limit.

SELECT total_size_mb = CAST(SUM(size) * 8. / 1024 AS INT)
FROM sys.master_files WITH(NOWAIT)
WHERE database_id = DB_ID()

Conclusion

We hope this article is useful to optimize and manage the data collection for various features in BizTalk360.
Get started with the free 30 days trial. For any queries/feedback please write to us support@biztalk360.com.

Author: Senthil Kumar Palanisamy

Senthil Kumar Palanisamy is the Technical Lead at BizTalk360 having 14 years of experience in Microsoft Technologies. Worked various products across domains like Health Care, Energy and Retail.

BizTalk Assessment: PowerShell to check what BizTalk Server 2016 Feature Packs are installed in your Servers got updated

BizTalk Assessment: PowerShell to check what BizTalk Server 2016 Feature Packs are installed in your Servers got updated

In the past, I release this PowerShell in order that any BizTalk Administrator to check and control which BizTalk Server Feature Pack was installed in your BizTalk environment. However, recently Niclas Öberg, another fellow “BizTalker” that I meet in London Integrate event warn me that my script didn’t recognize the existence of FP3… and he was correct

An alternative was to use BHM but the current version of BHM is also not checking for FP3 (only 1 or 2 – this limitation for sure will be fixed in next versions of BHM), you may get a warning messages saying that no CU or FP is installed, or you may get an FP2 warning messages like me:

BizTalk Server 2016 Feature Packs: BHM
Nevertheless, I have installed FP3 in my environment.

Of course, once again you can easily check this manually in the “Control PanelProgramsPrograms and Features” and then view the “View Installed Updates”, although this can be a very annoying task.

So, I decided to fix my PowerShell script and release a new version that will solve this problem. Now, if you run the script you will be able to receive the following output describing what FP was or is installed in your environment:

Microsoft BizTalk Server 2016 Feature Pack 1 is installed
Microsoft BizTalk Server 2016 Feature Update 2 with CU3 [KB4054819]LDR is installed
Microsoft BizTalk Server 2016 Feature Update 2 with CU4 [KB4077058]LDR is installed
Microsoft BizTalk Server 2016 Feature Update 3 [KB4103503]LDR is installed

The beauty of this script is that now it is prepared for detecting any other future PF that may be released by Microsoft, without the need for you to change the script.

...
    foreach ($Property in $key.Property) 
    {
        if ($Property -like '*Microsoft BizTalk Server 2016 Feature Update *') 
        { 
            $findF2 = 1
            Write-Host "$($Property) is installed"
        }
    }
...

This script follows the sequence of scripts that I release in the past to check what Cumulative Updates where installed in the machines.

THIS POWERSHELL SCRIPT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND.

Check which BizTalk Server 2016 Feature Packs installed with PowerShellCheck which BizTalk Server 2016 Feature Packs installed with PowerShell (2 KB)
Microsoft | TechNet Gallery

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.

BizTalk DevOps: Monitor your BizTalk environment using PowerShell – Monitoring BRE Policies Pending to be Deployed

BizTalk DevOps: Monitor your BizTalk environment using PowerShell – Monitoring BRE Policies Pending to be Deployed

After a tweet exchange with Mark Brimble a fellow “BizTalker” that I respect and admire, I realized that I had developed a PowerShell script some time ago to monitoring BRE Policies that could help him solve his problem. The initial question or need he was facing was:

  • How would you detect when there is no rules policy in a deployed state? I don’t know how many times I import a rule an forget to set it to deployed…

Monitoring BRE Policies Pending to be Deployed: problem

This is a common problem, and to be honest, I sometimes forget what is the correct state of a policy: Deployed or Published. (the correct answer is Deployed). And unfortunately, there isn’t a simple solution to solve this need and the solutions that I found was:

  • Using BizTalkFactory PowerShell Provider that nowadays come with BizTalk Server (“SDKUtilitiesPowerShell” folder)
  • Create my own monitor script – more work involved

Using BizTalkFactory PowerShell Provider is quite simple, but it has some limitations for what I would like to archive, for example, it only shows the policies that are bound to a particular BizTalk Application.

Monitoring BRE Policies Pending to be Deployed: BizTalkFactory PowerShell Provider

And I would like to know and have visibility to all of them because you don’t need to bind a policy to a BizTalk Application on the BizTalk Administration Console to use that policy.

And for that reason, I decide to create my own monitor script that I can easily change and optimize for my scenarios.

The problem I faced in the past was in fact quite similar to what Mark Brimble was describing, maybe with some small differences but the end goal is the same, so I decide to help him (at least try) and publish this PowerShell script.

The purpose of this PowerShell script is to:

  • Monitor BRE Policies and check if the highest version of a given policy is on Deployed state
    • If not notify someone (your BizTalk administration team);

So how can PowerShell help us?

With this script, you can be able to monitor your BizTalk Server BRE Policies, highest versions, that aren’t in the deployed state. Only if the script finds any non-compliance, an email notification will be sent.

Taking this sample:

Monitoring BRE Policies Pending to be Deployed: Policies Sample

The result will be a notification that includes two warnings:

  • Policy1 is in a non-compliance state because version 1.2 is not deployed
  • Policy2 is in a compliance state
  • Policy3 is in a non-compliance state because version 1.2 is neither published nor deployed (this is optional, but I choose to include in my monitoring script)

This script is a combination of a PowerShell script and a SQL Server Script that allows you to set:

  • Set your email notification settings:
#Set mail variables
[STRING]$PSEmailServer = "mySMTPServer" #SMTP Server.
[STRING]$SubjectPrefix = "MBV Notification Report -  "
[STRING]$From = "biztalksupport@mail.pt"
[array]$EmailTo = ("sandro.pereira@devscope.net")
  • And configure a SQL Server script, that in fact were the magic happens. The SQL Script will have the ability to check what rules are in a non-compliance state:
/****** Sandro Pereira & José Barbosa - DevScope  ******/
;with 
cteHist as (
        select h.* from [BizTalkRuleEngineDb].[dbo].[re_deployment_history] h
join (select strname, max(dttimestamp) as dttimestamp from [BizTalkRuleEngineDb].[dbo].[re_deployment_history] group by strname) q on h.strName=q.strName and h.dtTimeStamp=q.dttimestamp
),
ctetDeployed as (
        SELECT StrName, nMajor, nMinor, nStatus
                                                FROM   (
                                                   SELECT StrName, nMajor, nMinor, nStatus
                                                                , row_number() OVER(PARTITION BY StrName ORDER BY nMajor, nMinor DESC) AS rn
                                                   FROM   [BizTalkRuleEngineDb].[dbo].[re_ruleset]
                                                   ) sub
                                                WHERE  rn = 1
)
select * from ctetDeployed d
where nStatus = 0
or exists (select 1 from cteHist h  where h.strName=d.strname and bDeployedInd=0)

The following Windows PowerShell script is a fragment that will help us demonstrate the monitoring capabilities:

$mydata = invoke-sqlcmd -inputfile $sqlQuery -serverinstance $server

Foreach ($log in $mydata)
{
    #Create mail body content
    $mailBody += "..."
}

Here is example expected report output from running the Windows PowerShell script sample, if any of the BRE Policies are in an unwanted state.

Monitoring BRE Policies Pending to be Deployed: Report

Note: This type of script must be viewed as a complement to the tools mentioned above or used in the absence of them. The script should also be adjusted to your needs.

THIS POWERSHELL & SQL SCRIPT ARE PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND.

Special thanks to my coworker José Barbosa for helping me optimize the SQL Server script!

The script can be found and download on Microsoft TechNet Gallery:
Monitoring BRE Policies in your BizTalk environment with PowerShell (18.0 KB)
Microsoft TechNet Gallery

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.