In the previous post, we analyze how you can monitor messages without reference counts (RefCounts), today it will be a similar topic, but this time we will be addressing messages with negative reference counts.
and the same question applies here:
Have you ever seen your Monitor BizTalk Server (BizTalkMgmtDb) job failing and complaining about the existence of messages with negative reference counts
Or do you ever want to know more about messages with negative RefCounts?
What are messages with negative RefCounts?
Do they show in the BizTalk Server Administration console? Are they impacting BizTalk Server performance?
Indeed messages with negative reference counts can appear from time to time in our environment, which you should definitely need to monitor. And to response to all previous questions:
What are messages with negative RefCounts?
Messages with negative RefCounts are messages in the MessageRefCountLogTotals with snRefCount less than zero. Once a refcount goes negative, the MessageBox cleanup jobs will not be able to clean up the corresponding messages.
Do they show in the BizTalk Server Administration console?
No, they don’t. The only way for you to know that exists messages with negative reference count is by:
Running the Monitor BizTalk Server (BizTalkMgmtDb) job
Executing the View Count of Messages With Negative RefCounts available on the BizTalk Health Monitor (maintenance)
Or executing a custom query against BizTalk Server databases.
Are they impacting BizTalk Server performance?
Yes, if they are too many.
As I mentioned before, the Monitor BizTalk Server SQL Agent job can detect these kinds of messages. In fact, it is able to identify any known issues in Management, MessageBox, or DTA databases. The job scans for the following issues:
Messages without any references
Messages without reference counts
Messages with negative reference counts
Messages with reference count less than 0
Message references without spool rows
Message references without instances
Instance state without instances
Instance subscriptions without corresponding instances
Orphaned DTA service instances
Orphaned DTA service instance exceptions
TDDS is not running on any host instance with the global tracking option enabled.
By default, the Monitor BizTalk Server job is configured and automated to run once in a week. Since the job is computationally intensive, it is recommended to schedule it during downtime/low traffic. The job fails if it encounters any issues; error string contains the number of issues found. Otherwise, it runs successfully.
Note: The Monitor BizTalk Server job only scans for issues. It does not fix the issues found.
However, are you ever curious to know how you can find these types of messages? Or did you already face the issue that you cannot open the BizTalk Health monitor because it is failing or you don’t have Internet connection to update them to the last version?
Well, now you have it here:
DECLARE @count bigint
SET @count = 0
SELECT @count = COUNT(*) FROM [dbo].[MessageRefCountLogTotals] WHERE [snRefCount] < 0
IF @count = 0
BEGIN
SELECT 'There are no negative RefCounts'
END
ELSE
BEGIN
SELECT COUNT(*) Count, [snRefCount] as 'RefCount Value'
FROM [dbo].[MessageRefCountLogTotals]
WHERE [snRefCount] < 0
GROUP BY [snRefCount]
ORDER BY [snRefCount]
END
Of course, needless to say, use it with care and thoughtfulness! It is preferred to call this query inside the BizTalk Health Monitor, and ideally, it is recommended to execute it during downtime/low traffic.
You need to run this script against BizTalkMsgBoxDB, and it will return the count of negative refcounts in the MessageRefCountLogTotals table. Equal to the script on the BizTalk Health Monitor.
Have you ever seen your Monitor BizTalk Server (BizTalkMgmtDb) job failing complaining about the existence of messages without reference counts or more know as messages without RefCounts? What are messages without RefCounts? Do they show in the BizTalk Server Administration console? Are they impacting BizTalk Server performance?
Indeed messages without reference counts can appear from time to time in our environment, which you should definitely need to monitor. And to response to all previous questions:
What are messages without RefCounts?
Messages without reference counts (RefCounts) are messages that don’t have correlating rows in the MessageRefCountLog tables and the MessageZeroSum table. Once they are in this state, the MessageBox cleanup job will not be able to clean up the corresponding messages.
Do they show in the BizTalk Server Administration console?
No, they don’t. The only way for you to know that exists messages without reference counts is by:
Running the Monitor BizTalk Server (BizTalkMgmtDb) job
Executing the View COunt of Messages Without RefCounts available on the BizTalk Health Monitor (maintenance)
Or executing a custom query against BizTalk Server databases.
Are they impacting BizTalk Server performance?
Yes, if they are too many.
As I mentioned before, the Monitor BizTalk Server SQL Agent job can detect these kinds of messages, in fact, it is able to identify any known issues in Management, MessageBox, or DTA databases. The job scans for the following issues:
Messages without any references
Messages without reference counts
Messages with reference count less than 0
Message references without spool rows
Message references without instances
Instance state without instances
Instance subscriptions without corresponding instances
Orphaned DTA service instances
Orphaned DTA service instance exceptions
TDDS is not running on any host instance with the global tracking option enabled.
By default, the Monitor BizTalk Server job is configured and automated to run once in a week. Since the job is computationally intensive, it is recommended to schedule it during downtime/low traffic. The job fails if it encounters any issues; error string contains the number of issues found. Otherwise, it runs successfully.
Note: The Monitor BizTalk Server job only scans for issues. It does not fix the issues found.
However, are you ever curious to know how you can find these types of messages? Or did you already face the issue that you cannot open the BizTalk Health monitor because it is failing or you don’t have Internet connection to update them to the last version?
Well, now you have it here:
DECLARE @nvcAppName nvarchar(256)
CREATE TABLE ##msgs_wout_refs (uidMessageID uniqueidentifier NOT NULL)
CREATE UNIQUE CLUSTERED INDEX [CIX_msg_wout_refs] ON [##msgs_wout_refs](uidMessageID)
INSERT INTO ##msgs_wout_refs (uidMessageID)
SELECT uidMessageID FROM Spool WHERE uidMessageID NOT IN(SELECT uidMessageID FROM MessageRefCountLogTotals UNION
SELECT uidMessageID FROM MessageRefCountLog1 UNION
SELECT uidMessageID FROM MessageRefCountLog2 UNION
SELECT uidMessageID FROM MessageZeroSum
)
DECLARE hostcursor CURSOR FOR
SELECT nvcApplicationName FROM Applications WITH (NOLOCK)
OPEN hostcursor
FETCH NEXT FROM hostcursor INTO @nvcAppName
WHILE (@@FETCH_STATUS = 0)
BEGIN
EXEC ('DELETE FROM ##msgs_wout_refs FROM ##msgs_wout_refs m, [dbo].[' + @nvcAppName + '_MessageRefCountLog] r WHERE m.uidMessageID = r.uidMessageID')
FETCH NEXT FROM hostcursor INTO @nvcAppName
END
CLOSE hostcursor
DEALLOCATE hostcursor
DECLARE @count bigint
SET @count=0
SELECT @count = count(*) from ##msgs_wout_refs
SELECT 'Messages w/o Refcounts: ' + cast (@count as nvarchar(10))
DROP TABLE ##msgs_wout_refs
Of course, needless to say, use it with care and thoughtfulness! It is preferred to call this query inside the BizTalk Health Monitor, and ideally, it is recommended to execute it during downtime/low traffic.
You need to run this script against BizTalkMsgBoxDB, and it will return the count of messages that don’t have correlating rows in the MessageRefCountLog tables and the MessageZeroSum table and should align with the ‘Messages w/out RefCounts’ issue that MessageBoxViewer identifies. Equal to the script on the BizTalk Health Monitor.
Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform and Azure iPaaS?
Integration weekly updates can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.
Continuous Integration and Continuous Deployment (CI/CD) is a practice that has become an essential aspect of Azure development. Although it is possible to execute each of the CI/CD pipeline steps manually, the actual value can be achieved only through automation.
And to improve software delivery using CI/CD pipelines, either a DevOps or a Site Reliability Engineering (SRE) approach is highly recommended.
In this whitepaper, I will address and explain how you can implement CI/CD oriented to Azure Function Apps using Azure DevOps Pipelines.
I will explain in detail all the basic things you have to know, from the creation of an Azure Function on Visual Studio 2022 to everything you need to create and configure inside DevOps to archive the implementation of the CI/CD process using Azure Functions.
What’s in store for you?
This whitepaper will give you a detailed understanding of the following:
An introduction to:
What are Continuous Integration (CI) and Continuous Deployment (CD)?
What are CI/CD Pipelines?
What is Azure DevOps?
Create an organization or project collection in Azure DevOps
Create a project in Azure DevOps
Building your Azure Function from scratch
Publish your code from Visual Studio
A step-by-step approach to building Azure Pipelines
A step-by-step approach to building Azure Release Pipelines
Oracle Database (commonly referred to as Oracle DBMS or simply as Oracle) is a multi-model database management system produced and marketed by Oracle Corporation. It is a database commonly used for running online transaction processing (OLTP), data warehousing (DW), and mixed (OLTP & DW) database workloads.
One of the most common scenarios we face in several organizations is to be able to integrate other systems and applications with Oracle, and for that, BizTalk Server uses the Microsoft BizTalk Adapter for Oracle Database or simply the WCF-ORACLE adapter.
The Microsoft BizTalk Adapter for Oracle Database exposes the Oracle database as a WCF service. Adapter clients can perform operations on the Oracle database by exchanging SOAP messages with the adapter. The adapter consumes the WCF message and makes appropriate ODP.NET calls to perform the requested operation. The adapter returns the response from the Oracle database back to the client in the form of SOAP messages.
The Oracle Database adapter surfaces metadata of Oracle database artifacts (tables, functions, procedures, etc.) that describes the structure of a SOAP message in the form of Web Service Description Language (WSDL).
It uses the Add Adapter Service Reference Visual Studio Plug-in, the Consume Adapter Service BizTalk Project Add-in, and the Add Adapter Metadata Wizard to enable adapter clients to retrieve metadata for operations and generate programming artifacts that can be used in your programming solution.
And it communicates with the Oracle database through the Oracle Data Provider for .NET (ODP.NET) and the Oracle client, which are part of the Oracle Data Access Components (ODAC) for Windows.
The following figure shows the end-to-end architecture for solutions that are developed by using the Oracle Database adapter:
This paper explains in detail – a step-by-step guideline – how to install and configure the WCF-OracleDB Adapter on a BizTalk Server 2020 standalone environment running Windows Server 2019.
What’s in the Whitepaper for you?
This whitepaper will give you a detailed understanding of the following:
The role of an adapter is to enable communications between BizTalk Server and external systems and trading partners. Users configure adapters by creating send ports and receive locations that define the properties for given instances of the adapter. Most adapters support both send and receive operations, whereas other adapters support communication in only one direction.
The HTTP Adapter is one of the adapters that support two-way communications, but unlike other adapters, this adapter has two characteristics that define it:
The HTTP “Receive” Adapter that is responsible for delivering messages to BizTalk is, in fact, a DLL that runs inside Internet Information Services (IIS)
And for that reason, it must be configured in IIS – it is not there out-of-the-box.
In this whitepaper, we will describe the step-by-step process of installing and configuring the HTTP Adapter in order to receive messages.
Some of you may think that the HTTP adapter is deprecated, but that is not true. The only adapters that were deprecated were the old SAP adapter (removed) and the SOAP and SQL adapter (that still are present in the Administration Console). The HTTP adapter is a classic but is not deprecated and is still very used today.
The idea for this whitepaper was partly out of a real need and at a customer’s request. Because it is a strange adapter with a peculiar configuration unlike any other adapter in BizTalk Server, when I was giving a training course, the attendees struggled to understand and put it working correctly. By coincidence, a week later, I needed to put this adapter working at another client. As a result of that request and that client’s need, I end up creating this step-by-step guide on how to install and configure the HTTP adapter to receive messages through HTTP requests. I think this is a good whitepaper about a classic adapter in BizTalk Server.
There is a similar whitepaper that I wrote in the past: BizTalk Server 2016: Receiving messages through the HTTP Adapter installation whitepaper focus on BizTalk Server 2016 as the name describes. This new one is just a small update on the BizTalk Server versioning and is to be updated with the latest version of the BizTalk Server.
What’s in the Whitepaper for you?
This whitepaper will give you a detailed understanding of the following:
Prerequisites
Configure the HTTP Receive Adapter in IIS
Configure the HTTP Adapter Receive and Send Handlers
There are many things to consider when planning this type of installation. This whitepaper will explain in detail – a step-by-step guideline – how to install and configure Microsoft BizTalk Server 2020 on a basic multi-computer environment using Windows Server 2019, i.e., installation of BizTalk Server with a remote SQL Server (1 SQL Server and 1 BizTalk Server). There will be 3 virtual servers:
1 Domain controller
1 Virtual Machine to host SQL Server
Windows Server 2019
SQL Server 2019SQL Server 2019 Enterprise Edition
1 Virtual Machine to host BizTalk Server.
Windows Server 2019
BizTalk Server 2020 Enterprise or Development Edition
In this scenario, I will perform a basic full installation of Microsoft BizTalk Server 2020, except for the SharePoint Adapter and additional components like Accelerators, ESB Toolkit or UDDI, emulating a production environment. The following components will be installed:
Enterprise Single Sign-On (SSO)
BizTalk Group
BizTalk Runtime
Business Rule Engine
BAM Tools and Alerts
BAM Portal (Although Microsoft has deprecated the BAM portal, it is still possible to install it.)
BizTalk EDI/AS2 Runtime
Microsoft BizTalk Adapters
This information will help you plan the installation and configuration of BizTalk Server 2020, applications, and components on which it depends focused on creating a UAT or Production environment (you can also follow this tutorial to help you create developer environments. However, if this is the case, you need to add some steps and additional components or software, in especially Visual Studio 2019).
Of course, it is assumed that all machines are already installed with the operating system and the latest critical Windows updates from Microsoft. Another presumption is that the domain controller is already installed and configured.
What’s in the Whitepaper for you?
This whitepaper will give you a detailed understanding of the following:
The need for a Domain Controller – Windows Groups and Service Accounts
Preparing Computers for Installation – Important considerations before setting up the
servers
Preparing and Install SQL Server 2019 machine
Prepare and install prerequisites on BizTalk Server 2020 machine
Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform and Azure iPaaS?
Integration weekly updates can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.
Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform and Azure iPaaS?
Integration weekly updates can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.
My day-to-day job focuses on enterprise integration between systems in the cloud and/or on-premises. Currently, it involves integration with D365 Finance and Operations (or Finance and Supply Change Management). One aspect of the integrations is monitoring. When a business has one or more Azure Integration Service running in production, the operation aspect comes into play. Especially integrations that support crucial business processes. The operations team requires the correct procedures, tools, and notifications (alerts) to run these processes. Procedures and receiving notifications are essential; however, team members need help identifying issues and troubleshooting. Azure provides tools, and so do third-party solutions. This blog post will discuss the value of having third-party monitoring in place, such as Serverless360.
Serverless360
Many of you who read blogs on Serverless360 know what the tool is. Moreover, it is a service hosted as a Software as a Service (SaaS). Therefore, operation teams can require access once a subscription is acquired or through a trial. Subsequently, they can leverage the primary business application, business activity monitoring, and documenter feature within the service. We will briefly discuss each feature and its benefits and value in the upcoming paragraphs.
BUSINESS APPLICATIONS
A team can configure, and group integration components with the business applications feature a so-called “Business Application” to monitor. It does not matter where the resources reside – within one or more subscriptions/resource groups.
As Lex Hegt, Lead Product Consultant at BizTalk360 points out:
Integration solutions can span multiple technologies, resource groups, tags, and even Azure subscriptions. With the Azure portal having the involved components in all those different places, it is hard to keep track of the well-being of those components. Serverless360 helps you utilize the concept of Business Applications. A Business Application is a container to which you can add all the components that belong to the same integration. Once you have added your components to a Business Application, you can set up monitoring for those components, provide access permissions, and administer them.
The Business Application brings another feature that provides an overview of the integration components and dependencies. You might be familiar with the service map feature in Application Insights on a more fine-grained level. The service map in Serverless360 is intended to show the state of each component and dependency on a higher level.
Within a business application, the configuration of monitoring components is straightforward. By selecting the component and choosing the monitoring section, you can set thresholds of performance counters and set the state.
BUSINESS ACTIVITY MONITORING
The second feature of Serverless360 is the end-to-end tracking capability called Business Activity Monitoring (BAM). The BAM feature organization can instrument their Azure resources that support integrations between systems. Through a custom connector and SDK, you can add tracking to Logic Apps and Azure Functions that are a part of your integration. A unique generated transaction instance-id in the first component will be carried forward to the subsequent stages in more functions and Logic Apps.
The operations team must do some work to leverage the BAM functionality. They need to set up the hosting of the BAM infrastructure, define the business process, instrument the business process and add monitoring (see, for instance, Azure Service Bus Logging with BAM – Walk-through). Once that is done, a clear view of the process and its stages are available.
The benefit of the BAM feature is a concise overview of the configured business processes. Moreover, you get an overview of the complete process and potentially see where things go wrong.
AZURE DOCUMENTER
The final feature Serverless360 offers Azure Documentation Generator, which is intended to generate documentation. Operations teams can generate documentation for the subscription that contains the integrations with the documenter. It is good to have a dedicated subscription for integration solutions to govern better and manage Azure resources.
When operations teams like to generate documentation, they can choose between different templates, storing of the document, and billing range.
Conclusion
Features and benefits of Serverless360 have been outlined in this blog post. Of course, there are many more features. Yet, we focused on the most significant one that provides Operations teams the most value. That is a clear overview of the state of integrations in a single-pane-of-glass and the ability to quickly drill down into integration components and spot issues at a fine-grained level. Furthermore, Business Activity Monitoring and Azure Documenter provide end-to-end tracking and generated documentation.
The post The value of having a Third-party Monitoring solution for Azure Integration Services appeared first on Steef-Jan Wiggers Blog.