BizTalk Server 2020 – 20 days, 20 posts: Zip File Pipeline Component for BizTalk Server 2020

BizTalk Server 2020 – 20 days, 20 posts: Zip File Pipeline Component for BizTalk Server 2020

BizTalk Server 2020 – 20 days, 20 posts – day 11. Following my recent blog posts here is another pipeline component migrate to BizTalk Server 2020 that makes part of my BizTalk Pipeline Components Extensions UtilityPack project: Zip File Pipeline Component.

Zip File Pipeline Component

The Zip File Pipeline Component is a pipeline component for BizTalk Server which can be used in a send pipeline (encode stage) and is intended to compress (zip/gzip) outgoing messages.

  • The capabilities are similar to those available in compression software such as WinZip or 7-zip.
  • No compression/decompression software needs to be installed in the BizTalk Server machines.
Zip File Pipeline Component for BizTalk Server 2020

To use this pipeline component in your projects you just copy the BizTalk.PipelineComponents.ZipFile.dll file into the Pipeline Components folder that exists in BizTalk Server Installation directory: “..Program Files (x86)Microsoft BizTalk ServerPipeline Components” on every server.

You do not need to add a custom pipeline component to be used by the BizTalk Runtime to the Global Assembly Cache (GAC).

This component requires two configurations that are:

  • the FileExtension: where you can specify if you want for example a .zip or .gz file.
  • and Enabled: that is a true or false value to activate the compression.

What is BizTalk Pipeline Components Extensions Utility Pack?

BizTalk Pipeline Components Extensions Utility Pack is a set of custom pipeline components (libraries) with several custom pipeline components that can be used in received and sent pipelines, which will provide an extension of BizTalk out-of-the-box pipeline capabilities.

The project is available on the BizTalk Server Open Source Community repository on GitHub (https://github.com/BizTalkCommunity), and everyone can contribute with new pipeline components that can be extended or improve the existing BizTalk Server capabilities.

BizTalk Pipeline Components Extensions Utility Pack: Zip File Pipeline Component for BizTalk Server 2020

At the moment this project is available for:

  • BizTalk Server 2020;
  • BizTalk Server 2016;
  • BizTalk Server 2010;
  • BizTalk Server 2006-2009

Where to download it?

You can download BizTalk Pipeline Components Extensions Utility Pack from GitHub here:

BizTalk Pipeline Components Extensions Utility Pack

The post BizTalk Server 2020 – 20 days, 20 posts: Zip File Pipeline Component for BizTalk Server 2020 appeared first on SANDRO PEREIRA BIZTALK BLOG.

BizTalk Server 2020 – 20 days, 20 posts: Unzip File Pipeline Component for BizTalk Server 2020

BizTalk Server 2020 – 20 days, 20 posts: Unzip File Pipeline Component for BizTalk Server 2020

BizTalk Server 2020 – 20 days, 20 posts – day 10. Following my last blog post here is another component migrate to BizTalk Server 2020 that makes part of my BizTalk Pipeline Components Extensions UtilityPack project: Unzip File Pipeline Component.

Unzip File Pipeline Component

The Unzip File Pipeline Component for BizTalk Server can be used in a Received pipeline (Disassemble stage), and it allows you to receive a compress (zip/gzip) file and extract its contents into different XML messages.

  • The capabilities are like those available in any compression software such as WinZip or 7-zip:
  • This component doesn’t require any configurations.

You do not need to add a custom pipeline component to be used by the BizTalk Runtime to the Global Assembly Cache (GAC).

Unzip File Pipeline Component for BizTalk Server 2020

The component doesn’t require any configuration.

What is BizTalk Pipeline Components Extensions Utility Pack?

BizTalk Pipeline Components Extensions Utility Pack is a set of custom pipeline components (libraries) with several custom pipeline components that can be used in received and sent pipelines, which will provide an extension of BizTalk out-of-the-box pipeline capabilities.

The project is available on the BizTalk Server Open Source Community repository on GitHub (https://github.com/BizTalkCommunity), and everyone can contribute with new pipeline components that can be extended or improve the existing BizTalk Server capabilities.

BizTalk Pipeline Components Extensions Utility Pack: Unzip File Pipeline Component for BizTalk Server 2020

At the moment this project is available for:

  • BizTalk Server 2020;
  • BizTalk Server 2016;
  • BizTalk Server 2010;
  • BizTalk Server 2006-2009

Where to download it?

You can download BizTalk Pipeline Components Extensions Utility Pack from GitHub here:

BizTalk Pipeline Components Extensions Utility Pack

The post BizTalk Server 2020 – 20 days, 20 posts: Unzip File Pipeline Component for BizTalk Server 2020 appeared first on SANDRO PEREIRA BIZTALK BLOG.

App Services and APIM App Registration process in Azure AD

App Services and APIM App Registration process in Azure AD

Hi All,

This post is to describe and take through the steps to set up App Registrations for App Services and API Management using OAuth 2.0 with Azure Active Directroy.

  1. App Registration in AAD for securing App Services (WebApps and FunctionApps) using OAuth 2.0 with Azure active directory

This App registration is for securing the App Services, and should be used by the consumers (registered on AAD) to access the respective WebAPIs and Functions.

  1. Sign into Azure Portal.
  2. Select App registration
  3. Select New Registration.
  4. Name the Application as “nonprod-<team>-appsvc-appreg” , where <team> will be organisational domain  etc.
  5. Click Register.

Create App Registration

6. In the newly created App Registration, select “Manifest” property.

Select Manifest APIMClientID (2)

7. In that section set the following oAuth2 permission in the manifest modifying “oauth2Permissions” element of JSON representation.

“oauth2Permissions”: [
{
“adminConsentDescription”: “Allows another AAD registered application to access the App services”,
“adminConsentDisplayName”: “Access App Services”,
“id”: “8jca6c61-0762-7b99-a1e1-e4bdf7e6e165”,
“isEnabled”: true,
“lang”: null,
“origin”: “Application”,
“type”: “User”,
“userConsentDescription”: “Allows another AAD registered application to access the App services”,
“userConsentDisplayName”: “Access App Services”,
“value”: “user_impersonation”
}
],

8. “Save” the App Registration changes. This also creates a scope associated to the App Registration.

9. Similarly follow the same for Staging and Production as per shown tabular column.

Non – Production
Staging
Production
App Service App Registration Name

Note:- This App Registration Client ID can be used when consuming any of the Web Apps and Function Apps in HIP

nonprod-<team>-appsvc-appreg stage-<team>-appsvc-appreg prod-<team>-appsvc-appreg
oauth2Permissions Use the above snippet as it is.

Id as given above.

Use the above snippet as it is.

Replaced Id with correct Id

Use the above snippet as it is.

10. Add user accounts as “App Owner” to this App Registration

2) App Registration in AAD for API Manager (APIM) to secure APIs using OAuth 2.0 with Azure active directory

This App registration is for the APIM which creates the necessary roles that should be used by the consumers to access the APIs.

  1. Select App registration
  2. Select New Registration.
  3. Name the Application as “nonprod<companyname>-apim-appreg
  4. Click Register.
  5. Select “Expose an API” and set “Application ID URI”

APIMClientID_ExposeAPI

6. Select Manifest.

Select Manifest APIMClientID (2)

7. In that section set the following oAuth2 permission in the manifest modifying “appRoles” element of JSON representation as shown below.

“appRoles”: [
{
“allowedMemberTypes”: [
“Application”
],
“description”: “Allow client apps to perform read and write operations on <name>.v1 API.”,
“displayName”: “<name>.v1.ReadAndWrite”,
“id”: “da9a6ab2-bc67-4213-102c-07d5749ea2s3,
“isEnabled”: true,
“lang”: null,
“origin”: “Application”,
“value”: “<name>.v1.ReadAndWrite”
},
{
“allowedMemberTypes”: [
“Application”
],
“description”: “Allow client apps to perform read operation on <name>.v1 API.”,
“displayName”: “<name>.v1.Read”,
“id”: “8ceee182-dcd0-48b3-af69-1a7924dc9d15”,
“isEnabled”: true,
“lang”: null,
“origin”: “Application”,
“value”: “<name>.v1.Read”
}

]

}

8. Similarly follow the same for Staging and Production as per shown tabular column.

Non – Production Staging Production APIM – API Roles
APIM App Registration Name nonprod-<companyname>-apim-appreg stage-<companyname>-apim-appreg prod-<companyname>-apim-appreg

<name>.v1.ReadAndWrite

<name>.v1.Read

<name>.v1.ReadAndWrite

9. Add user accounts as “App Owner” to this App Registration.

BizTalk Server 2020 – 20 days, 20 posts: Local Archive Pipeline Component for BizTalk Server 2020

BizTalk Server 2020 – 20 days, 20 posts: Local Archive Pipeline Component for BizTalk Server 2020

BizTalk Server 2020 – 20 days, 20 posts – day 9. This week I decide to start migrating a project that I enjoy and use them in several of my clients: BizTalk Pipeline Components Extensions UtilityPack. And the first component that I choose was: Local Archive Pipeline Component.

Local Archive Pipeline
Component

BizTalk Server Local Archive pipeline component it’s a pipeline component that can be used for archiving incoming/outgoing message from any adapters. It will provide the following capabilities :

  • It can be used in any stage of a receive pipeline or send pipeline;
  • It can be used in multiple stages of a receive pipeline or send pipeline;
  • It provides an option for you to specify the location path for where you want to save the message: local folder, shared folder, network folder.
  • It can be used from any adapter:
    • If the adapter provides the ReceivedFileName property promoted like the File adapter or FTP adapter the component will take this value in consideration and save the message with the same name;
    • Otherwise, it will use the MessageID, saving the file with the MessageID has its name without extension.

The component has two parameters:

  • PerformBackup: True/False value to enable or disable the archive functionality;
  • Folder: Local (or shared) folder for where you want to perform the archive of the message;

What is BizTalk Pipeline Components Extensions Utility Pack?

BizTalk Pipeline Components Extensions Utility Pack is a set of custom pipeline components (libraries) with several custom pipeline components that can be used in received and sent pipelines, which will provide an extension of BizTalk out-of-the-box pipeline capabilities.

The project is available on the BizTalk Server Open Source Community repository on GitHub (https://github.com/BizTalkCommunity), and everyone can contribute with new pipeline components that can be extended or improve the existing BizTalk Server capabilities.

BizTalk Pipeline Components Extensions Utility Pack: Local Archive Pipeline Component

At the moment this project is available for:

  • BizTalk Server 2020;
  • BizTalk Server 2016;
  • BizTalk Server 2010;
  • BizTalk Server 2006-2009

Where to download it?

You can download BizTalk Pipeline Components Extensions Utility Pack from GitHub here:

BizTalk Pipeline Components Extensions Utility Pack

The post BizTalk Server 2020 – 20 days, 20 posts: Local Archive Pipeline Component for BizTalk Server 2020 appeared first on SANDRO PEREIRA BIZTALK BLOG.

BizTalk360 Compatibility with BizTalk Server 2020

BizTalk360 Compatibility with BizTalk Server 2020

Already there is a lot of buzz happening around the new release of the BizTalk Server 2020 version. It is the most expected version by the integration of community members. This new version consists of many exciting new features and enhancements to ease the life of different business users.

We continue our efforts to deliver uninterrupted services to all our customers across the globe, even during this pandemic crisis. That’s why BizTalk360 is pleased to announce the compatibility of its latest version with BizTalk Server 2020.

As a practice, we aim to bring you long-term focused features which add significant value to our existing and new customers by identifying all sort of obstacles they are facing while supporting/operating their BizTalk environment.

I wanted this blog to cover all the new bringings in the product related to BizTalk Server 2020. Alright then! Here, we will have a look at all the features you will get with the new BizTalk360 version, aligned with BizTalk Server 2020:

  1. Installer Support
  2. New BizTalk Group properties
    1. Receive Location (Fault Tolerance)
    2. Audit Log Operation
    3. Analytics
  3. Receive Location Advanced scheduling operation

Installer Support

We started the implementation from the installer and wanted to eliminate the restriction of this new BizTalk Server version in the installer. The first thing you will want to do is download the latest version of the BizTalk360 MSI. You can get it from the BizTalk360 Free trial page or in the application itself – either is fine, but from the application is easier, as it shows the New version availability notification in the top right bar.

BizTalk360-Installer-Support

Once you have got the MSI, the installation is a very simple process as BizTalk360 offers ready-to-go installation. As the instructions are very clear and precisely shown in the installer, wizards make your life easier to progress the installation without referring to any other articles. Perhaps, if you are running into any error, you can always refer to the installation documentation. Refer to the below screenshot where we have included the check for new BizTalk Server versions.

The customers who are using older versions, on the other hand, need not worry at all. All the older versions are still compatible with the BizTalk360 even after the latest update.

BizTalk360-Prerequisite-Check

New BizTalk Group Properties

In the new BizTalk Server 2020 version, a few properties have been implemented in the BizTalk group level. These are:

  1. Receive Locations
    1. Enable fault tolerance
    2. Retry Interval
  2. Tracking and Reporting
    1. Audit Management operations
    2. Maximum number of audit entries

Receive Location Fault Tolerance

Until previous versions, if any error occurs (be it transient) the receive locations will be disabled and effect further message transactions.

This new configuration prevents the receive location from getting disabled on errors.

BizTalk attempts to recover the receive location in an interval you set. The receive location continues running in the other host instances.

Audit Log

Every business transparency is important at a certain level. Perhaps, someone who does anything wrong will lead to a huge impact on business. To bring transparency in BizTalk Server, they have introduced Auditing capabilities. The new properties enable the admins to track all the activities performed by different users and the maximum number of audit entries the user wants to capture. Once the configurations are in place, BizTalk will start to poll the audit data into the audit table in the BizTalk Management database.

To know more about BizTalk Server 2020 auditing capability, refer to this article.

For easy reference, all above mentioned new properties have been added in BizTalk360 in the Group Properties screen as shown below.

BizTalk-Group-Properties

Advanced Scheduling Options for the Receive Locations

Now with BizTalk Server 2020, the new advanced scheduling comprises all the options which administrators wanted for scheduling. Based on the configuration, the recurrence option will be displayed in the receive location property screen as shown in the screenshot.

New Options

  1. The preferred time zone can be set based on the business scenario
  2. Automatically adjustment for Daylight Saving Time
  3. Recurrence options like daily, weekly, and monthly

Receive-Location-Properties

Conclusion

In a nutshell, BizTalk360 is compatible with the new version of BizTalk Server. Our next v9.1 will be released within this week with interesting new features. Why not give BizTalk360 a try! It takes only 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity of your own BizTalk Environments. 

The post BizTalk360 Compatibility with BizTalk Server 2020 appeared first on BizTalk360.

March 30, 2020 Weekly Update on Microsoft Integration Platform & Azure iPaaS

March 30, 2020 Weekly Update on Microsoft Integration Platform & Azure iPaaS

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform and Azure iPaaS?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

 

Microsoft Announcements and Updates

 

Community Blog Posts

 

Videos

 

Podcasts

 

How to get started with iPaaS design & development in Azure?

  • Robust Cloud Integration with Azure
  • Microsoft Azure for Developers: What to Use When
  • Serverless Computing: The Big Picture
  • Azure Logic Apps: Getting Started
  • Azure Logic Apps: Fundamentals
  • Microsoft Azure Developer: Creating Enterprise Logic Apps
  • Microsoft Azure API Management Essentials
  • Azure Functions Fundamentals
  • Cloud Design Patterns for Azure: Availability and Resilience
  • Architecting for High Availability in Microsoft Azure

Feedback

Hope this would be helpful. Please feel free to reach out to me with your feedback and questions.

BizTalk Server 2020 – 20 days, 20 posts: Installing BizTalk Server 2020 in a Standalone Machine Guide

BizTalk Server 2020 – 20 days, 20 posts: Installing BizTalk Server 2020 in a Standalone Machine Guide

BizTalk Server 2020 – 20 days, 20 posts – day 8. Microsoft announced a few months ago, the release of Microsoft BizTalk Server 2020 – the 11th major release of BizTalk Server –and, as usual with previous versions, I updated my installation and configuration manual for BizTalk Server 2020. This whitepaper will explain in detail – a step-by-step guideline – how to install and configure Microsoft BizTalk Server 2020 on a standalone environment running Windows Server 2019. This information will help you plan the installation and configuration of BizTalk Server 2020, applications and components on which it depends focused on creating a development environment (you can also follow this tutorial to help you create production environments, however, if this is the case you need to skip some steps).

Installing BizTalk Server 2020 in a Standalone Machine Guide

Assumptions and out of scope

We will be assuming that the operating system: Windows Server 2019 and the latest critical Windows updates from Microsoft already have been installed.

In this scenario, we will be performing a full installation of Microsoft BizTalk Server 2020. The following components will be installed:

  • Enterprise Single Sign-On (SSO).
  • BizTalk Group.
  • BizTalk Runtime.
  • Business Rule Engine.
  • BAM Tools and Alerts.
  • BAM Portal.
    • BAM Portal it is deprecated, but we still can use it.
  • BizTalk EDI/AS2 Runtime.
  • REST APIs (Management data APIs)
    • Management data APIs are endpoints that let you remotely update, add, and query the status of different artifacts in your BizTalk Server environment. The endpoints are added using REST and come with a swagger definition.
  • BizTalk TMS.
    • BizTalk TMS is a service that refreshes authentication tokens used by BizTalk. It is a prerequisite for the Office 365 adapters.
  • Microsoft BizTalk Adapters for Enterprise Applications.

What’s in store for you?

This whitepaper will give you a detailed understanding of the following:

  • Preparing the Prerequisites and Installation for BizTalk Server 2020
  • Installing and Configuring BizTalk Server 2020
  • BizTalk Server 2020 Environment Optimization
Installing BizTalk Server 2020 in a Standalone Machine Guide

Where I can download it

You can download the whitepaper here:

I would like to take this opportunity also to say thanks to Tom Canter for being a reviewer of this installation guide and Lex for all the work to publish this resource.

I hope you enjoy reading this paper and any comments or suggestions are welcome.

You can also find, download and enjoy several other free whitepapers of my own here:

The post BizTalk Server 2020 – 20 days, 20 posts: Installing BizTalk Server 2020 in a Standalone Machine Guide appeared first on SANDRO PEREIRA BIZTALK BLOG.

Data Purging in BizTalk Server and BizTalk360

Data Purging in BizTalk Server and BizTalk360

The term purge means removing or erase something completely. Data purging is a process or activity mostly heard with Database technologies, i.e. Cleaning up of Database.

There are many different techniques and strategies for data purging, which is often contrasted with data deletion. Deletion is more of a temporary process whereas Purging removes data permanently which in turn frees up the storage and/or memory space for other uses.

Purging process lets you archive the data even though it removed from the main source permanently, giving us an option to retrieve from the archive in case there is a need for it. The deleting process also removes data permanently but doesn’t necessarily involve keeping a backup and generally involves on insignificant amounts of data.

Why Data Purging is Necessary?

For any organization and in any environment, it is important to keep track of database growth and execute the data purging policies whenever necessary.

When the database size grows, the SQL server needs more memory and CPU to read data from the tables, which slows down the database operation.

Besides that, there is also the risk that, due to the growing database(s), you are running out of disk space.

Database Administrators spend a lot of their time dealing with the problem of database processes consuming too much disk space. So, it’s very important to monitor the database size to ensure the database is not seizing the memory and CPU for smooth operation. Also, it’s essential and necessary to automate the purging process.

Data-Purging

The Purge Process

The purge process moves data between and deletes data from three categories of data or data sets:

  • The current data set, which contains data that needs to be available to users. Users can change or review the data.
  • The history data set, which contains data that can only be reviewed or aggregated by users. Users can’t change the data.
  • The archived data set, which contains data that falls out of the availability threshold but needs to be stored to meet legal or business requirements. This data is archived, so it can’t be changed, reviewed, or aggregated by users.

Available Options for Purging

In any BizTalk environment, to purge the SQL database there are two main categories of options available. The most common way is

  1. SQL Server Jobs (by using SQL Agent)
  2. Windows Services

SQL Server Jobs

The purpose of the SQL Agent is to serve as a job scheduler. Many experienced DBAs use jobs running inside the SQL Agent to perform routine tasks such as backups, updating statistics, and rebuilding indexes as needed.

Scheduling jobs is one of the core SQL Server functions. Many businesses have numerous SQL Server jobs scheduled that perform any number of different tasks from database maintenance jobs like backup and index rebuilds to running queries and kicking off ETL (Extract. Transform and Load) tasks.

Sql-Server-Jobs

What is SQL Agent?

The SQL Agent is the SQL Server subsystem that’s responsible for automatic task scheduling. The SQL Agent is available in all of the editions of SQL Server except the SQL Server Express edition. Taking advantage of SQL Agent enables you to automate many of your routine IT database infrastructure tasks.

Advantages of SQL Agent

The SQL Server Agent gives you a more specific focus around scheduling specific types of jobs around SQL Server. As the Agent is a part of the cluster resource when a failover occurs as long as the cluster has been set up correctly the SQL Agent job will start once the SQL Server Service has started.

Windows task Scheduler is a component of Microsoft Windows that provides the ability to schedule the launch of programs or scripts at pre-defined times or after specified time intervals. SQL Server Scheduler is created for Agent job scheduling, so using SQL Server Scheduler to schedule the job is the best method.

Disadvantages of SQL Agent

If the Agent is not running at the time a job is supposed to run, when the Agent is started it will run the job at the next scheduled time. It will not automatically run the job.

SQL Server Agent jobs that are running at the time of a failover event on a SQL Server failover cluster instance do not resume after failover to another failover cluster node. Jobs that begin but fail to complete because of a failover event are logged as started but do not show additional log entries for completion or failure. SQL Server Agent jobs in these scenarios appear to have never ended. So if your server failover, you need to run this job again in the other node.

Disadvantages-Sql-Agent

Windows Services

Definition of Windows Services

Windows Services is a core component of the Microsoft Windows operating system and enables the creation and management of long-running processes.

Advantages

Unlike regular software that is launched by the end-user and only runs when the user is logged on, Windows Services can start without user intervention and may continue to run long after the user has logged off. The services run in the background and will usually kick in when the machine is booted. An option ideal for use on servers when long-running functionality is needed without interference with other users on the same system.

Data Collection in BizTalk360

BizTalk360 consists own database. Collecting the necessary data is important to be able to represent the data in BizTalk360. As many of you aware that BizTalk360 is a one-stop solution for BizTalk monitoring. So for monitoring and analytics purposes, we do collect different data from various environments (it collects the information from BizTalk, SQL Server, and Tracking database) and stored in the BizTalk360’s database for future references.

  • Advanced Event viewer
  • Analytics

They are the sparkling examples for the data collection within BizTalk360 when you have configured/added more environments (for each environment there will be multiple machines say example a simple multi-server environment consists of 2-BizTalk servers for HA, 1-SQL server. For cluster servers and Load balancing you can imagine the numbers). However, you don’t want unlimited growth of your database.

BizTalk360 comes out of the box with the “Data Purging” feature to able to manage the size of the BizTalk360 database.

Monitor BizTalk360 Database Growth

There might be a situation where the BizTalk360 database grows due to the amount and size of event log data and performance counter data. The BizTalk360 purging policy will take care of the data growth, but it is also important to monitor the size regularly to ensure BizTalk360 is working seamlessly.

Data Purging in BizTalk360

As mentioned, BizTalk360 comes out of the box with the ability to purge the historic data by setting up the purging duration (days to be monitor) after the specified period. The Administrators/Superusers can set up the “Purge duration” under “Settings -> BizTalk360 Health -> Data Purging”. This will control the database growth and hence the performance of BizTalk360 will not get affected.

Purge Mechanism in BizTalk360

BizTalk360 uses Windows Services to trigger the purging. BizTalk360 owns two services,

  1. Monitoring service
  2. Analytics service

Out of those, purging was created as a sub-service of Monitoring service.

Purge-Mechanism-BizTalk360

In a regular interval time (for every 2 hours) the “Purge Data – a sub-service of Monitoring service” will purge the data by hitting the Stored Procedures for purging available in the BizTalk360 database.

We have chosen the windows service to utilize the advantages of Sub-service and the services that can be started and paused. As most of you aware that the BizTalk360 services should be always running to collect the data and perform few operations, so we would like to effectively use the service rather than creating SQL job and monitor separately. Also, if any exception appears for the sub-service, the exception will be captured from the log files and it will be shown adjacent to the respective sub-service in the UI.

Based on the values set in the data purge duration, for every 2 hours, the data will be purged using the stored procedures with the trigger of Purge Data sub-service.

Until BizTalk360 v5.0, there was a single stored procedure (sp_b360_admin_PurgePolicies) called from our monitoring service which takes care of all the purging. In v6.0, we improved the logic by splitting it into multiple stored procedures (sp_b360_admin_PurgePolicies) which takes care of various parts of data purging.

How to Check if Purging Service Status is Healthy?

Purge Data – a sub-service of Monitoring service from BizTalk360 Settings -> BizTalk360 Health -> Monitoring Service Status -> Purge Data, will help you to understand and find the information that the purging is running for every 2 hours.

If you can compare the started time and Last execution time, for every 2 hours the execution should happen. Even if there is an exception, the exception will be captured in the log files and it will be shown adjacent to the purge data sub-service.

Additional Capabilities

Along with the purging service within BizTalk360, there are few more options to check if the database size is under the expected size.

  1. Using the Database size monitoring feature, users can monitor the database and log file size of BizTalk and BizTalk360 databases, by simply configuring the error and warning threshold values for the database and log file sizes. By this when the database size exceeds the expected size, you will receive an alert from BizTalk360.
  1. Use Database Query monitoring to Monitor the size of a database.

Whenever the database grows beyond expectations, the first thing to be checked is the standard database reports. This will ensure which table occupies more space. Then we can act on the purging policy and change it according to the business needs and data flow.

What are the Stored Procedures Involved in Purging?

Till the latest release, there were 14 Stored Procedures involved in Purging for 32 tables. Each stored procedure will purge multiple tables. Let’s see what the stored procedures are used for purging the corresponding tables.

  1. sp_b360_admin_PurgePolicies_MBVReports – It will purge the BizTalk Health Check tool execution results.
  • b360_mbv_Executions
  1. sp_b360_admin_PurgePolicies_GovernanceAudit – It will purge the complete Governance and Audit tables, which is related to BizTalk and BizTalk360 activities.
  • b360_governance_Application
  • b360_governance_HostInstance
  • b360_governance_ServiceInstance
  • b360_governance_rules
  • b360_governance_Server
  • b360_governance_ESBException
  • b360_audit_History
  1. sp_b360_admin_PurgePolicies_AlertNotification – Will purge the Alert notifications which is sent out from BizTalk360 like Email, SMS and Notification channel.
  • b360_alert_History_Email
  • b360_alert_History_SMS
  • b360_alert_History_System
  • b360_historical_Alert_Maintenance
  1. sp_b360_admin_PurgePolicies_EventLog – Will purge the data of event logs that are already collected from different machines.
  • b360_ev_EventLogData
  1. sp_b360_admin_PurgePolicies_Throttling – Will purge the data collection for Throttling analyzer
  • b360_td_Throttling
  • b360_td_ThrottlingDataCollectionError
  1. sp_b360_admin_PurgePolicies_MonitoringDashboard – The alarm status update of the monitoring dashboard will be purged
  • b360_alert_MonitorExecution
  1. sp_b360_admin_PurgePolicies_MonitoringLogState – will purge the monitoring log state, monitoring will put the entries of current and expected state.
  • b360_alert_Monitor_LogStateCheck
  1. sp_b360_admin_PurgePolicies_MonitoringLogThreshold – will purge the monitoring threshold log, monitoring will put the current, warning and error levels.
  • b360_alert_Monitor_LogThresholdCheck
  1. sp_b360_admin_PurgePolicies_ProcessMonitoring – will purge the data of all types of data monitoring like Process, Message box data, Tracking data, BAM, EDI, ESB, Logic Apps, and Event log data monitoring.
  • b360_st_ScheduleActivity
  • b360_st_ProcessMonitorResults
  • b360_st_DataMonitorResults
  • b360_st_DataMonitorTaskActionResults
  1. sp_b360_admin_PurgePolicies_PerformanceAnalyzer – The data collected for the Analytics module will be purged
  • b360_perf_PerfCounter_Data
  • b360_perf_PerfCounter_Expanded
  1. sp_b360_admin_PurgePolicies_LiveFeed – Live feed data will be purged
  • b360_broadcast_Message
  1. sp_b360_admin_PurgePolicies_TrackingData – The data collected for the Analytics module will be purged.
  • b360_dta_Counter_Data
  • b360_dta_Counters_Expanded
  1. sp_b360_admin_PurgePolicies_DBCache
  • b360_admin_Cache
  1. sp_b360_admin_PurgePolicies_BizTalkReporting – The data collected for the Analytics – Reporting module will be purged.
  • b360_BizTalkDB_DiskUsages
  • b360_BizTalkDB_Tbl_sizes
  • b360_Report_History_Email
  • b360_report_ReportArchive

Purge Settings in BizTalk360

Let’s see how purge helps. The default purging settings in BizTalk360 can be seen in the below screenshot;

Purge-Setting-BizTalk360

Based on the values/duration provided by the purging policy, the data will be purged. We can see that the purging duration for Data Monitoring is 2 months. Hence the historical data for 2 months will be present in the BizTalk360 database.

Purging needs to be done to remove the historical data, thereby making the database healthy. BizTalk360 purges the data by running the stored procedure in the specified duration specified in the settings. The purging settings can be altered by the customers according to their business needs and data flow. If a large volume of data flows through the ports, they can set the purge duration to a minimum value so that data growth is controlled.

The customer configured 3 environments in a single installation of BizTalk360. In each environment, there were 5 servers available due to BizTalk and SQL server. Hence there were 15 servers to collect data for Advanced Event Viewer and they have set-up 30days to monitor. The Database has grown almost to 300 GB because of the number of servers to monitor.

We have recommended the customer to decrease the purge duration from 1 month to 2days and TRUNCATE the particular table (Since the table belongs to BizTalk360 and it is historical data, it won’t affect the BizTalk server). After modifying the purge duration, the data were reduced to 2 GB started working as expected.

Can the Purging be done Manually?

BizTalk360 will purge the data automatically. Sometimes the data collection may be out/beyond our expectation and that leads to a huge volume of data.

Our first suggestion is to reduce the number of purge days. So that the data will be reduced.

Until v6.0, the stored procedure sp_b360_admin_PurgePolicies was used and in recent versions of BizTalk360, you can use the stored procedure sp_b360_admin_CleanupDB to manually purge the database in case of large database growth.

When the number of records in the database grows quickly, call to the stored procedure(s) can start to get timed out from the monitoring services. This will result in data not being cleared and resulting in unusual database growth.

Conclusion

Considering the feedback provided by our customers we have improved and fine-tuned the collection of the data into the BizTalk360 database.

Whenever the database grows beyond expectations, the first thing to be checked is the standard database reports. This will ensure which table occupies more space. Then we can act on the purging policy and change it according to the business needs and data flow.

If you have any questions, contact us at [email protected]. Also, feel free to leave your feedback in our forum.

The post Data Purging in BizTalk Server and BizTalk360 appeared first on BizTalk360.

BizTalk Server 2020 – 20 days, 20 posts: Install BizTalk Server Visual Studio Project Template

BizTalk Server 2020 – 20 days, 20 posts: Install BizTalk Server Visual Studio Project Template

BizTalk Server 2020 – 20 days, 20 posts – day 7. And for this post, I choose a crucial piece for BizTalk developers: How to install BizTalk Server Visual Studio Project Template.

On previous versions of BizTalk Server and Visual Studio, when we were installing BizTalk Server, it would detect that we had Visual Studio present and the BizTalk Server installation process will also install on our machines all the Visual Studio Templates necessary for us to build our BizTalk Server solutions.

Not with this new version of BizTalk Server. Now we need to go to the Visual Studio Marketplace (https://marketplace.visualstudio.com/) to get the BizTalk Server development tools.

BizTalk Server development tools are an integrated development
environment for BizTalk Server application development and testing, which
contains:

  • BizTalk project template, templates, and tools for schema, orchestration, pipeline, map development, and test.
  • BizTalk server application project template for automatic deployment and application lifecycle management (ALM) experience.
  • WCF LOB Adapter SDK for developers who need to integrate dynamic and complex line-of-business systems as part of an enterprise application integration effort.
BizTalk Server development tools

And you can get it in two ways:

BizTalk Server development tools

Or directly thru Visual Studio:

  • Open Visual Studio and use the Manage
    Extensions
    dialog box to install and manage Visual Studio extensions.
  • To open the Manage
    Extensions
    dialog, choose Extensions à Manage Extensions. Or, type Extensions in the search box and choose Manage
    Extensions
    .
BizTalk Server 2020: Visual Studio 2019 Extensions
  • On the Search, type BizTalk
    Server
    and chose to download the BizTalk Server development tools
    provided by Microsoft.
BizTalk Server 2020: Visual Studio 2019 Extensions
  • It will automatically download
    and install these components.
BizTalk Server 2020: Visual Studio 2019 Extensions
  • To finalize you need to close
    all Microsoft Visual Studio windows that are open. And next time you will open
    Visual Studio, the BizTalk Server project template will be available for you to
    start developing your solutions.
BizTalk Server 2020: Visual Studio 2019 New BizTalk Project

The post BizTalk Server 2020 – 20 days, 20 posts: Install BizTalk Server Visual Studio Project Template appeared first on SANDRO PEREIRA BIZTALK BLOG.

BizTalk Server 2020 – 20 days, 20 posts: BizTalk MapperExtensions Functoid Project Template for BizTalk Server 2020

BizTalk Server 2020 – 20 days, 20 posts: BizTalk MapperExtensions Functoid Project Template for BizTalk Server 2020

BizTalk Server 2020 – 20 days, 20 posts – day 6. In the last post, I published the BizTalk MapperExtensions Functoid Wizard to be compatible with BizTalk Server 2020 and Visual Studio 2019 but I reference that Visual Studio 2019 has a completely different look and fill and you will not be able to see this project under “BizTalk” projects in your Visual Studio 2019, instead, you need to search for it or scroll down to find it. So I decided to develop an alternative approach: BizTalk MapperExtensions Functoid Project Template.

BizTalk MapperExtensions Functoid Project Template

BizTalk MapperExtensions Functoid Project Template is a Custom Functoid Project Template for Visual Studio 2019. It allows you to create new Functoids project for BizTalk Server 2020 without having to create manually the project, in other words, having to manually create:

  • A new class library project in Visual Studio;
  • Add a reference to the Microsoft.BizTalk.BaseFunctoids assembly;
  • Add New Class and having to code the entire class;

This Template will automatically create a project sample reference for you to create your custom Functoid. You them need to make the necessary changes and implement the required logic.

The difference between this resource (BizTalk MapperExtensions Functoid Project Template) and the BizTalk MapperExtensions Functoid Wizard are:

  • You don’t required to install nothing, you just need to copy this resouce to a specific folder (explain bellow);
  • There is no Wizard involved to customize the project, instead a sample project will be created and you need to implememet all the required changes;
  • You can filter on the Visual Studio menu for BizTalk project and this template will appear, that doesn’t happen with the wizard; 
Visual Studio 2019 Filter for project templates: BizTalk MapperExtensions Functoid Project Template)

Installation

To use this project template just download the BizTalk MapperExtensions Functoid Template.zip file (without unzip) and copy to the following location:

  • C:Users<your-account>DocumentsVisual Studio 2019TemplatesProjectTemplates

Download

You can download the resource from:
BizTalk MapperExtensions Functoid Wizard for BizTalk Server 2020BizTalk MapperExtensions Functoid Project Template for BizTalk Server 2020
GitHub



The post BizTalk Server 2020 – 20 days, 20 posts: BizTalk MapperExtensions Functoid Project Template for BizTalk Server 2020 appeared first on SANDRO PEREIRA BIZTALK BLOG.