by Lattetapriyanka Vishu | Aug 6, 2020 | BizTalk Community Blogs via Syndication
Introduction
What are all the various authenticating methods in which a user can use to connect to an SFTP server (depending on how the server is set up), How shall I keep track on my SFTP Port’s Location for data pile up, What are all the possible way to monitor the File count and Directory size in SFTP port, How does BizTalk360 gives an extensive SFTP support, wanted to know the answer for these queries and more, let us get into the blog.
In the upcoming version of BizTalk360, BizTak360 extends its support in SFTP by supporting the Multi-Factor Authentication and /n software SFTP Keyboard Interactive Authentication. We BizTalk360 always gives importance to the customer’s feedback/request and keeps business improvement as its vital important thus provide the customer outbox support which will keep the users in ease and more productive. Some of our customers are using /n software sftp adapter, we got a request on the same for supporting the /n software SFTP
How does BizTalk360 Support SFTP Location
Keeping up the connection of the SFTP server is very much important to ensure that the data are transferred between the SFTP port. There are many possible reasons that the SFTP Server connection may not be reachable like,
- SFTP server is down
- Time out in reaching the SFTP Server
- Incorrect password used
- Providing the incorrect Server Address
- SFTP port not reachable etc.
The major role of the BizTalk Administrator is to keep monitoring the Server connection, but it is not possible to keep tracking on the SFTP port’s location and SFTP server connection, thus with BizTalk360, you can monitor the SFTP File count and Directory size of a particular SFTP port by configuring the same to a BizTalk360 Threshold or Health Alarm with the preferred condition.BizTalk360 monitoring service will check for the configured SFTP location and will send an alert whenever the condition gets violated. It is also possible to monitor on a regular cycle to cross-check whether the SFTP location threshold condition is healthy.
SFTP Client Authentication Mode Support in BizTalk360
BizTalk Server provides the BizTalk SFTP adapter to get connected to the SFTP Server, additionally supports the /n Software SFTP adapter which holds different authentication modes. Authentication modes are,
|
BizTalk SFTP
|
/n software SFTP (Supports from V10.0)
|
BizTalk360 support in 9.1 and lower Version
|
BizTalk360 support in upcoming Version
|
|
Password
Public Key
Multi-Factor
|
Password
Public Key
GSSAPIWith Mic
None
Multi-Factor
Keyboard Interactive
|
Password
Public Key
|
Password
Public Key
Keyboard Interactive
Multi-Factor
|
In the BizTalk360 configure the SFTP Location in Monitoring > Manage Mapping > -File Location > SFTP, which will help to monitor the health of the Port’s Location. On configuring the SFTP Location, BizTalk360 will automatically pick the Client Authentication Mode data which is configured in the BizTalk Admin console port.
Password Authentication
This is the simplest form of authentication using the traditional username/password method. A client logs in to SFTP using Username/Password. No user private/public keys are required. The “username” and “password” can be configured under: SFTP > Configure > Security > Client Authentication Mode
In BizTalk360 you can choose the Password by clicking on the Client Authentication Mode drop-down in the Security Details of BizTalk360 and will be allowing you to provide the username and password, it is also possible to switch the authentication to “PublicKeyAuthentication”.
Public key Authentication
Public key authentication is a method where the SFTP client identifies itself to the server by using public/private key pairs. Prior to connection, the user’s public key must first be uploaded and registered on the SFTP server, SFTP > Configure > Security > Client Authentication Mode > Private Key
In BizTalk360 you can choose the “Public Key Authentication” by clicking on the Client Authentication Mode drop-down which will allow you to provide the Username, Private Key, and the Private Key password. The password here does not mean the username’s password, instead, you need to provide the password of the Public Key, if you provide the password of the username then the Port location which you have configured will be orphaned which means the location is not reachable using the Provide Public key and its password.
In case SFTP Location / Server is not reachable BizTalk360 will show the state of the port as Orphaned and will let you know the reason for the port being Orphaned by hover on the warning symbol and also in the alert notification via mail.
Multi-Factor Authentication
In the Multi-Factor Authentication we need to provide the Username & Password and the Public/Private key and password. Public/Private key and password configuration are like steps described in the latter section, except that it includes the private key’s password.
In the BizTalk360 Current version, the Multi-Factor Authentication support will not be available for you but very soon in the upcoming BizTalk360 V10 release, you will be facilitated with this feature. Thus, you could choose the Multifactor Authentication in BizTalk360 in the Client Authentication Mode list under the security details blade by allowing you to provide the Username, Password, Private Key, and the Private Key password.
Keyboard Interactive Authentication
Keyboard-interactive (KBI) authentication is the most recently introduced form of authentication for SSH. It involves the server sending prompts to the client, which the client must respond to correctly authenticated connection. Its purpose is permitting the client to support a variety of authentication mechanisms without knowing anything about them. In edtFTPj/PRO, when the server sends its list of prompts that require responses, the client searches the list of loaded SSHAuthPrompts for each prompt. It then sends the set responses back to the server. If the expected responses are supplied, the authentication succeeds.
If the prompt is for a user’s password, it will typically be something like “Password:”. The SSHPasswordPrompt class is supplied to make it easier to set up a password prompt.
In the upcoming BizTalk360 release, you could able to choose the Keyboard Interactive Authentication in the Client Authentication Mode list under the security details blade by allowing you to provide the Username and password.
Extensive SFTP Support in BizTalk360
SSH Tunnel
SSH is a standard for secure remote logins and file transfers over untrusted networks. It also provides a way to secure the data traffic of any given application using port forwarding, basically tunnelling any TCP/IP port over SSH. This means that the application data traffic is directed to flow inside an encrypted SSH connection so that it cannot be eavesdropped or intercepted while it is in transit. SSH tunnelling enables adding network security to legacy applications that do not natively support encryption.
Considering the importance of SSH Tunnel and its business impact, BizTalk360 supports the proxy SSH tunnel to connect the SFTP or another server by not keeping the username and password as mandatory.
To get connected to the SFTP server with a normal proxy connection in the current BizTalk360 provide the Address, Port, Type, Username, and Password, it will reach the SFTP location using the credentials and check for the connection and its location details of the SFTP port.
Remote path location pattern support
In the BizTalk server while configuring the remote location, the user may provide different types of patterns. As we BizTalk360 comforts the BizTalk user at every point to the core level.
Say for instance: Remote path can be configuring with the following combination, C:/Users/BT360Admin/nS_SFTP_in1
C:/Users/BT360Admin/nS_SFTP_in1/*.*
/C:/Users/BT360Admin/nS_SFTP_in1
/C:/Users/BT360Admin/nS_SFTP_in1/*.*
/C:/Users/BT360Admin/nS_SFTP_in1/
C:/Users/BT360Admin/nS_SFTP_in1
BizTalk360 is capable enough to monitor the File count and Directory size of the Receive Location or Sendport with all those remote path configurations
Extensive File Mask Support
In the BizTalk Server the file mask is a string that specifies the type of file that the File receive handler will pick up from the receive location. The file name is a string that specifies the name of the file where the File send handler will write the message. The key point to remember in configuring the file mask is that the file mask and file name are not case-sensitive.
To keep up its immense support, we are supporting all the combinations of file masks. BizTalk360 will fetch the File count base on the file mask configured, in spite of its case sensitiveness.
Here are the few samples for File mask,
CHPAPO*.txt *.txt * . * *outbound322* PPI* Tes*.*
.TXt *.tXT * . XMl Orderdata.Xml .XML TES*.*
Macros support
You can use a predefined set of macros to dynamically create the files in which the File send handler writes messages. Before creating a file on the file system, the File send handler replaces all the macros in the file name with their individual values. You can use several different macros in one file name.
You can use the file name macros while configuring the File send handler in BizTalk Explorer, or by using the BizTalk Explorer object model.
The File send handler does not replace the macros with a value if any of the following are true:
- The corresponding system property is not set.
- The macro is misspelled.
- The value for the macro contains symbols that are not valid in the file name.
In BizTalk360, you can able to monitor the file count and Directory size of the Send ports with macros by keeping the BizTalk File mask configuration configurable.
Conclusion
Hopefully, this blog helps you in understanding the basics of SFTP client authentication, as well as the types of SFTP Client Authentication Mode, its importance, and BizTalk360’s extensive support in handling and monitoring variant authentication Mode. We are happy to announce that the /n software SFTP and Macros support will be available for your service soon in the forthcoming BizTalk360 Version 10.0 release. To explore further give a try on BizTalk360 will take only a minute to place over BizTalk environment
The post Different SFTP Authentication Support in BizTalk360 appeared first on BizTalk360.
by Christhini | Jul 21, 2020 | BizTalk Community Blogs via Syndication
BizTalk360 exists for more than 9 years. In these 9 years, we have constantly brought new features and for every year, we have around 3 to 4 releases. We have always believed that listening to the requests of our customers and partners is the right way to evolve the product. So, for this webinar, we decided to explain 3 different phases you might go through when implementing BizTalk360. This will help in bringing structure in choosing which features to use in which phase of the implementation of the product.
Overview of the Webinar
During the webinar, we will identify these 3 different phases and will explain the purpose of each phase. For each phase, we will show different features you could use. Additionally, we will provide links to useful resources to ease the implementation on your side. This way, we intend to make the webinar as useful and practical as possible.
Date: 30 July 2020
Time: 10 – 11 AM BST
Join us at the Webinar
The webinar will be hosted by two of our product consultants, who have years of experience in the area of the BizTalk server and BizTalk360. They would be more than happy to share their real-world expertise, discuss your challenges, and come up with solutions for the same.
Do you want to join us at the webinar on July 30th? Why wait! Registration is simple, just click here to register your details.
Also, if you think you cannot attend the webinar at the specified time, no worries! Go ahead and register, our team will make sure to send you the webinar recording.
We are looking forward to welcoming you during the webinar!
The post Webinar Spoiler: Managing BizTalk Server – Step-by-Step implementation in BizTalk360 appeared first on BizTalk360.
by Praveena Jayanarayanan | Jul 2, 2020 | BizTalk Community Blogs via Syndication
BizTalk360 is the one-stop monitoring product for your BizTalk environment. Installing and upgrading BizTalk360 is a seamless process when all the permissions and configurations are in place. Each software product has a frontend and a backend. Most often people focus on the frontend more to give a rich user experience. On the other hand, the backend would be considered only when it comes to the performance of the product. Of course, it is the DBAs responsibility to check the health of the database.
BizTalk360 being an on-premise product, it has its own database which becomes created while installing the product. It is always a good practice to take care of the database and the data, to avoid the performance problems as per the saying,
“Data is a precious thing and will last longer than the systems themselves”
In this blog, I will give in-depth insight into the BizTalk360 database and some useful tips to take care of the database and the data to make the DBA’s work a little easier.
The BizTalk360 Database
The BizTalk360 application comes with the BizTalk360 database which contains about 116 tables, and a few stored procedures and views in it. Let us dig in and see what the important tables in the database are. One main point to be noted is that no changes should be done at the database level manually unless and until it is important and required.
The below table gives a clear picture of the important database tables related to the BizTalk360 modules.
|
Modules
|
Table Names
|
|
Installation
|
b360_admin_BizTalkEnvironment
b360_admin_GlobalProperties
|
|
Database information
|
b360_admin_DBMigration_Status
b360_admin_DBVersion
b360_admin_DBVersion_History
|
|
License information
|
b360_admin_LicenseActivation
|
|
Monitoring
|
b360_alert_Monitor
b360_alert_MonitorExecution
b360_ev_MonitoredServers
b360_mon_Service
|
|
Data Monitoring
|
b360_st_DataMonitorTasks
b360_st_DataMonitorResults
b360_st_DataMonitorTaskActionResults
b360_st_ProcessMonitorResults
b360_st_Schedules
|
|
Analytics – Performance
|
b360_perf_PerfCounter
b360_perf_PerfCounter_Data
b360_perf_PerfCounter_Expanded
|
|
Analytics – Messaging Patterns
|
b360_analytics_flows
b360_analytics_artifact_versions
|
|
Analytics – Reports
|
b360_report_Schedules
b360_report_Dashboards
|
Let us look at a few scenarios where updating of the tables would be required.
Connecting the BizTalkMgmtDb Database on a New Installation
It is often suggested as a best practice to install BizTalk360 in a separate standalone server over a BizTalk server, for better performance. In this case, to monitor the BizTalk server, the BizTalkMgmtDb details must be updated in the b360_admin_BizTalkEnvironment table in the BizTalk360 database or through the UI, else the following error message would appear in BizTalk360.
Migrating the BizTalk Server
Consider a scenario where your BizTalk environment is migrated to a higher version. When the BizTalk server is upgraded, the BizTalk360 server also needs to be upgraded with the same version of the admin components. If not, the following error will be seen in the BizTalk360 UI.
The below tables need to have the BizTalk version changed appropriately and the MS_BizTalk_install_location can be updated.
- b360_admin_BizTalkEnvironment -> BizTalk Version
- b360_admin_GlobalProperties -> MS_BIZTALK_VERSION & MS_BIZTALK_INSTALL_LOCATION
Monitoring Scenarios
Monitoring is considered a core functionality of BizTalk360. There may be some cases where the alerts may not be triggered. When checked in the logs, there might be timeout exceptions.
When there are a large number of data monitoring alarms and if much of their historical data is present in the database, it may result in timeouts and the exception can be seen in the Data monitoring alarm grid, for example when trying to delete an alarm. Maintaining the historical data for a long time may result in large database growth. This can be reduced by minimizing the purge duration.
The artifacts information which is mapped for monitoring can be found in the b360_alert_monitor table.
Data Purging
BizTalk360 has its own purging procedure to take care of purging data in the BizTalk360 database. In the Settings side, you can configure after how much time you want specific data to be purged. If you feel, the database grows with huge volumes of data, you can decide to change the purge policies.
The Event Log Data Collection
BizTalk360 can collect event log information from the BizTalk and SQL servers which are connected to the BizTalk environment. The event log data can be viewed in BizTalk360 by configuring and using the Advanced Event Viewer. The data is collected from the servers and stored in the b360_ev_EventLogData table. By default, there are many sources configured in BizTalk360.
This will collect the data for all the event levels and sources and store it in the table. However, this may also result in huge database growth thereby affecting the performance of the product.
To resolve this problem, the Event Log levels and sources, for which the data needs to be viewed, can be configured in BizTalk360 and the remaining sources can be removed from the configuration.
This will reduce the amount of data being polled for and stored in the database. Also, the purge duration can be reduced to remove the historical data.
Analytics Performance Data Collection
The Analytics module in BizTalk360 mainly deals with the performance-related information of the BizTalk and SQL servers. The performance data gets collected through the perfmon counters and stored in the performance-related tables mentioned above. The BizTalk360 Analytics service collects the data by contacting the servers through PerfMon and store it in the BizTalk360 database. By default, all the available metrics would be collected. This may also result in huge volumes of data which in turn might affect the performance.
To enhance the performance and to reduce the database growth, there has been added an option in BizTalk360 to choose the metrics for which the data needs to be collected and used in BizTalk360.
Data purging is also available for performance-related information. The purge duration can be minimized to reduce the database growth.
There are different stored procedures available in the database to take care of the purging action. The purging subservice, running under the BizTalk360 monitoring service, is responsible for executing the purge stored procedures.
In SQL Server Management Studio, there is the Disk usage by the top tables database report. This report helps to find out the tables which are containing high volumes of data.
Database Permissions for the Service Account
As a prerequisite to installing BizTalk360, it is recommended to provide sysadmin permission for the BizTalk360 service account for smooth installation. Since there would be inserts, updates, creates, and alters, sysadmin permission is recommended.
Small Talk on the BizTalk Databases
Having said about the BizTalk360 database, let us now talk shortly about the BizTalk databases. BizTalk360 communicates with the BizTalkMgmtDb database to fetch the data of the application and its artifacts for monitoring. The service instances information would be obtained from the BizTalkMsgBox database. Hence, it is necessary to keep an eye on the growth of the BizTalk databases as well in the dba perspective.
There might be a case where the Process monitoring results in the actual count -1. The reason might also be a timeout exception. This will happen when the BizTalkDTADb database is huge in size. This can be found out by checking the database reports to find out the disk usage of the top tables. We can check for the purging of the tracking database. The recommendations to control the size of the databases can be found here.
Conclusion
We hope this article gives you some more information about the BizTalk360 database. When the purging is in place, the database growth can be controlled, and it will be healthy. Happy monitoring with BizTalk360!
The post BizTalk360 Database – An In-Depth Insight appeared first on BizTalk360.
by stephen-w-thomas | Jun 3, 2020 | BizTalk Community Blogs via Syndication
This blog is an extraction of the session “BizTalk360 – The past, the present, and the future” presented by the CEO of Kovai.co, Saravana Kumar at Integrate 2020.
This blog gives an overview of how BizTalk360 started and how the challenges have been enhanced and what will be future of BizTalk360.
About Saravana
He is a hard-core technical person and Microsoft MVP since 2007. As he was working in the BizTalk Server integration space for a decade. He found himself challenging to while work with BizTalk Server projects to manage and monitor the applications and other components. Hence the birth of BizTalk360.
The origin story of BizTalk360
BizTalk360 was founded in 2011. It’s almost 10 years now. Saravana started this as a hobby project, an idea evolved in one of the MVP summit in 2010. The birth of BizTalk360 started and from there, and now it is an essential tool for enterprises that uses Microsoft BizTalk Server.BizTalk360 is completely a web-based admin console that provides a solution to Operations, Monitoring, and Analytics your BizTalk Server environment.
Interesting topics Covered
BizTalk360 – 10 years Journey (Past)
New Features (Present)
What’s next? BizTalk360 – v10 (Future)What is BizTalk360?
The business that uses BizTalk Server usually relies on standard admin console to manage their day to day activities but that won’t sufficient to do complete operational, Monitoring, and Analytical activities. Other third-party solutions like SCOM and custom development solutions are not matured enough to fill the gaps in BizTalk Server. Moreover, in an analytics perspective, BizTalk admins doesn’t have any clue about what is happening in the system? Our tool BizTalk360 is a single operational, Monitoring, and Analytics for the Microsoft BizTalk Server.
A single tool to improve your productivityBizTalk360 addresses some of the key features. Consider if you don’t have a BizTalk360 environment and the way you manage the BizTalk server environment is by using at least seven of the below productivity tools. Such as
- BizTalk admin Console
- SQL Management Studio
- Event viewer
- Performance Monitor
- BAM portal
- ESB Portal
- BizTalk Health Monitor
- Monitoring Consoles
Dashboard
The main challenge in the admin console is that there is not an efficient way to get an overview of their BizTalk server environment status. Whereas in BizTalk360, a customized dashboard is available in order to view the overall health of your BizTalk Server in a single view In this way, you can bring in your business people to support your BizTalk environments.
Event Viewer
When it comes to troubleshooting operational problems in the BizTalk Server, the first-place admins/support person will look is the admin console. If they can’t find anything obvious, their next point of search is the Event Viewer. To address these challenges, “Advanced Event Viewer“ came into existence where all the events from all the BizTalk/ SQL server are stored in a central database.
Performance Tools
BizTalk360 aims to offer an out of box tool with similar capabilities as the Performance Monitor tool in Windows servers. Analytics offers a visual display of the most important performance counters in a consolidated way and arranged on a single screen so that the information can be monitored in a glance.
Business Activity Monitoring (BAM)
BizTalk360 comes with an integrated BAM portal that allows the business users to query BAM views, perform activity search, view the user permissions, and the activities time window.
ESB exception management framework
Even though the Exception management framework itself is a stable offering and fully supported by Microsoft still it is missing some important capabilities like
- Bulk Edit/Resubmit – It is very important since when there is a failure you will typically have 10’s-100’s of failed messages for the same reason.
- Auditing – There is no ability to trace all the activities like edit, resubmit actions.
To address all those challenges as highlighted
Our tool provides a rich ESB exception dashboard with a full search/filter/display of exception details.
BizTalk Health Monitor
BHM is a tool that can be used to analyze and fix issues in the BizTalk server environment. The predecessor of the BHM tool, Message Box Viewer (MBV), was initially built as a hobby project by one of the Microsoft Support Persons.
User Access Policy
On the left-hand side of the image is the standard admin console. Basically standard admin console doesn’t provide any access/security. There is no restriction on users. Once the user gets control of the BizTalk admin console. Users will be able to view the entire admin console.
In the shown example,
Tom is our BizTalk360 admin who has all access to all the integrations in the BizTalk server but John from HR and Claire from finance altogether from different teams. They must not be provided with the same permission as Tom.
This sort of application isolation security has been provided in BizTalk360.
Governance and Audit
The standard BizTalk server Admin console doesn’t come with any in-built auditing capabilities for user activities. Once someone has access to BizTalk Admin Console that is, pretty much entire BizTalk support team, they are free to perform any activities without a traceIn the BizTalk server, if any of the host instances is in stop state or if any of the host instances goes down, such activity has not been traced. In BizTalk360, the auditing section comes as an in-built tool to trace what’s been done in the BizTalk server console.
Advanced Tooling
Some of the advanced tools that have been developed in BizTalk360 in the past 10 years.
- Throttling Analyser
- EDI Manager
- Graphical Message Flow
- Backup/DR visualizer
- ESB manager
- BAM Portal
- Rules Composer
- Tracking Manager
Throttling analyzer
The idea behind the BizTalk360 throttling analyzer is to simplify the complexity in understanding the BizTalk throttling mechanism and provide a simple dashboard view. This helps people to understand the throttling conditions in their BizTalk environment, even when they do not have very deep technical expertise
Graphical Message Flow
In standard admin console, it is time-consuming as everything is stored in a flat-file format. Especially, if you got a more complex, very loosely coupled system the challenges get worse diagnosing/understanding the message flow. And another disadvantage is, when somebody has access to the tracking queries, confidential information might be revealed to unauthorized people.
In BizTalk360, it assists to visualize the entire message flow in a single view where it encapsulates
- All service instances
- Send port
- Orchestration
- Receive port details
Rules Composer
For example, if users want to roll out rule dynamically on Christmas, users need to seek help from development and this can be time consuming process as well. BRE is a simplified tool that will empower business user to manage business rules in a complex business process
Monitoring
Another key capability of BizTalk360 is Monitoring. The tool provides extensive monitoring which ensure your health of the BizTalk server environment. BizTalk being a middleware, it is always connected to various systems, we make sure that all the connected systems are in a healthy state.IBizTalk360 comes with integrated systems like
- SQL
- MSMQ
- IBMMQ
- Web End Points
- Azure Service Bus
- Web Jobs
- SFTP locations
- Disks
We are super proud to convey that no other product provides a complete monitoring tool for the BizTalk server.
Over 20 years of experience in field and that has been covered in Monitoring which meets up the customer requirements.
On a confident note, there is no other product in the market which serves 100% BizTalk Server monitoring.
Biztalk360 can be integrated with your current Enterprise Monitoring Stack
- Service now
- AppDynamics
- Dynatrace
- New Relic
BizTalk360- 10 years of innovationFor the last 9 years, consistently there will be 4-5 releases happening every year. For more information regarding release taken in every year.
Consistently, we are filling in the gaps Microsoft has left for the global BizTalk server Customers.
To align with the latest migration of Biztalk application in the cloud, the same has been brought in BizTalk360.
BizTalk360 has become the de-facto tool for BizTalk server customers. Used by over 600 large enterprises including Microsoft.
The Future
One of the biggest change its been take up is rewriting the entire UI in a new way. Backend API are pretty solid, but front end its initially started with silver light. In the upcoming years, UI will be complete change to Angular 8
Currently, the development team is working on the up-gradation of UI in BizTalk360.
Continued Innovation
The above mentioned is the user forum and based on the customer’s feedback, if it is appropriate it’s been added to the product for every release.
Wrap Up
In this blog post, we discussed, what is the past, present, and future of BizTalk360. Still, there are lots of announcements and feature enhancements that are yet to come. Stay tuned for further updates. Happy Learning!
The post BizTalk360 – The past, present and the future appeared first on BizTalk360.
by Priyadharshini | Jun 1, 2020 | BizTalk Community Blogs via Syndication
Introduction
This blog is an extraction of the session “BizTalk Server 2020 – Migration Path” presented by Sandro Pereira at Integrate 2020.
Back in January 2020, Microsoft announced the availability of BizTalk Server 2020 for public release with lots of exciting features and enhancements. It’s the time to migrate to the new version to enjoy the features. Sandro, Microsoft Azure MVP talks about the upgrade process, key information, and step-by-step instructions for upgrading from any of the previous BizTalk Server version. It also covers the reasons to upgrade to BizTalk Server 2020.
Why should you upgrade your BizTalk Platform?
People in the Integration space would always remember the BizTalk stickers that they receive from Sandro and here it is for BizTalk 2020.
Here are a few reasons:
- Lots of support
- If you are in an older version of BizTalk, then it’s time for you to migrate for continuous support especially the Enterprise clients.
- Environmental constraints
- BizTalk Server heavily depends on performance of the Windows server and SQL server. It’s time to upgrade for better performance
- New features
- Every release of BizTalk comes with a bunch of exciting features for Hybrid Integration
- Support for TLS 1.2
- Support for XSLT 3.0 which is a feature asked for many years
- More privacy and security for GDPR
- Feature packs
- New features can be added through the feature packs, so you need not wait for 3 to 4 years for the new features
- Move to cloud
- Ability to run on Azure VM as IaaS
- Administrator and Developer productivity enhancements
- New Visual Studio console features
What are the migration options?
- Migrate to Azure
- Migrate to BizTalk Server 2020
Migrate to Azure
- Hosted Azure Servers
- This is the easy option as most of them run BizTalk on the Azure VMs. There are options of high availability and, they can be hosted in various regions.
- Migrating to Azure Services
This is completely moving out of BizTalk. All the BizTalk related artefacts will be migrated to the Azure services. It is a complete redesign of the Integration solution. For example, there will be no MessageBox and Service Bus needs to be used to persist the messages. It is a difficult option even with the new BizTalk Migration Tool.
BizTalk Server has a rapid release cadence of major version every 2/3 years with radical changes. Azure is completely different. BizTalk server with the long release history is very stable. On the other hand, there is release for Azure every week and the features get deprecated. There is no one to one mapping in Azure. The process on arrival needs to be improved in Azure. There are some interesting options still missing in Azure like Business Rules.
BizTalk 2020 Migration
In- Place Upgrade
Here is the list of the supported and unsupported migration options:
There are few constraints like the SQL Adapter that has been deprecated, TIBCO Enterprise Message Adapter has been removed and the RosettaNet has issues in BizTalk 202 for which the hotfix needs to be installed for smooth functioning.
Migrate to the brand-new environment
The AS-IS migration
This is without any non-mandatory changes. The .Net framework needs to be changed, the deployment properties need to fix and the solution to be rebuilt.
Migrate and Improve
This needs to be done on the analysis of the business requirements and what needs to be improved on the solution side. This requires more prolonged version of testing as lot of improvements would be added and for the integration between different teams.
With his experience of more than 16 years working in BizTalk, Sandro shares the statistics on the effort of different migration options.
There are some pain points while migrating the BizTalk Server artifacts/components.
- Mappings will be critical
- Older versions are difficult to migrate
- SAP adapter is removed. So, all generated schemas need to be recreated
- EDI might have problems. Because of TPM, parties cannot be directly migrated. You will need to use the BizTalk Server Party Migration Tool
- BTDF is not available
- RFID and Human workflows are removed
- BRE is a simple export/import
How easy is the migration from the older versions
Wrap Up
Hope the insights on the BizTalk Server 2020 gives a clear picture on the migration path. Sandro and Tom Canter are working on the book on the BizTalk2020 Migration which would be available soon.
Happy migrating!
The post BizTalk Server – 2020 Migration Path appeared first on BizTalk360.
by stephen-w-thomas | May 22, 2020 | BizTalk Community Blogs via Syndication
Author Credits: Martin Peters, Senior Consultant at Codit
During deployments of BizTalk application, it is common practice to put BizTalk360 in maintenance mode before the deployment and switch to normal mode again after the deployment.
This avoids alerts being sent to various people due to stopping and starting of BizTalk during the deployment. If you are using Azure DevOps to deploy BizTalk applications automatically, you do not want to have a manual process to put BizTalk360 in and out of maintenance mode.
The good news is that BizTalk360 provides a set of APIs which allow you to automate this.
An article from Senthil Palanisamy named BizTalk Application Deployment Using Azure Pipeline with BizTalk360 API’s inspired me to implement this for a customer. In Senthil’s blog, Powershell scripts are used to access the BizTalk360 APIs and turn the maintenance mode off and on. But, as more and more customers are using Azure DevOps and BizTalk360, you need a copy of the PowerShell scripts. The PowerShell scripts might change over time (changes, bug fixing), so the next step is to put the PowerShell scripts under version control and create a DevOps extension.
The DevOps extension is maintained on GitHub, and compatible changes are automatically distributed to all organizations using this extension.
Single Server Scenario
When your Test, Acceptance, and Production environment consist of a single server, you can create a task group and use this in the release pipeline. Note that in this example, the task group uses the BTDF extension, which is useful if you are using the BizTalk Deployment Framework for deployments.
The BizTalk360 tasks require the hostname of the server where BizTalk360 is installed and the Environment ID. You can find the Environment ID in http://<yourbiztalk360server>/BizTalk360/Settings#api.
Note: You must have a license to use the BizTalk360 API.
If you do not have a license, please contact your BizTalk360 representative. The BizTalk360 API offers an extensive API that allows you to automate other tasks as well.
Multi-Server Scenario
In a multi-server scenario, you want to put BizTalk360 in maintenance mode before deployment to the first server and out of maintenance mode after deployment to the last server. This case, your release pipeline might look like;
You can find the DevOps extension on the VisualStudio Marketplace.
The source code is available on GitHub. Please feel free to make any improvements, enhancements, etc.
The post BizTalk360 Maintenance Mode with BizTalk Deployments via Azure DevOps appeared first on BizTalk360.
by Lex Hegt | May 19, 2020 | BizTalk Community Blogs via Syndication
Continuing our series of webinars around BizTalk360, Atomic Scope, and BizTalk Server, this time we will explain how you can use BizTalk360 to monitor your BizTalk environment effortlessly!
One of the main goals of using BizTalk360 is probably that you want to have peace of mind that your BizTalk environment is healthy, while you are doing smarter things than constantly checking the BizTalk Server Administration console and all kind of other relevant consoles. In this webinar, we will show you several different ways how you can achieve that peace of mind. Among other things, the following topics are discussed.
Organize your Alarms
BizTalk360 gives you a lot of freedom around alarm creation, but this might leave you uncertain around how to approach setting up monitoring. That is why, besides explaining the basics around the different monitoring types and setting up monitoring in BizTalk360, in this webinar, we will be explaining different approaches to organize your alarms.
Inform your Business Users
In general, we see that BizTalk360 notifications are mainly being transmitted to the BizTalk administrators. However, not just BizTalk administrators, and other IT roles, are involved in operating the BizTalk environment. Also, your business users play a role in it.
BizTalk360 has multiple capabilities to support your business users in such a way that they can help themselves. This way they do not need to create support tickets or pick up the phone, they can check their relevant information themselves via BizTalk360!
Return of Investment in BizTalk and BizTalk360
Your organization has heavily invested in a BizTalk platform and have developed all kind of interfaces which are deployed in that platform. By making the best of your investment in BizTalk360, you do not only get ROI in BizTalk360 but also in BizTalk Server. Using a couple of examples, we show you how that happens.
Note: As you know, in early June we will run our annual Integrate event, which will take place remotely this year. In case, you did not register to join us in this event, join this webinar, as we will have something interesting for you!
External Guest
As we have done a couple of times meanwhile, also for this webinar we have an external guest. During this webinar, we are joined by Frédéric Gobert, who is an Integration analyst at ArcelorMittal, which is the biggest producer of steel in the world. Among other things, Frédéric will explain how BizTalk360 is used beside SCOM and Dynatrace and how BizTalk360 has been a lifesaver in multiple scenarios!
BizTalk360 v9.1 has been Released!
Following our pattern of bringing a new release every 3 to 4 months, we have just released v9.1. During the webinar, we will show you what’s new in that release. Are you already interested in knowing more about this release? You can check the Release Notes here.
Conclusion
In summary, do you identify yourself in one or more of the following situations?
- you want to be aware of any issues earlier than your IT colleagues or business users
- you want to be able to monitor pro-actively
- you want to do smarter things than constantly perform manual checks
- you want to take more benefit in your investment in BizTalk Server (and BizTalk360)
- you want higher availability of your interfaces
Then, this webinar is for you! It will take place on May 28th, 10 AM (BST). Just click here to register and leave your details. Shortly we will send you the confirmation mail.
Also, if you think you cannot attend the webinar at the specified time, no worries! Go ahead and register, our team will make sure to send you the webinar recording.
We are looking forward to meeting you there!
The post Webinar Spoiler: Monitor your BizTalk environment effortlessly with BizTalk360 appeared first on BizTalk360.
by Saranya Ramakrishnan | Apr 21, 2020 | BizTalk Community Blogs via Syndication
As you know BizTalk360 keeps up its standard by introducing new features or by improving existing features, that ease the user experience to the next level in every single release. We are delighted to inform you that the next version of BizTalk360 V9.1 will be released for production use within a couple of days!
New Features Introduced in BizTalk360 V9.1
Clustered NT Service Monitoring
BizTalk360 is already capable of monitoring clustered BizTalk and SQL Server resources such as Disks, System Resources, Event Logs, and NT Services. In the case of the failover scenario, the BizTalk360 monitoring service will automatically take the active server for monitoring.
From this version on, the user can monitor clustered NT Services by setting up the expected state of the clustered service as ‘At least One Active’.
Consider a scenario where you have a clustered BizTalk environment, in which your Enterprise Single sign-on service is configured as Generic Type, i.e the SSO service is clustered, and you want to ensure that the SSO service should up and running in at least one of the clustered nodes. This can be easily achieved through BizTalk360 NT service monitoring.
With this feature, you can monitor the state of your NT services in your BizTalk or SQL servers by configuring the Expected state as Started or Stopped. The clustered NT service can be monitored by configuring the expected state as “Atleast One Active”. The BizTalk360 monitoring service will ensure that the configured service is always up and running in any of the respective clustered nodes. An alert will get triggered to the configured email-Id when the service goes down in all the nodes.
NOTE: For better understanding, the clustered nodes are grouped with similar colour code.
The user can also set up AutoCorrect for this. In this case, when the SSO service goes down in both the clustered nodes, then the BizTalk360 monitoring service will automatically try to start the service in any of the nodes. With this, you can ensure that there is no downtime in the service.
Recurrence Scheduled Maintenance
The latest version of BizTalk360 will hold the enriched form of Stop alerts for maintenance, which is renamed as “Scheduled Maintenance”. Schedules can be configured to stop monitoring immediately or at a later point in time, based on your maintenance plan.
Immediate Maintenance
You can set up the maintenance immediately from the current time by enabling the ‘Immediate’ option and providing the maintenance End Date and Time in the schedule configuration.
Future Maintenance
Configure the schedule with Start and End Date/Time to stop monitoring during the future maintenance period. A future maintenance schedule can be configured for both once or recurrence execution.
- One Time Execution – You can create a One Time Schedule, if you want to stop the monitoring during, for example, the deployment of BizTalk applications, which has been planned only once in the future date, i.e on June 1st from 9 AM to 11 AM.
- Recurrence Execution – The schedule will be created once and can be executed multiple times based on the recurrence pattern daily, weekly, monthly, i.e daily 10-11 PM, alternate weekends, 4th Wednesday of every month, etc.
The user can now also edit the schedule and manually stop the maintenance in between, i.e. before the maintenance period gets over. So, it is not required to wait until the end of the configured maintenance cycle.
Schedule actions such as create, delete, modify, stop maintenance are audited for further reference.
Monitoring Improvements
Web Endpoint Monitoring
Monitoring Status Enabled/Disabled is introduced in Web Endpoint monitoring. This enables the user to define when BizTalk360 should start/stop monitoring the configured Web Endpoint.
For instance, you may configure multiple Web Endpoints, but you temporarily don’t want to monitor all the endpoints. In this case, you can set the monitor status as Disabled for those endpoints.
BizTalk360 will start to monitor the configured Web Endpoints only if the status is set as Enabled.
Clone Alarm
Users can now easily copy all the artifacts mapped for monitoring and the data monitoring configurations from one alarm to another. With the Clone alarm feature, users can copy both the alarm configurations and all the artifacts mapping configurations.
MSMQ Age Monitoring
With this enhancement, the user can monitor the oldest message in an MSMQ queue by using the threshold parameter ‘Message Age’. This threshold parameter is added to monitor and alert the oldest message in MSMQ, based on the configured message age units’ Days/hours/minutes.
Mapped Artifacts
All the artifacts that have been configured for monitoring under a particular alarm are listed in the Manage Mapping section in Monitoring Dashboard.
SMTP Notification Channel
The user can define for which error type(s) notifications should be sent through SMTP notification channels, i.e. only in case of Error & Warning or Error/Warning/Healthy state of the artifacts.
Rename Data Monitoring Schedule
The created Data Monitoring schedule can be edited and renamed at a later point in time. Also, users are restricted to create a duplicate schedule for the same alarm under the same monitor type.
Analytics Improvements
SQL Server Performance Widget
Widget to show the SQL server performance (CPU & Memory Usage) is added in the Analytics dashboard.
Reassociate the SQL Query
In the Reporting section, the Secure SQL Query widget can be edited and re-associated to any configured Secure SQL Query.
Event Log Data Collection Optimization
In earlier versions, the Advanced Event Viewer feature collected all the Event log entries for the configured Event Logs and Event Log Sources, but irrespective of error levels. From this version on, to avoid data growth, the user can (de)select the error levels (Information, Warning, Error, etc ) which should be collected. This way, BizTalk360 will start collecting event logs based on the selected error levels.
View MDN Message in EDI AS2
EDI AS2/MDN message content can be viewed as an encoded and wired format.
BizTalk Server 2020 Compatibility
We are proud to share that BizTalk360 is now compatible with BizTalk Server 2020!
Group Properties
Under BizTalk Group properties, users can view newly added properties in BizTalk Server 2020.
- Receive Location (Fault Tolerance)
- Audit Log Operation
Receive Location Advanced Schedule Configuration
The recurrence schedule configured in BizTalk Server 2020 receive locations can now be viewed in BizTalk360.
Conclusion
Considering the feedback from our customers, BizTalk360 will continue to provide more useful features. Why not give BizTalk360 a try! It takes only 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity of your own BizTalk Environments.
The post BizTalk360 V9.1 – New Features and Enhancements appeared first on BizTalk360.
by Mekala Ramesh | Apr 2, 2020 | BizTalk Community Blogs via Syndication
Already there is a lot of buzz happening around the new release of the BizTalk Server 2020 version. It is the most expected version by the integration of community members. This new version consists of many exciting new features and enhancements to ease the life of different business users.
We continue our efforts to deliver uninterrupted services to all our customers across the globe, even during this pandemic crisis. That’s why BizTalk360 is pleased to announce the compatibility of its latest version with BizTalk Server 2020.
As a practice, we aim to bring you long-term focused features which add significant value to our existing and new customers by identifying all sort of obstacles they are facing while supporting/operating their BizTalk environment.
I wanted this blog to cover all the new bringings in the product related to BizTalk Server 2020. Alright then! Here, we will have a look at all the features you will get with the new BizTalk360 version, aligned with BizTalk Server 2020:
- Installer Support
- New BizTalk Group properties
- Receive Location (Fault Tolerance)
- Audit Log Operation
- Analytics
- Receive Location Advanced scheduling operation
Installer Support
We started the implementation from the installer and wanted to eliminate the restriction of this new BizTalk Server version in the installer. The first thing you will want to do is download the latest version of the BizTalk360 MSI. You can get it from the BizTalk360 Free trial page or in the application itself – either is fine, but from the application is easier, as it shows the New version availability notification in the top right bar.
Once you have got the MSI, the installation is a very simple process as BizTalk360 offers ready-to-go installation. As the instructions are very clear and precisely shown in the installer, wizards make your life easier to progress the installation without referring to any other articles. Perhaps, if you are running into any error, you can always refer to the installation documentation. Refer to the below screenshot where we have included the check for new BizTalk Server versions.
The customers who are using older versions, on the other hand, need not worry at all. All the older versions are still compatible with the BizTalk360 even after the latest update.
New BizTalk Group Properties
In the new BizTalk Server 2020 version, a few properties have been implemented in the BizTalk group level. These are:
- Receive Locations
- Enable fault tolerance
- Retry Interval
- Tracking and Reporting
- Audit Management operations
- Maximum number of audit entries
Receive Location Fault Tolerance
Until previous versions, if any error occurs (be it transient) the receive locations will be disabled and effect further message transactions.
This new configuration prevents the receive location from getting disabled on errors.
BizTalk attempts to recover the receive location in an interval you set. The receive location continues running in the other host instances.
Audit Log
Every business transparency is important at a certain level. Perhaps, someone who does anything wrong will lead to a huge impact on business. To bring transparency in BizTalk Server, they have introduced Auditing capabilities. The new properties enable the admins to track all the activities performed by different users and the maximum number of audit entries the user wants to capture. Once the configurations are in place, BizTalk will start to poll the audit data into the audit table in the BizTalk Management database.
To know more about BizTalk Server 2020 auditing capability, refer to this article.
For easy reference, all above mentioned new properties have been added in BizTalk360 in the Group Properties screen as shown below.
Advanced Scheduling Options for the Receive Locations
Now with BizTalk Server 2020, the new advanced scheduling comprises all the options which administrators wanted for scheduling. Based on the configuration, the recurrence option will be displayed in the receive location property screen as shown in the screenshot.
New Options
- The preferred time zone can be set based on the business scenario
- Automatically adjustment for Daylight Saving Time
- Recurrence options like daily, weekly, and monthly
Conclusion
In a nutshell, BizTalk360 is compatible with the new version of BizTalk Server. Our next v9.1 will be released within this week with interesting new features. Why not give BizTalk360 a try! It takes only 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity of your own BizTalk Environments.
The post BizTalk360 Compatibility with BizTalk Server 2020 appeared first on BizTalk360.
by Sivaramakrishnan Arumugam | Mar 22, 2020 | BizTalk Community Blogs via Syndication
The term purge means removing or erase something completely. Data purging is a process or activity mostly heard with Database technologies, i.e. Cleaning up of Database.
There are many different techniques and strategies for data purging, which is often contrasted with data deletion. Deletion is more of a temporary process whereas Purging removes data permanently which in turn frees up the storage and/or memory space for other uses.
Purging process lets you archive the data even though it removed from the main source permanently, giving us an option to retrieve from the archive in case there is a need for it. The deleting process also removes data permanently but doesn’t necessarily involve keeping a backup and generally involves on insignificant amounts of data.
Why Data Purging is Necessary?
For any organization and in any environment, it is important to keep track of database growth and execute the data purging policies whenever necessary.
When the database size grows, the SQL server needs more memory and CPU to read data from the tables, which slows down the database operation.
Besides that, there is also the risk that, due to the growing database(s), you are running out of disk space.
Database Administrators spend a lot of their time dealing with the problem of database processes consuming too much disk space. So, it’s very important to monitor the database size to ensure the database is not seizing the memory and CPU for smooth operation. Also, it’s essential and necessary to automate the purging process.
The Purge Process
The purge process moves data between and deletes data from three categories of data or data sets:
- The current data set, which contains data that needs to be available to users. Users can change or review the data.
- The history data set, which contains data that can only be reviewed or aggregated by users. Users can’t change the data.
- The archived data set, which contains data that falls out of the availability threshold but needs to be stored to meet legal or business requirements. This data is archived, so it can’t be changed, reviewed, or aggregated by users.
Available Options for Purging
In any BizTalk environment, to purge the SQL database there are two main categories of options available. The most common way is
- SQL Server Jobs (by using SQL Agent)
- Windows Services
SQL Server Jobs
The purpose of the SQL Agent is to serve as a job scheduler. Many experienced DBAs use jobs running inside the SQL Agent to perform routine tasks such as backups, updating statistics, and rebuilding indexes as needed.
Scheduling jobs is one of the core SQL Server functions. Many businesses have numerous SQL Server jobs scheduled that perform any number of different tasks from database maintenance jobs like backup and index rebuilds to running queries and kicking off ETL (Extract. Transform and Load) tasks.
What is SQL Agent?
The SQL Agent is the SQL Server subsystem that’s responsible for automatic task scheduling. The SQL Agent is available in all of the editions of SQL Server except the SQL Server Express edition. Taking advantage of SQL Agent enables you to automate many of your routine IT database infrastructure tasks.
Advantages of SQL Agent
The SQL Server Agent gives you a more specific focus around scheduling specific types of jobs around SQL Server. As the Agent is a part of the cluster resource when a failover occurs as long as the cluster has been set up correctly the SQL Agent job will start once the SQL Server Service has started.
Windows task Scheduler is a component of Microsoft Windows that provides the ability to schedule the launch of programs or scripts at pre-defined times or after specified time intervals. SQL Server Scheduler is created for Agent job scheduling, so using SQL Server Scheduler to schedule the job is the best method.
Disadvantages of SQL Agent
If the Agent is not running at the time a job is supposed to run, when the Agent is started it will run the job at the next scheduled time. It will not automatically run the job.
SQL Server Agent jobs that are running at the time of a failover event on a SQL Server failover cluster instance do not resume after failover to another failover cluster node. Jobs that begin but fail to complete because of a failover event are logged as started but do not show additional log entries for completion or failure. SQL Server Agent jobs in these scenarios appear to have never ended. So if your server failover, you need to run this job again in the other node.
Windows Services
Definition of Windows Services
Windows Services is a core component of the Microsoft Windows operating system and enables the creation and management of long-running processes.
Advantages
Unlike regular software that is launched by the end-user and only runs when the user is logged on, Windows Services can start without user intervention and may continue to run long after the user has logged off. The services run in the background and will usually kick in when the machine is booted. An option ideal for use on servers when long-running functionality is needed without interference with other users on the same system.
Data Collection in BizTalk360
BizTalk360 consists own database. Collecting the necessary data is important to be able to represent the data in BizTalk360. As many of you aware that BizTalk360 is a one-stop solution for BizTalk monitoring. So for monitoring and analytics purposes, we do collect different data from various environments (it collects the information from BizTalk, SQL Server, and Tracking database) and stored in the BizTalk360’s database for future references.
- Advanced Event viewer
- Analytics
They are the sparkling examples for the data collection within BizTalk360 when you have configured/added more environments (for each environment there will be multiple machines say example a simple multi-server environment consists of 2-BizTalk servers for HA, 1-SQL server. For cluster servers and Load balancing you can imagine the numbers). However, you don’t want unlimited growth of your database.
BizTalk360 comes out of the box with the “Data Purging” feature to able to manage the size of the BizTalk360 database.
Monitor BizTalk360 Database Growth
There might be a situation where the BizTalk360 database grows due to the amount and size of event log data and performance counter data. The BizTalk360 purging policy will take care of the data growth, but it is also important to monitor the size regularly to ensure BizTalk360 is working seamlessly.
Data Purging in BizTalk360
As mentioned, BizTalk360 comes out of the box with the ability to purge the historic data by setting up the purging duration (days to be monitor) after the specified period. The Administrators/Superusers can set up the “Purge duration” under “Settings -> BizTalk360 Health -> Data Purging”. This will control the database growth and hence the performance of BizTalk360 will not get affected.
Purge Mechanism in BizTalk360
BizTalk360 uses Windows Services to trigger the purging. BizTalk360 owns two services,
- Monitoring service
- Analytics service
Out of those, purging was created as a sub-service of Monitoring service.
In a regular interval time (for every 2 hours) the “Purge Data – a sub-service of Monitoring service” will purge the data by hitting the Stored Procedures for purging available in the BizTalk360 database.
We have chosen the windows service to utilize the advantages of Sub-service and the services that can be started and paused. As most of you aware that the BizTalk360 services should be always running to collect the data and perform few operations, so we would like to effectively use the service rather than creating SQL job and monitor separately. Also, if any exception appears for the sub-service, the exception will be captured from the log files and it will be shown adjacent to the respective sub-service in the UI.
Based on the values set in the data purge duration, for every 2 hours, the data will be purged using the stored procedures with the trigger of Purge Data sub-service.
Until BizTalk360 v5.0, there was a single stored procedure (sp_b360_admin_PurgePolicies) called from our monitoring service which takes care of all the purging. In v6.0, we improved the logic by splitting it into multiple stored procedures (sp_b360_admin_PurgePolicies) which takes care of various parts of data purging.
How to Check if Purging Service Status is Healthy?
Purge Data – a sub-service of Monitoring service from BizTalk360 Settings -> BizTalk360 Health -> Monitoring Service Status -> Purge Data, will help you to understand and find the information that the purging is running for every 2 hours.
If you can compare the started time and Last execution time, for every 2 hours the execution should happen. Even if there is an exception, the exception will be captured in the log files and it will be shown adjacent to the purge data sub-service.
Additional Capabilities
Along with the purging service within BizTalk360, there are few more options to check if the database size is under the expected size.
- Using the Database size monitoring feature, users can monitor the database and log file size of BizTalk and BizTalk360 databases, by simply configuring the error and warning threshold values for the database and log file sizes. By this when the database size exceeds the expected size, you will receive an alert from BizTalk360.
- Use Database Query monitoring to Monitor the size of a database.
Whenever the database grows beyond expectations, the first thing to be checked is the standard database reports. This will ensure which table occupies more space. Then we can act on the purging policy and change it according to the business needs and data flow.
What are the Stored Procedures Involved in Purging?
Till the latest release, there were 14 Stored Procedures involved in Purging for 32 tables. Each stored procedure will purge multiple tables. Let’s see what the stored procedures are used for purging the corresponding tables.
- sp_b360_admin_PurgePolicies_MBVReports – It will purge the BizTalk Health Check tool execution results.
- sp_b360_admin_PurgePolicies_GovernanceAudit – It will purge the complete Governance and Audit tables, which is related to BizTalk and BizTalk360 activities.
- b360_governance_Application
- b360_governance_HostInstance
- b360_governance_ServiceInstance
- b360_governance_rules
- b360_governance_Server
- b360_governance_ESBException
- b360_audit_History
- sp_b360_admin_PurgePolicies_AlertNotification – Will purge the Alert notifications which is sent out from BizTalk360 like Email, SMS and Notification channel.
- b360_alert_History_Email
- b360_alert_History_SMS
- b360_alert_History_System
- b360_historical_Alert_Maintenance
- sp_b360_admin_PurgePolicies_EventLog – Will purge the data of event logs that are already collected from different machines.
- sp_b360_admin_PurgePolicies_Throttling – Will purge the data collection for Throttling analyzer
- b360_td_Throttling
- b360_td_ThrottlingDataCollectionError
- sp_b360_admin_PurgePolicies_MonitoringDashboard – The alarm status update of the monitoring dashboard will be purged
- b360_alert_MonitorExecution
- sp_b360_admin_PurgePolicies_MonitoringLogState – will purge the monitoring log state, monitoring will put the entries of current and expected state.
- b360_alert_Monitor_LogStateCheck
- sp_b360_admin_PurgePolicies_MonitoringLogThreshold – will purge the monitoring threshold log, monitoring will put the current, warning and error levels.
- b360_alert_Monitor_LogThresholdCheck
- sp_b360_admin_PurgePolicies_ProcessMonitoring – will purge the data of all types of data monitoring like Process, Message box data, Tracking data, BAM, EDI, ESB, Logic Apps, and Event log data monitoring.
- b360_st_ScheduleActivity
- b360_st_ProcessMonitorResults
- b360_st_DataMonitorResults
- b360_st_DataMonitorTaskActionResults
- sp_b360_admin_PurgePolicies_PerformanceAnalyzer – The data collected for the Analytics module will be purged
- b360_perf_PerfCounter_Data
- b360_perf_PerfCounter_Expanded
- sp_b360_admin_PurgePolicies_LiveFeed – Live feed data will be purged
- sp_b360_admin_PurgePolicies_TrackingData – The data collected for the Analytics module will be purged.
- b360_dta_Counter_Data
- b360_dta_Counters_Expanded
- sp_b360_admin_PurgePolicies_DBCache –
- sp_b360_admin_PurgePolicies_BizTalkReporting – The data collected for the Analytics – Reporting module will be purged.
- b360_BizTalkDB_DiskUsages
- b360_BizTalkDB_Tbl_sizes
- b360_Report_History_Email
- b360_report_ReportArchive
Purge Settings in BizTalk360
Let’s see how purge helps. The default purging settings in BizTalk360 can be seen in the below screenshot;
Based on the values/duration provided by the purging policy, the data will be purged. We can see that the purging duration for Data Monitoring is 2 months. Hence the historical data for 2 months will be present in the BizTalk360 database.
Purging needs to be done to remove the historical data, thereby making the database healthy. BizTalk360 purges the data by running the stored procedure in the specified duration specified in the settings. The purging settings can be altered by the customers according to their business needs and data flow. If a large volume of data flows through the ports, they can set the purge duration to a minimum value so that data growth is controlled.
The customer configured 3 environments in a single installation of BizTalk360. In each environment, there were 5 servers available due to BizTalk and SQL server. Hence there were 15 servers to collect data for Advanced Event Viewer and they have set-up 30days to monitor. The Database has grown almost to 300 GB because of the number of servers to monitor.
We have recommended the customer to decrease the purge duration from 1 month to 2days and TRUNCATE the particular table (Since the table belongs to BizTalk360 and it is historical data, it won’t affect the BizTalk server). After modifying the purge duration, the data were reduced to 2 GB started working as expected.
Can the Purging be done Manually?
BizTalk360 will purge the data automatically. Sometimes the data collection may be out/beyond our expectation and that leads to a huge volume of data.
Our first suggestion is to reduce the number of purge days. So that the data will be reduced.
Until v6.0, the stored procedure sp_b360_admin_PurgePolicies was used and in recent versions of BizTalk360, you can use the stored procedure sp_b360_admin_CleanupDB to manually purge the database in case of large database growth.
When the number of records in the database grows quickly, call to the stored procedure(s) can start to get timed out from the monitoring services. This will result in data not being cleared and resulting in unusual database growth.
Conclusion
Considering the feedback provided by our customers we have improved and fine-tuned the collection of the data into the BizTalk360 database.
Whenever the database grows beyond expectations, the first thing to be checked is the standard database reports. This will ensure which table occupies more space. Then we can act on the purging policy and change it according to the business needs and data flow.
If you have any questions, contact us at [email protected]. Also, feel free to leave your feedback in our forum.
The post Data Purging in BizTalk Server and BizTalk360 appeared first on BizTalk360.
