Why did we build ESB Exception Management Portal in BizTalk360?

Why did we build ESB Exception Management Portal in BizTalk360?

BizTalk360 ESB Exception Management Portal

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

Microsoft shipped the ESB Toolkit back in 2007-2008 that extends the functionality of Microsoft BizTalk Server to provide a range of new capabilities for building SOA/ESB applications that incorporate things like itinerary based invocation for lightweight service composition without using Orchestrations, dynamic resolution of endpoints maps, Web services, exception management and reporting. They also provided a sample “ESB Exception Management” web application along with the toolkit.

The ESB Toolkit created a love/hate relationship with a lot of BizTalk Server customers even though it added a lot of value addition to BizTalk Server (as an add-on) while in some cases it simply complicated the solution.

The one thing that everyone loved about ESB Toolkit is the Exception Management framework and the sample web application that shipped with the Toolkit. Since the Exception Management part addressed two important challenges in an integration solution in an end-to-end Exception management framework, visualizing it in a web portal and ability to edit/resubmit failed messages, which were missing in BizTalk Server core.

What are the current challenges?

Sample Portal: The Exception Management Framework itself is a stable offering and fully supported by Microsoft. However, the portal that shipped with the ESB Toolkit is a “sample web application” built on top of the ESB Exception Database. It’s not fully supported by Microsoft and it’s kind of a half-baked solution with a lot of bugs.

Difficult to Install and Configure: Typically it takes few hours to one or two days to install and configure the ESB Exception management portal. It’s not maintained or updated as required over the years, and it uses certain components like older versions of “.NET Logging Application Blocks” that makes server level changes and affects your main BizTalk Solutions.

Missing Functionalities: As the ESB Exception Portal was shipped as a simple sample web application, it’s not matured and misses some important capabilities like “Bulk Edit/Resubmit” and functional alerting. Bulk edit/resubmit is very important since when there is a failure in your environment, you’ll typically have 10’s-100’s of failed messages for the same reason and you wanted to take bulk action.

The other important missing aspect is restricting users by permission and auditing, an example – you probably do not want all of your support people to have the ability to edit and resubmit messages. Even if they do, you need to have the traceability of who performed that action.

How BizTalk360 solves this problem?

We wanted to address the challenges highlighted above and also wanted to give a rich unified tooling experience for BizTalk Administrators. Hence we built the ESB Exception Management portal within BizTalk360 from the ground up. All you need to configure the ESB Exception management within BizTalk360 is to simply provide the connection string to your ESB exception database relevant to your BizTalk Environment. That’s it! You are set (takes about 3 minutes).

ESB Exception Management Portal BizTalk360

Since we built the ESB Exception Management portal from the ground up, we have thought through all the challenges in the sample portal and addressed them. It comes with the following set of features

  • Rich ESB Exception Dashboard (utilizing our powerful customizable dashboard framework)
  • Full Search/Filter/Display of exception details
  • Edit Resubmit – both single and multiple records
  • Download Messages – you can either download or email exception messages directly from the portal.
  • Integrated Knowledgebase – you can associate a Knowledgebase article with known exceptions
  • Security – ability to restrict user either to the full ESB portal section or allow them to do specific tasks like edit/resubmit
  • Governance & Auditing – all the core activities like editing/resubmit by the users are audited.
  • Rich Functional alerting – ex: if there are over 30 errors matching a specific error code in an application alert the administrator.

One of the other important objectives of BizTalk360 is to reduce the number of different tools the BizTalk Administrator has to use to support their BizTalk solution – tools such as Admin Console, BAM Portal, ESB Portal, SQL Management Studio, Perfmon and so on. This makes them totally unproductive and switch context between different applications. It is also difficult to on-board new people and bring them up to speed.

By bringing the ESB Exception Management portal within BizTalk360, we eliminate the need to use the sample ESB portal that comes with the toolkit.

Get started with a Free Trial today!

If you are struggling with all the above mentioned challenges, why not give BizTalk360 a try. It takes about 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity on your own BizTalk Environments. Get started with the free 30 days trial.

BizTalk360-Free-Trial

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.

Why did we build Advanced Tracking Manager in BizTalk360?

Why did we build Advanced Tracking Manager in BizTalk360?

Advanced Tracking Manager BizTalk360

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

Message Body and Message Context (properties) tracking is one of the key characteristics of BizTalk Server. A typical flow in BizTalk Server will involve a message that is received from the source system via the receive port, then transmitted to an orchestration (not always) for some business process, and then finally transmitted to the destination systems via the  send ports. During this journey, the original message might have gone through various processing and transformations. It’s important to keep track of all the state changes by logging the message content and context at each change point. BizTalk Server provides this functionality out of the box through a very rich tracking mechanism at various points of the journey.

You can track the message body, properties and events at various points as shown below.

Receive Port & Send Port (One way)

  • Message Bodies  – before and after port processing
  • Message Context  – before and after port processing

Receive Port & Send Port (Two way)

  • Message Bodies (Request) – before and after port processing
  • Message Context (Request) – before and after port processing
  • Message Bodies (Response) – before and after port processing
  • Message Context (Response) – before and after port processing

Orchestration

  • Message Bodies – before and after orchestration processing
  • Message Properties – incoming and outgoing messages
  • Track Events – Orchestration start and end
  • Track Events – Message send and receive
  • Track Events – Shape start and end

Pipeline

  • Message Bodies – before and after pipeline processing
  • Track Event – Port start and end
  • Track Event – Message send and receive

Business Rule Policy

  • Fast Activity – track instance data
  • Condition Evaluation – true/false results of condition
  • Rule Firing – actions started as result of policy
  • Agenda updates

Schemas

  • Promoted Properties

As you can see, BizTalk Server provides a very rich tracking framework and mechanism to track pretty much anything that goes through the system. It also gives the end customer full control of what data/property/event needs to be tracked at precise points in the message flow.

What are the current challenges?

Even though BizTalk Server comes with a strong framework for Tracking, the management capabilities of tracking settings is not ideal in the standard BizTalk Admin Console.

Central Management: In the BizTalk Admin Console, there is no option for you to visualize the tracking configuration of all the entities (like receive port, pipeline, orchestration etc.) from a single place. If you wanted to view or change a tracking setting, you need to manually visit each entity  and make required changes.

Bulk Actions: You cannot perform bulk actions like changing the tracking settings of all the entities with a single click.

Undesired Tracking Configuration: Often times when the BizTalk applications are migrated from development or QA environments to production, the tracking settings are also copied unwittingly (during export/import binding files). In the development and QA environment you typically have more tracking, but in production you’ll have only required minimum tracking due to performance reasons.

How BizTalk360 solves this problem?

BizTalk360 addresses all the above challenges with the “Advanced Tracking Manager” functionality.

biztalk360-tracking-manager

As you can see from the above screenshot, BizTalk360 provides a single unified view of the tracking configuration in a single place with easy to understand colour coding. You can easily make bulk changes with almost single click. You can visit the Advanced Tracking Manager at any time and see the level of tracking you have for the environment.

We have noticed interesting customer scenarios where they typically turn off all tracking for performance reasons and the moment they need to identify some problem, they enable required tracking via Advanced Tracking  Manager and turn it off once the diagnosis is done.

In another customer scenario, as part of the deployment process check list, they check the tracking settings in the environment after each BizTalk Application deployment. This reduces the risk of having unwanted tracking settings in their production environment.

If you are interested, there is a more detailed article on all the capabilities of Advanced Tracking Manager.

Get started with a Free Trial today!

If you are struggling with all the above mentioned challenges, why not give BizTalk360 a try. It takes about 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity on your own BizTalk Environments. Get started with the free 30 days trial.

try biztalk360 for free

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.

Why did we build Centralized Advanced Event Viewer for BizTalk Administration?

Why did we build Centralized Advanced Event Viewer for BizTalk Administration?

Advanced Event Viewer BizTalk360

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

When it comes to troubleshooting operational problems in BizTalk Server, the first place the BizTalk administrator or support person will look is the standard BizTalk Admin Console. Once if they can’t find anything obvious in the Admin Console, their next point of search will be the Event Viewer in the physical BizTalk Server machines and sometimes in the SQL Server machines.

Here are few challenges with this approach —

Correlating events across multiple BizTalk/SQL Servers: A typical BizTalk Enterprise production environment will have at least 2 BizTalk servers and 2 SQL servers to make the environment highly available. If there is a problem, you need to look into the Windows Event Viewer of each server and try to correlate and nail down the problem. This is time consuming.

Security Challenge – Administrator Access to BizTalk/SQL Servers: If you want to have access to Event Viewers in Windows Servers, then you need to have pretty much Administrator rights on the server(s). This is something that the organization needs to avoid.

Too much noise in the Event Viewer: Generally, every single problem or logging that happens in Windows Servers goes into the Event Viewer. Even though Event Viewer has the concept of Categories, it will still be noisy and the Administrator needs to have enough knowledge to understand what they are looking for in the Event Viewer. This could be a potential time loss exercise.

What is the current limitation in BizTalk Server?

BizTalk Server out of the box does not have any special tooling to address this challenge. It simply relies on Administrators to use the standard Windows Event Viewer MMC snap in. This will result in all the three challenges highlighted in the previous section. What this means is that people have gotten adapted to certain ways of solving the problems and Microsoft BizTalk Server lacks the proper tooling to address the challenge.

How does BizTalk360 solves this problem?

BizTalk360 comes with a key feature called “Centralized Advanced Event Viewer”.  It periodically collects the events that are related to BizTalk (out of the box we collect certain Event Log Categories, Administrators can also add additional categories) from all the BizTalk and SQL servers in the environment and store them in a central database. The collected data is then searched and then made viewable in the BizTalk360 web based user interface.

biztalk360-advanced-event-viewer

This approach seamlessly solves all the problems we highlighted previously. The BizTalk Administrators or support people need not physically access the servers via RDP (remote desktop) connection. They can easily correlate events in a single place and construct complex filters like for example: events related to Enterprise SSO across all the BizTalk Servers for a time range.

What is the business value?

Address Security Concerns: Businesses do not want too many people logging on/off into their production servers with highest privileges during business hours. This can be rectified using BizTalk360 Advanced Event Viewer.

Reduce time to diagnose issue: It’s much faster to look into the reason for errors in one place rather than looking at many places and try to correlate events. This dramatically reduces the time it takes to solve the problem. In a typical BizTalk Server troubleshooting exercise, this is how the operations/support person loses a ton of time.

Get started with a Free Trial today!

If you are struggling with all the above mentioned challenges, why not give BizTalk360 a try. It takes about 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity on your own BizTalk Environments. Get started with the free 30 days trial.

try biztalk360 for free

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.

Why did we build the Secure SQL Queries capability in BizTalk360?

Why did we build the Secure SQL Queries capability in BizTalk360?

Secure SQL Queries BizTalk360

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

In a BizTalk Server support scenario, one of the common things that happen is people accessing the BizTalk System and some custom databases regularly using tools like SQL Management Studio to run some reporting queries. Some of the system databases will include BizTalk Message Box (BizTalkMsgBoxDb), BizTalk Management (BizTalkMgmtDb), BizTalk Tracking  (BizTalkDTADb), Business Activity Monitoring – BAM (BizTalkPrimaryImport).  BizTalk Administrator and support people access these databases for a variety of requirements such as to check tables like spool or host tables in Message Box, Message In-out table in tracking database, looking for some values in BAM database tables, cross reference data, check for large messages etc.

What are the current challenges?

Performance Impact: Some of the BizTalk databases can grow extremely bigger. It’s not uncommon to have over 1 million records in certain tables (ex: MessageInOut table in Tracking Database). Executing some queries like just a plain “SELECT * ..” on those tables without any query optimization or locks can have serious performance impact during business hours.

Productivity Improvement: BizTalk Support Team will typically have a bunch of standard SQL queries they run on a regular basis to get some reporting on BizTalk databases. Each individual team members will have their own set of SQL queries and they manage it in their local machines.  This basically results in two things, useful reporting queries are not shared between team members and maintaining the queries in SQL files and accessing different files for different environment is not productive.

No consolidated tooling: If BizTalk Administrator or support person needs to run some reporting queries, they need to use an external tool like SQL management studio.

What is the current limitation in standard BizTalk Admin Console?

The standard BizTalk Admin Console does not come with any capabilities related to accessing SQL data. This leaves the support people to rely on external SQL tools like SQL Server Management Studio (SSMS). Tools like SSMS are designed for DBA’s and Developers and it’s not particularly friendly when it comes to pure support and management – a level of technical expertise is required to use SSMS.

How BizTalk360 solves the problem?

We understood the practical challenges of not having an integrated SQL data access tool along with BizTalk Administration and the challenges it exposes (as explained above). This resulted in building “Secure SQL Queries” platform in BizTalk360.

The principles are quite simple — a DBA or someone who understands SQL well writes optimized SQL queries and store them with friendly names in BizTalk360. Ex: “Depth of Message Box Spool Table”. Then, assign who will have permissions to run the queries. May be not all the queries are required for every support person. BizTalk360 can then work like a reporting tool for these custom SQL queries and display them in the web interface directly.

secure-sql-queries-biztalk360

This approach eliminates people from directly running non-optimized queries in production environments and it also simplifies having access to SQL data within a single web console instead of switching tools.

One of the biggest advantage is that useful SQL reporting queries in your BizTalk environment can be centrally stored and shared between team members.

NOTE: The platform is very generic; you can store and execute SQL queries across any databases and not necessarily just BizTalk Server databases as long as the service account has right privileges to access the data.

What is the business value?

The business benefits are very obvious. Providing a secure web based platform for accessing SQL sever data (which is critical to the day-to-day operations of any BizTalk Server solution) increases the productivity of support/BizTalk Admin people and makes it extremely secure. In general, you can keep your database administrators (DBA’s) happy in the organisation.

Get started with a Free Trial today!

Download and try BizTalk360 on your own environments free for 30 days. Installation will not take more than 5-10 minutes.

BizTalk360 30 Days Free Trial

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.

Why did we build Team Knowledgebase feature in BizTalk360?

Why did we build Team Knowledgebase feature in BizTalk360?

Team Knowledgebase BizTalk360

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

Microsoft BizTalk Server being a middleware platform sitting right in the middle of the organisation connecting all the systems together, it’s inevitable there will be some regular operational challenges. When you are talking to multiple systems, it’s hard to control every end systems. Some of the common problem you encounter will be a partner sending the message that contains some unknown characters, having the format of the date in wrong format etc.

In BizTalk Admin console, when such problems occur and depending on how the exceptions are handled most of the time, it will result in some form of suspended service instances with some error code. Typically a BizTalk support person will start analysing it within the BizTalk Admin console or via the Event Viewers in BizTalk Servers and find a relevant solution.

Over a period, the BizTalk support team will know the answers for pretty much all the known issues and they will be able to solve the problem quickly. However, what happens if someone leaves the team or you bring someone new to the team. It will take equal amount of time for that new person to pick up all the known issues.

Few organisations maintain some kind of internal wiki or SharePoint documents to cover all the known issues, however they are not always updated and in sync, multiple documents, and no one actively look into those documents for solution. They always tend to start from scratch and end up spending time diagnosing the problem.

To address this particular challenge we have built “Team Knowledgebase” capability within BizTalk360.

What is the current limitation in BizTalk Server Admin Console?

BizTalk Server Admin console doesn’t come with any such capability to address this real world challenge. BizTalk Admin Console is designed more for hard-core technical BizTalk people and it expects a level of BizTalk knowledge, it only displays the technical error descriptions whenever there are suspended instance.  There is no option for team collaboration and ability to store solutions to the problem.

How does BizTalk360 solve this problem?

We identified this feature could be a huge time saver when it comes to BizTalk Solutions support and also it helps to bring non-BizTalk people into supporting BizTalk solutions easily.

BizTalk360 provides a platform to store team knowledge base articles for known issues. Example: If an instance gets suspended with some error code “0x34564” for a particular application, the support person can easily write a short knowledge base article saying this particular error happens due to “Known issue with our partner Contoso, their internal systems are not capable of sending date time in the format our system is expecting”, the support person can also set the rules saying this is applicable for Application XYZ, the instance status should be “Suspended-Non resumable”, the Environment should be “Production”, the error description text should contain “Contoso”.

biztalk360-team-knowledgebase

Once the knowledge base article is created with set of rules, BizTalk360 will keep watching for any errors that happens in the system and if it matches an existing knowledge base condition, it will show a small book icon next to the error. This helps the support person to quickly realize it’s a known issue and also see the solution to the issue.

The team knowledge base capability is available for Suspended Instances, Event Viewer, ESB Errors, Throttling Conditions in BizTalk360.

What is the business value?

As you can see the business value of the team knowledgebase feature is huge, it drastically reduces the time taken to troubleshoot the problem, especially for the repeated ones. The feature also helps to mitigate the risk of good people leaving the organisation with the knowledge and also helps to on-board new people into the team and bring them up-to speed pretty quickly.

Get started with a Free Trial today!

Download and try BizTalk360 on your own environments free for 30 days. Installation will not take more than 5-10 minutes.

try biztalk360 for free

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.

Why did we build Auditing & Governance for BizTalk Server Administration?

Why did we build Auditing & Governance for BizTalk Server Administration?

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

In the previous article “Why did we built User Access Policy to Manage BizTalk Server Security?”, we looked at the importance of securing the BizTalk Server environments, what are the limitations of existing security mechanisms in BizTalk Server and how BizTalk360 User Access Policy mechanism helps to address the gaps. The other important aspect that’s more closely related to security is the governance & auditing.

In a nutshell, Governance and Auditing simply means recording all the activities performed by a BizTalk Administrator or Operations person in your BizTalk Server environments. Auditing is such a crucial part for any enterprise software. Let’s take some example scenarios and see how the impact of such activities can cause huge business disruption.

Let’s imagine you have an integration scenario picking up purchase orders from a FTP location, processing it via a BizTalk Orchestration and finally sending it to the SAP system. In this simplest scenario, a BizTalk Administrator can potentially do the following 5 activities intentionally or accidentally —

  • Disabling the BizTalk FTP Receive Location
  • Unenlisting the BizTalk Orchestration
  • Stopping the BizTalk SAP Send Port
  • Stopping the Host Instances that run receive location, send port and orchestration
  • Terminate a BizTalk Service Instance that’s processing the purchase order

Any one of the above activity would have resulted in a business impact of not processing that purchase order.

When such incidents happen, you must have the system in place to look at the audit logs and see who actually performed such activity and take necessary steps. In critical industries like Healthcare and Financial institutions, Auditing and Governance are mandatory and governed by industry bodies like SOX and HIPAA.

What is the current limitation in BizTalk Server?

The standard BizTalk Server Admin console doesn’t come with any in-built auditing capabilities for user activities. Once someone has access to BizTalk Admin Console (i.e pretty much your entire BizTalk support team), they are free to perform any activities without a trace.

As mentioned in the previous section, pretty much every single activity the BizTalk Server administrator or support person performs in a controlled environment like Production will have significant consequences.

It’s a very common scenario in large teams for no one taking the blame when things go wrong. Question like “Do you know who stopped that host instance?” are common. The difficult part is you probably don’t know how long that particular host instance was in stopped state, since you don’t have the audit trace.

How does BizTalk360 solve this problem?

Once we built the web based BizTalk Server Admin console, the first top most priority we addressed in the product is sorting out the Security and Audit capabilities for administrative activities.  As you can see from the below picture, all the actions performed by the BizTalk Administrators are logged/audited.

BizTalk360 Governance Auditing

The actions could be something related to BizTalk Applications like starting/stopping Receive Locations, Send Port, Orchestration, it could be related BizTalk Host Instance like starting/stopping host instances, Service Instance activities like terminating, resuming, suspending, ESB management activities like resubmitting messages, and so on.

For every new feature we add to BizTalk360, we make sure auditing capabilities are also taken care of. The organisations can keep the audit data for however long they want based on their corporate policy; you can easily configure the data retention period in BizTalk360.

In order for us to build a system that’s capable of auditing user activities, we need to make sure BizTalk Administrators can use BizTalk360 instead of the standard BizTalk Admin console, that means covering each and every feature that’s available in BizTalk Admin Console needs a counter part in BizTalk360. Apart from deployment and configuration changes (ex: changing the password of FTP receive location), BizTalk Administrators can perform every activity in BizTalk360 that can be performed in the standard BizTalk Admin Console.

Can we audit user activities if it’s performed via BizTalk Admin Console?

This is one of the common questions we receive when we talk about Governance and Auditing. Unfortunately we cannot capture activities that’s performed outside BizTalk360. The idea is you restrict access to standard BizTalk Admin Console to very few people (super users) and force majority of the people to use BizTalk360 for controlled environments.

Get started with a Free Trial today!

Download and try BizTalk360 on your own environments free for 30 days. Installation will not take more than 5-10 minutes.

BizTalk360 30 Days Free Trial

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.

Why did we built User Access Policy to Manage BizTalk Server Security?

Why did we built User Access Policy to Manage BizTalk Server Security?

BizTalk360 User Access Policy

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

Being a middleware product, BizTalk Server is often going to sit right in the center of the organisation. All your critical backend systems like SAP, CRM, Oracle, SQL, and so on are connected to each other via BizTalk Server. So in theory, if someone has full access to your middleware platform like BizTalk Server, technically they can also get access to any of the underlying systems.

When it comes to security or performance, it is determined by the weakest point in the link chain. You might have created a highly protective security wall for each one of those critical backend systems, but when there is a punch whole for an external system (BizTalk Server) to get through, it’s important to protect the security of that external system to the same strength as your backend system.

We felt the current Administrative Security capabilities of BizTalk Server is very limited (will see in detail) and hence the birth of Advanced User Access Policy & BizTalk Server Security feature in BizTalk360.

What is the current limitation in BizTalk Server?

When it comes to day-to-day BizTalk Server Administration and Security, currently the scenario is like this. The Administrator will access the BizTalk Environment using the BizTalk Admin Console, depending on whether he/she belongs to either “BizTalk Server Administrators” or “BizTalk Server Operators” Window NT Group. Certain features will be enabled or disabled in BizTalk Admin Console based on these user roles.

There are few challenges in this approach. If you are a member of the BizTalk Server Administrators group, you have all the access required wide open. However, if you are a member of the BizTalk Server Operators group, your options are limited and the rules are hard coded by Microsoft in terms of what you can and cannot do.

The table below shows the actions that can and cannot be performed by members of BizTalk Operators Group

Can DoCannot Do
View service state and message flowModify the configuration for BizTalk Server.
Start or stop applications (Send Ports, Send Port Groups, Orchestrations, Receive Locations)View message context properties classified as Personally Identifiable Information (PII) or message bodies.
Terminate/Resume Service InstancesModify the course of message routing, such as removing or adding new subscriptions to the running system, including the ability to publish messages into the BizTalk Server runtime.

In pretty much every case, those rules are not practical for the day-to-day administrative work and pretty much within days every single person supporting/administering the BizTalk Server environment will be working with Administrative privileges.

Common challenges in BizTalk Server security

Let’s highlight some of the common day-to-day challenges that are not addressed by the out of the box BizTalk Server security mechanism.

Application Level Security: Your BizTalk Server environment will be a shared environment in many organisations. For various practical and cost reasons, you are not going to have many BizTalk Server environments. However, many business units in the organisation will be deploying their integration solutions into the platform. In such scenarios, for example Team A might need access to Applications A,B,C and Team B might need access to Applications X, Y, Z. You cannot set this level of Application isolation security using the standard BizTalk Server security mechanism.

Read Only Access: In some cases, you might want to give read-only access to your BizTalk Server environments. Example: It’s very common to allow few developers who worked on the project to have access to production environments. You wanted to make sure developers do not modify or deploy anything without change control. This is not possible with the standard BizTalk Server security mechanism.

Mixed Privilege Scenario: For practical reasons, there will be scenarios where you wanted the support person to have some level of mixed privilege. Example: They might need to start and stop the BizTalk Host Instances periodically, they may need to turn on/off tracking settings, or you may NOT want them to terminate or resume service instances etc.  If you want to achieve this, there is no other option than making that person (or team) part of BizTalk Server Administrators group. This basically results in no security.  As you can see, hard coded rules are not practical and it results in elevated privilege. Sadly this is the scenario with most organisations.

Security for different tools: The default security mechanism only covers the BizTalk Admin console and WMI access. However, when you are working with BizTalk Server, there are at least 5-8 other tools you’ll be using such as BAM Portal, ESB Portal, Event Viewers, Azure Services etc. Currently they all need their own security mechanisms, making it super complex and often vulnerable.

Multiple BizTalk Environments: Typically, organisations will have 2-3 BizTalk Server environments. Example: Production, DR, QA (System Integration, User Acceptance) etc. You can simply multiply the above mentioned problems by number of BizTalk Server environments you have.

How does BizTalk360 solves these problems?

From the above points discussed, it is pretty clear and obvious that a better security mechanism is required when it comes to day-to-day BizTalk Server  Administration and Operations. Let’s take a quick look at how BizTalk360 addresses the challenges in a seamless way.

The below screenshot shows how you can add BizTalk Administrator/Operator to your BizTalk environment. There are few things to note — in the first place, BizTalk360 supports multiple BizTalk Server environment management from a single console; so you can set up security and access rights from a single place. You can either configure security for individuals or as team (ex: Create an NT Group called “BizTalk Production Support”)

biztalk-server-security-biztalk360-user

In the second screenshot below, you can see that you do not need to give access for the administrator to complete BizTalk Server environment. You can carefully choose required BizTalk Applications for which a BizTalk Server administrator requires access. The entire BizTalk360 application is context aware, for example, if you have given permission for Application A,B,C and they depend on specific Host and Host Instance, only those related host/host instances will be visible for the user.

biztalk-server-security-biztalk360-applications

In the final screenshot, you can see how BizTalk360 provides full flexibility in terms of what level of permission you wanted to assign for the user. There are no hard coded rules; this makes it practical to give the correct level of permission to BizTalk Administrators/support people. BizTalk360 also comes with pre-defined security templates which you can apply as shown below. The “Can Action” section basically allows users to take some actions in the BizTalk Server environment, for example, if you do not want the user to terminate/resume service instances you can simply uncheck the check box “Service Instances”.

biztalk-server-security-biztalk360-features

What is the Business Benefit?

Every business likes to run their systems in a secure way. You do not want your critical business systems like BizTalk Server to run with security challenges. This particular feature in BizTalk360 was one of the first key feature built in the product after the original web based BizTalk Admin Console functionality is completed.

Get started with a Free Trial today!

If you are not using BizTalk360, it is pretty clear that you are compromising on certain Administrative security aspects in your BizTalk Environment. So, why don’t you download and try BizTalk360 on your own environment. We provide 30 days free trial of the fully functional product.

try biztalk360 for free

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.

Why did we built BizTalk Host Throttling Monitoring?

Why did we built BizTalk Host Throttling Monitoring?

BizTalk360 Host Throttling Monitoring

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

BizTalk Server being a Middleware product connected to various legacy backend systems, it needs to make sure the entire ecosystem can work in an optimal way. If one of the legacy system connected to BizTalk is slow for any reason, then BizTalk Server needs to act sensibly and not to overload that system with messages more than what it can handle. In such scenarios, BizTalk Server will throttle itself (slow down itself) and make sure the messages are delivered to the backend in an optimal rate.

BizTalk Server achieves this capability by continuously monitoring various performance counters (memory footprint, thread count, message publishing rates, database size etc.) and self tuning itself. There are over 50 performance counters related to throttling in BizTalk Server which monitors both inbound and outbound traffic.

So it’s important to understand whether your BizTalk Environment is working correctly or under throttling condition.

What are the current challenges?

BizTalk Expert Required: If you wanted to analyse and detect a throttling condition in your BizTalk Environment, an experienced BizTalk person is required. There are no out of the box tooling from Microsoft to understand throttling conditions. The person should configure all the required Perfmon counters, collect data, analyse it and predict any throttling condition.

Time Consuming: Even if you have an experience person, setting up Perfmon counters, data collection and analysis is time consuming process. In our experience, it takes anywhere from a day to a week to fine tune Throttling in BizTalk Environment.

No Monitoring: There are no out of the box monitoring solution for BizTalk Throttling conditions.

How BizTalk360 Solves the problem?

BizTalk360 addresses this problem by introducing two important features called BizTalk360 Throttling Analyser and BizTalk Host Throttling Monitor.

Throttling Analyser periodically collects all the required Performance Counters relevant to throttling and stores the values in a SQL database, so there is no necessity for setting up Perfmon as and when required. Once the data is collected, it presents the status in a very interactive graphical dashboard. The BizTalk Administrators can easily visualize the throttling condition and take appropriate action. The throttling data is kept for last 7 days, so you can login to BizTalk360 anytime and see historic throttling condition as well.

BizTalk360 Throttling Analyser

Throttling Monitoring is the new addition to the product where the Administrator can enable BizTalk Host Monitoring in almost a single click or fine tune it according their requirements. Once configured, BizTalk360 will periodically check for any throttling condition violations and alert appropriate users via various notification channels like Email, SMS, Slack, etc.

BizTalk360-Host-Throttling-Monitoring

Interested to try this feature?

Download and try BizTalk360 on your own environments free for 30 days. The installation will not take more than 5-10 minutes.

try biztalk360 for free

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.

Why did we build monitoring for IBM MQ, MSMQ, Azure Service Bus Queue?

Why did we build monitoring for IBM MQ, MSMQ, Azure Service Bus Queue?

MSMQ, IBM MQ Azure Service Bus Queue Monitoring

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

BizTalk and Queuing Platforms

In any Enterprise integration scenario, message queuing technologies like IBM MQ, MSMQ, Azure Service Bus Queues etc., play a significant role. Message Queuing provides the solution to some critical business scenarios like the store and forward, first in first out, batch processing and so on.

Often times, queuing platform works more or less like a database — it accumulates the messages that was received from the source systems. The real power of the platform is realized when messages are pulled and utilized by the downstream systems like order processing, shipping etc.

BizTalk Server comes with native adapters for IBM MQ, MSMQ, Service Bus making it a powerful platform when your integration solution needs to communicate to Queuing endpoints.

Why do we need this feature?

Given the above scenario, the importance of a queuing platform in an Enterprise integration solution is pretty clear. In an integration scenario, it’s important to make sure both the BizTalk Server platform as well as the systems to which it is connected are all working seamlessly without any downtime. If an order receiving IBM MQ is down, then the entire integration journey of end-to-end order processing is also down. Hence it’s pretty important, you monitor those external queuing systems in addition to your BizTalk Server platform.

What are the current challenges?

No Monitoring: There is no out of the box monitoring solution that comes with BizTalk Server to monitor either BizTalk Server platform or the external sources like IBM MQ, MSMQ, Azure Service Bus Queue etc.

Custom Solutions: Often, developers write some C# or PowerShell scripts to monitor the external Queues which is hard to maintain, which will not have a good configuration experience, and will go out of sync over a period of time. Even though at the face it will look like few lines of code, real implementation will require correct type of client libraries, handling SSL certificates etc.,

Limited 3rd party solutions: There are third party options available to monitor the queues, however, they come with limited functionality. It’s not tightly integrated with your BizTalk Solution and the configuration is not seamless.

How BizTalk360 solves this problem?

The monitoring for Queuing technologies that work with BizTalk Server is built from the ground up in BizTalk360. The user experience of configuring the monitoring for Queues is seamless. If your environment uses any Queue based BizTalk Receive Port/adapter or Send Port/adapter, BizTalk360 picks up all the configuration from them. You only need to specify the conditions you are interested in monitoring.

BizTalk360 IBM MQ ServiceBus Queue MSMQ Monitoring

Here is the list of supported thresholds for different Queuing technologies that comes along with BizTalk360

IBM MQ:

  • Current Queue Depth
  • Current Queue Usage %
  • Backout Queue Depth
  • Backout Queue Usage %

MSMQ:

  • Queue Size
  • Active Message Count
  • Journal Message Count
  • Dead Letter Message Count

Azure Service Bus Queue:

  • Queue Status
  • Active Message Count
  • Dead Letter Message Count
  • Transfer Message Count
  • Queue Size
  • Message Count
  • Transfer Dead Letter Message Count
  • Scheduled Message Count
  • Queue Size

You can combine multiple check conditions using a logical AND/OR as shown in below, this makes it extremely powerful

BizTalk360 Queue Configurations

General Monitoring Benefits

BizTalk360 monitoring comes with a set of standard features that’s applicable to all monitoring plugins. The Queue monitoring can also take advantage of the following capabilities.

Import/Export: The import/export function allows you to move your monitoring configuration from one environment to another. Example: From your UAT/Staging environment to Production.

Notification Channel: BizTalk360 comes with a list of Notification Channels like email, SMS, slack, HP Operations Manager etc., making it powerful to notify administrators as soon as something goes wrong.

Monitoring Dashboard: BizTalk360 comes with rich monitoring dashboard that visualizes the system health in a graphical way.

Notification History: You can also keep track of all the notification history that’s been sent to the support people.

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.

Why did we build a web based BizTalk Admin Console?

Why did we build a web based BizTalk Admin Console?

web based BizTalk Admin Console

Back in 2010, the original idea of BizTalk360 was very simple — to create a web based version of the BizTalk Admin Console (BAC). However, over the years, the product has evolved into a full blown Operations, Monitoring and Analytics solution for Microsoft BizTalk Server; thanks to constant customer and community feedback.

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

History of BizTalk Server and Admin tools

If you look at the history of BizTalk Server, the first version was released in 2000 and then an updated version was released in 2002. Both these initial versions were very preliminary. They used Visio as the Orchestration designer, had the concept of channels, worker queues etc., and the standard MMC snap (like any other Microsoft Product).

In 2004, Microsoft released BizTalk Server 2004 — one of the major step forward in the history of BizTalk Server. The product was developed from scratch using .NET technologies since the previous two versions (BizTalk Server 2000 and BizTalk Server 2002) were built on top of COM/COM+ technology (most of the young generations now probably won’t know much about it).

BizTalk Server 2004 came with the integrated Orchestration and Map designer within Visual Studio and also introduced some major new concepts like Business Activity Monitoring, Enterprise Single Sign on, Business Activity Services, Business Rules Engine so on.  It also came with a management tool called “Health and Activity Tracker (HAT)”. HAT was the only tool available for BizTalk Administrators/Support people to run the day-to-day operations of your BizTalk Environment.

Health and Activity Tracker in BizTalk Server 2004

In BizTalk Server 2004, there was no concept of BizTalk Applications (to group/segregate things together).  You simply deploy your solutions containing schemas, maps, orchestrations etc., to the environment and soon the environment became too complex.

Two years later, Microsoft released BizTalk Server 2006. One of the major update to this edition of BizTalk Server was the new architecture when compared to BizTalk 2004. Microsoft addressed some of the shortcomings in the previous version of the product. One of the major functionality of BizTalk Server 2006 was the introduction of “BizTalk Admin Console” and the concept of Applications where you can group related artifacts like Schemas, Maps, Orchestrations, .NET dlls, batch files, Rules Policies etc., together. This version also introduced the ability to package applications into MSI’s for easy deployment.

BizTalk-Server-Admin-Console

Why did we build a web based BizTalk Admin Console?

After 2006, Microsoft constantly released new versions of BizTalk Server pretty much every 2 years once. BizTalk Server 2006 R2, BizTalk Server 2009 and BizTalk Server 2010. However, there was no major improvement that was made on the BizTalk Admin Console.

Being active in the BizTalk community, we constantly started to hear people’s frustration of both not modernizing the BizTalk Admin Console to a web based console and a lot of missing capabilities when it comes to Enterprise requirements like enhanced security, auditing, governance etc.

This is when we realized there is a clear need for an Enterprise Grade Web Based BizTalk Admin console to help and support the thousands of BizTalk Server Enterprise customers out there.

Having worked with Microsoft for nearly 2 decades, we pretty much knew how the company operates. We understood that Microsoft will not invest their valuable time in creating a web based management solution. Microsoft is always focused on creating a scalable/robust platform and it always embraces the partner ecosystem to build tooling around it. If you look at other popular products in Microsoft stack like SQL Server, IIS, Active Directory Manager, Windows Event Log etc, the default tooling will be pretty basic, none of them have a web based solution.

Birth of BizTalk360

In 2010, for all the above mentioned reasons, we took the big bet on building a complete web based BizTalk Admin console i.e the birth of BizTalk360. The below screen shot shows how BizTalk360 is simply a web based BizTalk Admin Console.

biztalk360-8-dashboard_thumb

Like any software product, BizTalk360 has evolved and matured in the past 6 years. Even though the original idea was to build just a web based BizTalk Admin Console, over the years, more and more features were added to the product and today BizTalk360 has become a single operations, monitoring and analytics solution for Microsoft BizTalk Server.

As a mid sized company (around 50 people), it’s super important for us to carefully select the features we wanted to build. We cannot afford to build features that no-one wants. In the series of blog articles over the new few weeks, we’ll reveal similar stories of why we built specific features and what’s the reasoning behind each one of them.

As you can see, if you are still using only the standard BizTalk Admin Console for your day-to-day operations, then you are at least 10 years behind in terms of tooling.

Download and Try BizTalk360 Free for 30 days today!

Download and try BizTalk360 on your own environments free for 30 days. Installation will not take more than 5-10 minutes.

BizTalk360-Free-Trial

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.