BizTalk: Using context for routing

[See Sample: Context routing and throttling with orchestration]

Imagine the project where most of the routing happens between orchestrations. I.e. routing is mostly between the MessageBox and orchestration with direct endpoints.
Imagine also the most of the messages are with the same Message type.
Usually in this case messages got the special node only for the routing. For example, the field can be the “Originator” or “Recipient” or “From” or “To”.
What wrong is with this approach, it creates the dependency between the message and the message processing. Message “knows” something about Originator or Recipient.
So what we can do with it? How can we “colorize” the same message to route it to the different places without changing the message itself?
One of the decisions is to use the message context.
BizTalk uses the promoted properties for routing.  There are two kinds of the properties: the content properties and the context properties. The content property extracts its value from inside the message, it is a value of the element or attribute. [See MSDN] The context property gets its value from the message environment. It can be the port name that receive this message, it can be the message Id, created by the BizTalk. Context properties look like the headers in the SOAP message. Actually they are not the headers but behave like headers. The context properties are the good match for our case.
First, we don’t have to change the message itself to set or change the routing property. The context is stored outside the message body.
Second, we don’t have to create the property schema to use the context properties. [See MSDN: How to create Property schema] BizTalk has the predefined schema set for the context properties. [See MSDN: Message Context Properties] Use one of them and that’s it.
The main purpose of the context properties is working on behalf of the BizTalk internals. But we can read, create and change them. Just do not interfere with BizTalk internals on this way.

 

BizTalk: Sample: Context routing and Throttling with orchestration

BizTalk: Sample: Context routing and Throttling with orchestration

The sample demonstrates using orchestration for throttling and using context routing.
Usually throttling is implemented on the host level (in BizTalk 2010 we can also using the host instance level throttling). Here is demonstrated the throttling with orchestration convoy that slows down message flow from some customers. Sample implements sort of quality service agreement layer for different kind of customers.
The sample demonstrates the context routing between orchestrations. It has several advantages over the content routing. For example, we don’t have to create the property schema and promote properties on the schemas; we don’t have to change the message content to change routing.
Use case:
 The BizTalk application has a main processing orchestration that process all input messages. The application usually works as an OLTP application. Input messages came in random order without peaks, typical scenario for the on-line users.
But sometimes the big data batch payloads come. These batches overload processing orchestrations. All processes, activated by on-line users after the payload, come to the same queue and are processed only after the payload. Result is on-line users can see significant delay in processing. It can be minutes or hours, depending of the batch size.
Requirements:
On-line user’s processing should work without delays. Big batches cannot disturb on-line users. There should be higher priority for the on-line users and the lower priority for the batches.
Design:
Decision is to divide the message flow in two branches, one for on-line users and second for batches. Branch with batches provides messages to the processing line with low priority, and the on-line user’s branch – with high priority.
All messages are provided by hi-speed receive port.
BTS.ReceivePortName context property is used for routing. The Router orchestration separates messages sent from on-line users and from the batch messages.
But the Router does not use the BizTalk provided value of this property, the Router set up this value by itself. Router uses the content of the messages to decide if it is from on-line users or from batches.
The message context property the BTS.ReceivePortName is changed respectively, its value works as a recipient address, as the “To” address for the next recipient orchestrations.
Those next orchestrations are the BatchBottleneck and the MainProcess orchestrations.
Messages with context equal “ToBatch” are filtered up by the BatchBottleneck orchestration. It is a unified convoy orchestration and it throttles the message flow, delaying the message delivery to the MainProcess orchestration.
The BatchBottleneck orchestration changes the message context to the “ToProcess” and sends messages one after another with small delay in between. Delay can be configured in the BizTalk config file as:
                <appSettings>
                                <add key=”GLD_Tests_TwoWayRouting_BatchBottleneck_DelayMillisec” value=”100″/>
                </appSettings>
 
Of course, messages with context equal “ToProcess” are filtered up by the MainProcess orchestration.
 

NOTES:
  • Filters with string values: In Orchestrations (the first Receive shape in orchestration) use string values WITH quotes; in Send Ports use string values WITHOUT quotes.
  • Filters on the Send Ports are dynamic; we can change them in run-time. Filters on the Orchestrations are static; we can change them only in design-time.
  • To check the existence of the promoted property inside orchestration use the Expression shape with construction like this:
          if (BTS.ReceivePortName exists myMessage) { …; }
    It is not possible in the Message Assignment shape because using the “if” statement inside Message Assignment is prohibited.
  • Several predefined context properties can behave in specific way. Say MessageTracking.OriginatingMessage or XMLNORM.DocumentSpecName, they are required some internal rules should be applied to the format or usage of this properties. MessageTracking.* parameters require you have to use tracking and you can get unexpected run-time errors in some cases. My recommendation is – use very limited set of the predefined context properties.
  • To “attach” the new promoted property to the message, we have to use correlation. The correlation type should include this property. [Here is a good explanation by Saravana ]
The sample code is here [sorry, temporary trubles with CodePlex].

 

Biztalk 2010: Light and Easy Series is Back

Biztalk 2010: Light and Easy Series is Back

Hey folks, Alan Smith and myself (much more Alan this time 🙂 have brought the series back for another version and another year!
Myself and many other Biztalk MVPs and some great BizTalk-ers with fantastic real world experience share their knowledge.

As always the webcast series is designed to be an easy watch, level 200 ish and I’ve even had some folks reporting they watched the needy webcast on the way to clients to talk about for e.g. an EDI solution.

BizTalk 2010 Light And Easy Series – here’s where you’ll find the series http://www.cloudcasts.net/Default.aspx?category=BizTalk+Light+and+Easy
(Alan is in the process of uploading them)

  integration with BizTalk 2010 using the BizTalk WSS Adapter.

I created a 2 part episode – the first one deals with explaining the SharePoint 2010 environment, and the 2nd one deals with integration from BizTalk.

Here’s my recordings, PPTs and sample files.

Title WebCast PPT
Integration to SharePoint Part 1  Part 1 (110MB)  (2MB)
Integration to SharePoint Part 2  Part 2 (60MB)  (2MB)
Sample Code

Source Files

 
     

 

Enjoy!

Take advantage of new Windows Event Viewer (Vista, 7, 2008) capabilities for BizTalk Monitoring

Ideally we should have proper monitoring solution in place for any BizTalk projects. But the reality is always different, even in big enterprises monitoring solutions are not adequate due to various internal political reasons.

In this article I’m going to show how you can take advantage of the new Event Viewer functionalities that comes as part of Windows to get some basic monitoring. The new event viewer that comes as part of Windows Vista, 7 and 2008 are fully reengineered with the event tracking and logging architecture. Some of the core enhancements include

  • Ability to create custom queries so you can essentially tweak Event Viewer to show you just the things you are interested in (see Figure below)
  • You can associate a task to particular events, example: send you an e-mail, run a program, and display a message.
  • You can collect events from multiple systems and forward it to one system, allowing you to centralize event logs (thanks to WS-Management implementation underneath).
  • Now event viewer reports its data in XML.
  • You can import and export custom view
  • Above all the main difference is all the changes you make are persistent; it won?t disappear when you close the event viewer.

Categorise Events using Custom Views:

One of the challenges of current event viewer is the volume of information you receive on the default Application log. There were few filtering options, like you can create a New Log and tweak some parameters to get to the events you are interested in. But that?s very basic and one of the biggest issue was the filter setup was not persisted and you lose it once you close the event viewer.

But the new EV allows you to create various custom views and categories them in a nice folder structure. As shown in the above figure, I’ve created a typical structure for BizTalk monitoring separating events relevant to SSO, BAM, Orchestration, Core Messaging etc

The below figure shows the Create Custom View window, which allows you create a sophisticated filter and save it in a desired folder.

This way you can put critical and high severity error events in a separate custom view and other less important information messages in a separate view. Later in the article, we?ll see how you can attach tasks/alerts to the views and get automatic notification.

The other important aspect here is, you don?t need to use the UI to create the desired view, you can create define them as xml and import it using the event viewer console. The xml looks as shown below, utilizing XPATH statements for filtering. The below xml snippet shows the same thing we configured earlier, event ids between 5640 and 5650.

Note: Creating folder structure is not very neat; you can only create a new folder when you are saving or importing the custom view.

Create a range of Event Id for your application and monitor it

The “Create custom view” window allows you to mix and match various filter options to get to what you want. One of the powerful options is specifying the event id in various combinations

  1. Comma separated values
  2. Specify a negative sign to exclude them, and
  3. Specify a range as shown below

Specifying a range will be a very useful option if you have designed your custom solution in a way it emits a range of event ids.

Create actions for critical events

One of the crucial advantages of monitoring systems like SCOM and HP Open view is their ability to send alerts based on the event. Event viewer’s functionality may not be like for like match with commercial products, but it definitely provides you the basic alerting functionality in conjunction with the “Task Scheduler” that comes with Windows. Both the products are nicely integrated, as soon as you create a task in Event Viewer; it automatically creates a scheduled task in the background. You can change the properties later if you wish.

Three different types of task can be configured, you can start any program, you can send an email and you can display a message in the console.

Start a program is a powerful option, which allows you to start any application. For example you could have scripter certain activities and configured it run every time a particular event occurs.

Send an email, is other useful function, you can configure to send an email whenever there is an event within a particular custom view. The functionality itself is very basic,

Example: It doesn’t attach the original event message, there is no way to consolidate event etc. But you can structure the custom views in a way, so that only high severity incidents send notification.

Creating the task in event viewer quietly creates the task in Windows Task Scheduler as shown in the below picture

Once you have created the task, you can modify it later either in the Task Scheduler or Event Viewer.

The email notifications:

Forward events from all BizTalk/SQL (within a group) servers to central place:

The new event viewer got the ability to forward or subscribe events between computers. This is a very powerful for distributed applications that are built on top of BizTalk. A typical BizTalk environment will have one or more host servers and one or more SQL servers acting as a group. Setting up a central event collection for monitoring will be very useful. You can extend this event forwarding to all the dependant applications like your web services, front ends, etc to consolidate all the events in a central location.

The new event viewers completely redesigned event tracing and logging architecture based on WS-management brings this capability. There are set of command line event viewer/ WS-RM (remote management) utility tools that enables your to set up event forwarding. There are tons of resources out there explaining how you can setup this, so I?m not going to get into details in this article.

Example: By executing the following command in a server will allow remote access to that machine for management.

winrm quickconfig

The Windows Remote Management Tool will respond like so:

WinRM is not set up to allow remote access to this machine for management.
The following changes must be made:
 
Set the WinRM service type to delayed auto start.
Start the WinRM service.
Create a WinRM listener on HTTP://* to accept WS-Man requests to any IP on this
machine.
Enable the WinRM firewall exception.
 
Make these changes [y/n]?

Summary:

Even though the concept explained here are not ideal for an enterprise solution, it gives some basic monitoring capabilities for free. There is no cost involved here, as long as you are running your solution on the latest Windows OS, you get these functionalities for free. Ideally you should have invested in a proper monitoring solution like SCOM or HP Openview, depending on the importance of your BizTalk solution.

Nandri

Saravana

BizTalk 2010: New Book on the Horizon

BizTalk 2010: New Book on the Horizon

Over the past months I’ve been reviewing a new BizTalk 2010 book – BizTalk 2010: Integrating Line of Business Systems

There’s a high caliber line up of Author’s all busily sharing their knowledge.

Kent’s got all the details here – http://kentweare.blogspot.com/2011/02/new-biztalk-2010-book-unveiled-line-of.html

Looking forward to when it hits the shelves.

Well done guys – looking great from what I’m reading 🙂

1902en_mockupcover_normal_0

SharePoint 2010: Getting Started – Navigation and Sharing

Hi folks, as promised here’s my little slide deck for the presentation given last
week. We had a great audience with some very interesting questions.

Thanks all that attended as part of the Microsoft Partner Readiness program.
Hope you’re enjoying and feel free to give me feedback as to what you’d like to see
more of and less of.

Have fun,

Mick.

Slides in ZIP PPT
Slide Deck

Slides and Code from NYC (Winter) Code Camp 2011-(How to) Put your Code First with Entity Framework

I would like to thank everyone for attending my presentation this Saturday at the NYC (Winter) Code Camp! I really enjoyed the high level of interest and participation from the attendees. Unfortunately, we only had 75 minutes to talk about Entity Framework and EF Code First but thankfully we have the internet!
Thanks to all […]