CRM BizTalk Integration using Azure Service Bus

You may face the same challenges that I experienced when integrating Microsoft Dynamics CRM and BizTalk. The objective is to capture events in real-time and transmit them to an outside system without losing the order of events. There are two web methods exposed by CRM web services, “Retrieve” and “RetrieveMultiple”. The CRM web services provide […]
Blog Post by: Rakesh Gunaseelan

White Paper : “The A-Y of running BizTalk Server in Microsoft Azure” now available for download from BizTalk 360 White Paper collection

White Paper : “The A-Y of running BizTalk Server in Microsoft Azure” now available for download from BizTalk 360 White Paper collection

I’m pleased to announce that the White Paper titled “The A-Y of running BizTalk Server in Microsoft Azure” which I have been working on for the last two months is now available to download on the BizTalk 360 White Paper collection. Writing this White Paper has been a momentous task, especially given that Microsoft Azure […]
Blog Post by: Johann

Capturing Custom Logs from Azure Worker Roles using Azure Diagnostics

Capturing Custom Logs from Azure Worker Roles using Azure Diagnostics

In this post I’ll show you how to correctly configure diagnostics in an Azure Worker Role to push custom log files (NLog, Log4Net etc.) to Azure Storage using the in-built Azure Diagnostics Agent.   Configuring our Custom Logger – NLog I’m not a massive fan of the recommended Azure Worker Role logging process, namely using […]
Blog Post by: modhul

How to retrieve R data visualization from Azure Machine Learning

As you may know, Azure Machine Learning can execute R scripts. You can interactively see the output console. But what about retrieving the result as part of a production call to the API generated by Azure ML?

Let’s test with a word cloud example in R. Mollie Taylor has posted one here (https://gist.github.com/mollietaylor/3671518) that we can reuse in Azure Machine Learning:

The details on how to create an Azure ML workspace, insert a dataset and an R script can be found here:

  • http://azure.microsoft.com/en-us/documentation/articles/machine-learning-walkthrough-develop-predictive-solution/

for R, just use that module:

 

The input of the Web API is set to the input dataset of the R Script and the output is set to the R Device port. As a reminder, here is how the inputs and outputs are positioned in an R Script module:

the detail is available in the help documentation.

In our case the interesting ports to publish are the following:

and

 

After running the experiment, we can see the result in Azure ML Studio:

So, how could we retrieve the pictures from an API that is published that way:

 

Here is some sample script in Python that shows how to do it. The script is a modified version of the sample given in the API Help page for Batch Execution. The idea is to get the base64 encoded pictures from the output file and decode them out to local disk.

# -*- coding: utf-8 -*-

# How this works:
#
# 1. Assume the input is present in a local file
# 2. Upload the file to an Azure blob - you'd need an Azure storage account
# 3. Call BES to process the data in the blob. 
# 4. The results get written to another Azure blob.
# 5. Download the output blob to a local file
#
# Note: You may need to download/install the Azure SDK for Python.
# See: http://azure.microsoft.com/en-us/documentation/articles/python-how-to-install/

import urllib2
import json
import time
from azure.storage import *
import sys
import base64
import json

storage_account_name = 'a****obfuscated***4'
storage_account_key = '/aV****obfuscated***vXA76w=='
storage_container_name = 'benjguin'

input_file = ur"C:\be****obfuscated***os\WordCloud\conventions.csv"
output_file = ur'C:\be****obfuscated***os\WordCloud\myresults.csv'
input_blob_name = 'conventions.csv'
api_key = r'Cczx****obfuscated***WemQ=='
url = 'https://ussouthcentral.services.azureml.net/workspaces/a7c****obfuscated***756/services/d328e03****obfuscated***5c2/jobs'
uploadfile=True
executeBES=True

blob_service = BlobService(account_name=storage_account_name, account_key=storage_account_key)

if uploadfile:
    print("Uploading the input to blob storage...")
    data_to_upload = open(input_file, 'r').read()
    blob_service.put_blob(storage_container_name, input_blob_name, data_to_upload, x_ms_blob_type='BlockBlob')

input_blob_path = '/' + storage_container_name + '/' + input_blob_name
debug_blob = blob_service.get_blob(storage_container_name, input_blob_name)

if executeBES:
    print("Submitting the BES job...")
    connection_string = "DefaultEndpointsProtocol=https;AccountName=" + storage_account_name + ";AccountKey=" + storage_account_key
    payload =  {
                "Input": {
                    "ConnectionString": connection_string,
                    "RelativeLocation": input_blob_path
                    }
                }

    body = str.encode(json.dumps(payload))
    headers = { 'Content-Type':'application/json', 'Authorization':('Bearer ' + api_key)}
    req = urllib2.Request(url, body, headers) 
    response = urllib2.urlopen(req)
    result = response.read()
    job_id = result[1:-1] # remove the enclosing double-quotes

    url2 = url + '/' + job_id

    while True:
        time.sleep(1) # wait a second
        authHeader = { 'Authorization':('Bearer ' + api_key)}
        request = urllib2.Request(url2, headers=authHeader)
        response = urllib2.urlopen(request)
        result = json.loads(response.read())
        status = result['StatusCode']
        if (status == 0):
            print("Not started...")
        elif (status == 1):
            print("Running...")
        elif (status == 2):
            print("Failed...")
            break
        elif (status == 3):
            print("Cancelled...")
            break
        elif (status == 4):
            print("Finished!")
            result_blob_location = result['Result']
            sas_token = result_blob_location['SasBlobToken']
            base_url = result_blob_location['BaseLocation']
            relative_url = result_blob_location['RelativeLocation']
            url3 = base_url + relative_url + sas_token
            response = urllib2.urlopen(url3)
            with open(output_file, 'w') as f:
                f.write(response.read())
            break

outputdata=open(output_file)
outputtxt=outputdata.read()
outputdata.close()

s=outputtxt.index('\"{')
e=len(outputtxt)
o1=outputtxt[s+1:e-3]

jsonresult = json.loads(o1)
i=1
for gd in jsonresult['Graphics Device']:
    fname = output_file + "." + str(i) + ".png"
    print 'writing png #' + str(i) + ' to ' + fname
    f = open(fname, 'wb')
    f.write(base64.b64decode(gd))
    f.close()
    i += 1

print("Done!")

Here is a sample execution output:

Uploading the input to blob storage...
Submitting the BES job...
Running...
Running...
Running...
Running...
Running...
Running...
Running...
Finished!
writing png #1 to C:\be***obfuscated***os\WordCloud\myresults.csv.1.png
writing png #2 to C:\be***obfuscated***os\WordCloud\myresults.csv.2.png
Done!

The output sent back by Azure ML looks like this:

R Output JSON

"{"Standard Output":"RWorker pushed \"port1\" to R workspace.\r\nBeginning R Execute Script\n\n[1] 56000\r\nLoading objects:\r\n  port1\r\n[1] \"Loading variable port1...\"\r\npng \r\n  2 \r\nnull device \r\n          1 \r\n","Standard Error":"R reported no errors.","visualizationType":"rOutput","Graphics Device":["iVBORw0K***(...)***RvX/wFzB5s8eym6ZgAAAABJRU5ErkJggg==","iVBORw0KGgo***(...)***dVorBuiQAAAABJRU5ErkJggg=="]}"

You can see the pictures

 

well, Python does:

The resulting files are:

 

and

 

R has tons of great data visualisation. Have a look at those blogs for instance:

 

 

Benjamin (@benjguin)

Blog Post by: Benjamin GUINEBERTIERE

BizTalk: BizTalk Flow Visualizer

Today I came across something I was keeping an eye on recently and that is the BizTalk
Flow Visualiser
on Codeplex.

It basically looks at your tracking data and gives you back call details.

– Errors/Succes

– # of times something is called

– amount of time taken to process each stage.

It’s pretty easy to setup, there’s a site and a serviceand the rest you can guess.

http://biztalkflowvisualizer.codeplex.com/

Check it out and feell free to contribute!!

Enjoy!!!

Blog Post by: Mick Badran

BizTalk360 – An easy way to evaluate

BizTalk360 is a full featured BizTalk Server Monitoring & Operations product. The product has matured from a small web based monitoring tool to a full blown feature rich product for monitoring BizTalk Server. The product includes features for BizTalk Operation support people like the Throttling analyzer, Topology Viewer, and Backup DR Monitor. Recently a new version 7.8 has been released with

BizTalk360 – An easy way to evaluate

BizTalk360 – An easy way to evaluate

BizTalk360 is a full featured BizTalk Server Monitoring & Operations product. The product has matured from a small web based monitoring tool to a full blown feature rich product for monitoring BizTalk Server. The product includes features for BizTalk Operation support people like the Throttling analyzer, Topology Viewer, and Backup DR Monitor. Recently a new version 7.8 has been released with another set of valuable features:

  • Advanced Process Monitoring and dashboard
  • EDI reporting capabilities in line with BizTalk Admin Console.
  • System Profile: Ability to remove features you don’t want in BizTalk360.
  • User Profile: Flexibility to set custom time zone and date/time format for each user.
  • Removed Orphaned artifacts from Monitoring: Ability to remove old orphaned artifacts like Applications, which no longer exist in the environment.

If you haven’t heard of BizTalk360 than you will like this blog post as it will explain how you can easily evaluate the tool by yourself and witness the great value this product can offer. Now this will be not an extensive post to showcase each feature. I will focus on how you can leverage the features: Graphical Flow (Tracking), Business Activity Monitoring (Portal), and the Tracking Manager.

Easy way to see these features in action is by deploying the BamEndToEnd example you can find in the <Samples Path>BamEndToEndInput folder of the installation of your BizTalk Server. This sample stems from version 2004 and can be used from 2006 up to 2013 R2. The demonstrations in this post is the BamEndToEnd example with BizTalk Server 2013 R2, and the latest BizTalk360 Version 7.8.

After deploying the BamEndToEnd example you will need to create the tracking profile yourself and deploy it. Next you will to add/verify the tracking is enable on the send, receive ports and the orchestrations.

Note that tracking can be very useful for troubleshooting and traceability in general. However in production environments you beware that extensive tracking can influence performance. For best practices for tracking see blog post on BizTalkAdminsBlogging Best Practices for Tracking. With regards to orchestration tracking is that Event Tracking is enabled by default. This can be helpful during development, testing and troubleshooting with the Orchestration Debugger. In Production, these tracking events should be disabled, because these events are written to the TrackingData_x_x tables in the BizTalkMsgBoxDb database and then moved to the dta_DebugTrace table in the BizTalkDTADb database by the Tracking host. If the TrackingData_x_x tables (BizTalkMsgBoxDb database) and/or the dta_DebugTrace table (BizTalkDTADb database) get too large*, performance will be impacted, see How to maintain and troubleshoot BizTalk Server databases.

After deployment and configuration is done you can drop the test messages in the folder belonging to the receive location. In SQL Studio Manager you can execute the query below to examine if data is present in the BAMPrimairyImport database tables belonging to the BamEndToEndActivity.

When that is in order you can switch to BizTalk360 product. The product is available for free trail. The installation is straightforward described on the site on this page. Once you have installed and configured BizTalk360 you can start looking at some of the features. One is the Graphical Flow (Viewer). You can select this viewer and leave everything default and click Execute Query.

Next you can select one of the tracked service instances.

On right hand side you can see the properties of the third orchestration of the BAMEndToEnd example. You can click the blue dot (where the message is received) and see a line going from the green dot (message being send from a location) of the second orchestration. Now you can click in the right hand pane the context and/or content tab. You will see the context of the message going from Orchestration 2 to 3 and the content.

clip_image010

You can click the blue dot again and reach orchestration 1 the first orchestration message are flowing through. This feature is a very valuable feature in case you need to troubleshoot a process that for instance consists of multiple orchestrations. From a messaging perspective you can look at messages coming in and out of BizTalk. In the dashboard you can select Tracked Message Events instead of Tracked Service Instances. Select one of the Send Type the eye icon.

A dialog will appear for the message you selected. You can see the general properties, context properties and the content. Note this depends on how tracking is configured. For both Tracked Message Events and Tracked Service Instances you add filters that can enhance search ability for a certain message(s) and instance(s).

Next we will look at the Business Activity Monitoring. BizTalk360 offers a view/dashboard for BAM that is similar to the BAM Portal. One of the concepts of BizTalk360 is offering a single operation and monitoring experience. This means one product for accessing data and user experience (UI) for BizTalk, BAM, BRE, EventViewer, BizTalk databases and ESB Toolkit.

If you go to Business Activity Monitoring Dashboard in BizTalk360 you can do similar things one can do in the BAM Portal. In The BAM Dashboard you can select a BAM View. In this case the BAMEndToEnd View. Select columns and execute a query.

As you can see the data can be viewed from the dashboard. There is no need to go to a separate Portal in case your BizTalk solutions include BAM. The BAM Dashboard in BizTalk360 will also give you a view of the sizes of the BizTalk BAM Databases.

Sizes of your BAM databases matter. In case BizTalk system is under a huge load a significant amount will be persisted to the BAM database. Therefor information on size and growth are valuable. You can monitor this by setting alarms in BizTalk360.

The final feature I like to discuss in this context is the Tracking manager. For the BAMEndToEnd example you can enable tracking on send/receive ports and the orchestrations. By selecting the Tracking Manager in BizTalk360 you can examine how tracking is enable on the receive ports, send ports and orchestrations.

You can select different applications in this dashboard, examine if global tracking is enabled and how per application the tracking is set on receive/send ports and orchestrations. This manager or dashboard if you like gives you a quick view on how tracking is set up in your BizTalk environment. This can be useful in cases you have not direct insight of the bindings of the applications that are being deployed. The bindings can contain tracking settings, see for instance binding collection of a receive port on MSDN.

In this post I quickly showed three powerful features in BizTalk360. However, there are many more which you can examine by yourself or by reviewing the feature tour on the BizTalk360 site. In case you like to further explore the features BizTalk360 offers you can do so by using the BizTalk out-of-the box examples on a VM and the evaluation edition of the product.

Cheers,

Steef-Jan

BizTalk Mapper Extensions UtilityPack for BizTalk Server 2013 Updated to v1.7.0.1

BizTalk Mapper Extensions UtilityPack for BizTalk Server 2013 Updated to v1.7.0.1

Exciting news Version 1.7.0.1 of BizTalk Mapper Extensions UtilityPack for BizTalk Server 2013 is now available! Project Description BizTalk Mapper Extensions UtilityPack is a set of libraries with several useful Functoids to include and use it in a map, which will provide an extension of BizTalk Mapper capabilities. Here’s the change-log for this release: Fixes […]
Blog Post by: Sandro Pereira

BizTalk360 version 7.8 released

BizTalk360 version 7.8 released

BizTalk360 just released their latest version 7.8 with interesting new features. With this new release comes some very interesting new features that make it even easier to actively monitor and maintain your BizTalk environment healthy. Be sure to check out this latest release and download the setup or request for a free trial version. Some […]
Blog Post by: mitchke

Creating a Windows 10 Preview VHD That Can Run as a Hyper-V VM and Can Boot Natively

Introduction
When the Windows 10 Technical Preview came out earlier this month, I wanted to see kick the tires a bit and see what was new. However, I need my laptop to work reliably, so I couldn’t take the risk of installing Windows 10 over my Windows 8.1 installation.
So, I decided to install it to a […]
Blog Post by: jgiessner