Custom Pipeline Component for High Performance Message Transformation

by | Oct 13, 2010 | BizTalk Community Blogs via Syndication

When developing pure messaging solutions (no orchestrations) in BizTalk often there’s a need to apply transformation on the message. BizTalk has out of the box functionality to execute maps on receive (inbound map) and send (outbound map) ports. Sometimes using classic BizTalk map is not the best solution, for example for complex transformations with grouping (for example, using Muenchian Method). In other cases, it’s strongly preferable to keep transformation easily configurable to be able to change it on the fly without overhead of deploying new map. Transformations using XSL stylesheet seems to fit that task very well.
One problem though, using standard .Net XML/XSL API won’t yield the best performance. Performance degradation will be especially noticeable in high throughput scenarious with large message size. BizTalk internally uses more efficient transformation API that is tuned for better performance for large messages. The classes that make this possible are VirtualStream (located in microsoft.biztalk.streaming.dll) and BTSXslTransform (located in Microsoft.xlangs.basetypes.dll). VirtualStream minimizes memory footprint by backing storage to file system.
Utilizing these classes I created generic pipeline component that can execute XSL transformation inside pipeline in a scalable fashion. As an additional feature it can be configured to execute standard BizTalk map.
Following configuration properties are available on the component:

Property
Values
Description
IsScalable
True | False
Indicates if transform should always be scalable
MapName
{XslUrl} | {BizTalk Map Name}
Defines the XSL URL or fully qualified map name to use for transformation
MessageType
{namespace}#{root}
Target message type
SchemaStrongName
{Class}, {Assembly}, Version={}, Culture={}, PublicKeyToken={}
Fully qualified .Net schema type name
TransformUsing
XslStylesheet | BizTalkMap
Indicates what kind of transformation is used
Source code with example project is available at http://www.box.net/shared/93p7n1vsl6

Book Sample Code, Our Commercial, and Repair & Resubmit Solution Video

by | Oct 13, 2010 | Stephen's BizTalk and Integration Blog

I have posted online all the sample code from my book, “Applied Architecture Patterns on the Microsoft Platform”.  You can download it at http://www.biztalkgurus.com/media/p/30260.aspx.  This code contains various samples using BizTalk, BizTalk ESB Toolkit, Server AppFabric, SharePoint 2010, .Net 4.0 Workflow, SQL, Stream Insight, Azure, SQL Azure, Platform AppFabric Service Bus, Server AppFabric Caching, and more.  Of course, you will get the most benefit of this code using it alongside with the chapter of the book that walks through it. 

The book is available now on Amazon.com and PacktPub.com. 

Thanks to our friends in Sweden we now have a Commercial for our book!  I have not seen many commercials for Technology books, but this video gives some context of the “why” behind our book.  You get to hear from Richard, Ewan, and myself about why we wrote the book and why we picked to release the book at the European BizTalk Conference in Stockholm, Sweden to a room full of top notch, hard-core BizTalk people.

This video is available at http://www.biztalkgurus.com/media/p/30247.aspx and on YouTube. 

 

If you are looking for more detailed information on the types of solutions covered in the book, I have put together a video walking through Chapter 10 – Repair and Resubmit with Human Workflow.  This solution uses SharePoint 2010 to store customer information.  This information is processed by a .Net 3.5 workflow that sends the data to a Server AppFabric hosted .Net 4.0 workflow service for processing. 

This video is available at http://www.biztalkgurus.com/media/p/30249.aspx and on YouTube. 

Windows Azure AppFabric Labs October release – announcement and scheduled maintenance

by | Oct 12, 2010 | BizTalk Community Blogs via Syndication

The next update to the Windows Azure AppFabric LABS environment is scheduled for October 26, 2010 (Tuesday).  Users will have NO access to the AppFabric LABS portal and services during the scheduled maintenance down time.    

When:

    START: October 26, 2010, 8 am PST

    END:  October 28, 2010, 11 am PST

 Impact Alert:

  The AppFabric LABS environment (Service Bus, Access Control and portal) will be unavailable during this period. Additional impacts are described below.

Action Required:

Existing accounts and Service Namespaces will be available after the services are deployed.

You will have an additional -mgmt entry as follows:

Your ns.sb.AppFabricLabs.com will remain ns.sb.AppFabricLabs.com and an additional entry and corresponding relying party will be created as ns-mgmt.sb.AppFabricLabs.com

However, ACS Identity Providers, Rule Groups, Relying Parties, Service Keys, and Service Identities will NOT be persisted and restored after the maintenance. As the service will not support the bulk upload of rules from the previous version, the user will be responsible for both backing up and restoring any rules they care to reuse after the Windows Azure AppFabric LABS October Release.

Thank you for working in LABS and giving us valuable feedback.  Once the update becomes available, we’ll post the details via this blog. 

Stay tuned for the upcoming LABS release! 

The Windows Azure AppFabric Team

BizTalk 2010: Musing of the ’new’ SharePoint 2010 WS Adapter

by | Oct 11, 2010 | BizTalk Community Blogs via Syndication

Today I decided to crack open the BTS 2010 SharePoint WS Adapter to see if it takes
advantage of the great new interfaces exposed by SharePoint 2010, specifically Microsoft.SharePoint.Client.dll
and Microsoft.SharePoint.Client.Runtime.dll.

At a glance, the benefits of this new Client APIs are:

  1. Runs on a non SharePoint installed box.
  2. Lightweight and flexible – only get back what you ask for. As opposed to the classic
    SP Server API that populates the SPWeb collection (for e.g.) only if you just want
    the title field and not 10MBs worth of other data.
  3. Batch approach – load up several commands and batch them over the wire when needed.
  4. Supports both read/write from the client back to SP Server.
  5. Uses XML and JSON over the wire – small and fast.
  6. We can’t do *everything* we can on the Server Side – e.g. Service Application management,
    i.e. kicking off a search index crawl.

A little piccy of what’s going on:

Some classic piece of code to achieve document library reading: 


 1 static void Main(string[]
args)  2  {  3  ClientContext
ctx = new ClientContext("http://intranet");  4  Web
web = ctx.Web;  5  List
docs = web.Lists.GetByTitle("Shared
Documents");  6  ListItemCollection
items = docs.GetItems(CamlQuery.CreateAllItemsQuery());  7  ctx.Load<Web>(web);  8  ctx.Load(docs);  9  ctx.Load(items); 10  ctx.ExecuteQuery(); 11  Console.WriteLine("The
list has {0} items.",
docs.ItemCount); 12  foreach (ListItem
item in items) 13  { 14  Console.WriteLine("Item:{0}",
item["Title"]); 15  } 16  //delete
an item. 17  //items[1].Update(); 18  //items[1].DeleteObject(); 19  //ctx.Load(items); 20  //ctx.ExecuteQuery(); 21  Console.ReadLine(); 22  }

Note: Line 10 is where all the magic happens – if you imagine, we
load up the client OM classes and the props etc. are all ’blank’ until we do an ExecuteQuery() which
then populates what we ask for.

The above sample is pretty simple showing how to connect to a document library on
a ’remote’ server (security allowing – I didn’t add a ctx.Credentials= line in the
above, but all possible).

So let’s move on a crack open the BTS 2010 SharePoint WS Adapter

Just before we go there I’d like to point out that the Microsoft.SharePoint.dll (aka
Server API) has the ability to connect to remote servers, although the code needs
to be executed on a machine that has a local SharePoint install.

e.g.

SPSite site = new SPSite(“http://remoteserver.acme.com”);

SPWeb web = site.OpenWeb();

What I am trying to avoid with the BTS SharePoint adapter is the need to have the
’BTS Web Service’ component installed on remote Farms. Just complicates the issue
far too much with the SharePoint admins.

The BTS 2010 Story

I setup and installed the BTS SharePoint WS Adapter through the Configuration.exe tool
successfully.

Essentially this tools runs a ’web site check’ to make sure SharePoint is successfully
setup and installed.

To make this happen, the configuration tool runs either:

  1. Microsoft.BizTalk.KwTpm.StsOmInterop3.exe – for WSSv3
  2. Microsoft.BizTalk.KwTpm.StsOmInterop4.exe – for WSSv4

to determine the site as follows:

Note: The URL and note the URL in the BTS Configuration above. Here
I’ve already configured the adapter and I’m just showing the commands that the configurator
runs behind the scenes.

Once configuration is complete you will see a new virtual directory added 
to your selected site e.g. http://intranet.

As shown in IIS Manager.

Depending on the SharePoint version this virtual directory will map to:

  1. C:\Program Files (x86)\Microsoft BizTalk Server 2010\Business Activity Services\BTSharePointV4AdapterWS
    or
  2. C:\Program Files (x86)\Microsoft BizTalk Server 2010\Business Activity Services\BTSharePointV3AdapterWS
    (previous bts2009 adapter)

A Basic BTS/SharePoint picture

Essentially the BTS SharePoint Adapter consists of 2 parts:

  1. A BTS Adapter that talks to the BTS SharePoint WS. This is a ’classic’ adapter and
    does not talk the newer WCF framework (which does have advantages
    and disadvantages)
  2. A BTS SharePoint WS – this does all the work against the SharePoint library and talks
    local SharePoint APIs.

 

Let’s look closer at the BTSharePointV4AdapterWS folder

– this folder, or addition needs to be available locally to whichever SharePoint site
you are calling through the OOTB BTS SharePoint adapter, even though the SharePoint
APIs support remote Servers.

– the bin folder has the Microsoft.BizTalk.KwTpm.WssV4Adapter.WebService.dll
which is 78kb.

I wanted to find out whether this DLL used the new SharePoint Client APIs when meant
having a peek at the ’references’ of this DLL in IL.

Dissassembling Microsoft.BizTalk.KwTpm.WssV4Adapter.WebService.dll

Using .NET Reflector I was able to get this picture

NOTE: on this list there is Microsoft.SharePoint, but not Microsoft.SharePoint.Client.dll
(this is not looking goodcould be late bound, but I doubt it)

Digging into the actual WssAdapter class we get the following of
note:

The GetDocuments(string, string, string, Int32, DocExtOfficeIntegration) is
a key method.

The APIs show that the 1st parameter is a siteUrl (and following
the implementation code through) which has the potential to point to another SharePoint
server to make the connection (in the RequestInfo class if you’re going to dig yourself
:))
Note: the PREVIOUS version, BTS2009 has the same Interface/Method signature
and it requires the BTS SharePoint Adapter WS to be deployed on the remote SharePoint
Server, even though the signature looks as though it will support the remote
server.

So in conclusion the BTS SharePoint Adapter WebService has:

  1. NOT got any newer SharePoint Client API code within in.
  2. The ability to contact a remote server through the WebService APIs.
  3. But depends on whether the BTS Adapter will pass the ’remote’ URL to the ’local’ WS,
    or will the Adapter try to contact the remote SharePoint Server directly looking for
    a WS there???

I’m thinking it’s the latter

A little more to unravel the SharePoint mystery