Introducing QuickCounters…

Have you ever wanted to get real-time metrics on:  How many times has a particular
orchestration executed on this node in the BizTalk group?  What does the average
execution time for my pipeline component looking like?  How many instances of
a particular orchestration are currently executing across the group, not including
dehydrated instances?

Questions like this can be tough to answer – the built-in BizTalk counters are too
coarse-grained, and Tracking & BAM are only partially useful here (particularly
if you want frequent samples.)  What you would like is the abilty to get fine-grained
“request metrics” on all of your BizTalk orchestrations and pipelines in a way that
is simple to implement and easy to consume.

I’ve been leading an open source project called QuickCounters for
quite some time now – though we’ve just recently moved into CodePlex.  It is
a library that will be of interest to BizTalk developers – but it can be used in much
broader settings.  I had a chance to present on this topic at the Twin
Cities .NET Users Group (on 11/2/2006 – presentation here.)

QuickCounters is a .NET library which makes it extremely easy to implement the common,
request-level performance counters that are interesting in just about any service
you might write.  You can use this library to instrument  general .Net components,
web services, BizTalk orchestrations, pipeline components, remoting interfaces, enterprise
service components…you get the idea.  There is special support in the library
for BizTalk and WCF scenarios, but it is quite easy to use in any case.

QuickCounters recognizes that the Windows Performance Counter technology that has
been with us for some time is often still the best choice for providing (and consuming)
detailed performance metrics.  QuickCounters also recognizes that for any given
service request there are several metrics which turn out to be interesting for performance
testing, operational health analysis, and historical trending. The idea with QuickCounters
is to raise the level of abstraction from that of an individual performance counter
up to the request itself, using a simple API.

Suppose you want to gather these metrics for each type of request in your system:

  • Requests Started
  • Requests Executing
  • Requests Completed
  • Requests Failed
  • Request Execution Time
  • Requests/Hour
  • Requests/Min
  • Requests/Sec

Although the .NET performance classes would give you a good start, it will still be
a chunk of work.  With QuickCounters, you describe your requests in a simple
xml format
, and include a simple code snippet in each request implementation:

void SampleRequest()
{
   RequestType someRequest = RequestType.Attach("MyApplication","someRequest");
   someRequest.BeginRequest();

   try
   {
      // Do useful work...
      someRequest.SetComplete();
   }
   catch
   {
      someRequest.SetAbort();
      throw;
   }
}
        

After a quick “install” of the xml that describes your requests (via the included
“QuickCounter viewer” utility) you are on your way.  You can see all eight metrics
described above for all of your requests (which each become a Performance object)
– in PerfMon, with MOM, or any other performance counter consumer. 

For a BizTalk orchestration, you will simply have a variable of type RequestType,
which you will assign in an expression shape at the top of your orchestration via
the static “RequestType.Attach” method, followed by a call to BeginRequest: 

quickCounterDemo = QuickCounters.RequestType.Attach(
    "QuickCounterDemo","demo",
    QuickCounterDemo(Microsoft.XLANGs.BaseTypes.InstanceId));            

Successful completion should end with another expression shape that calls SetComplete,
otherwise SetAbort.  (Full sample referenced below.)  And yes, the library
understands that an orchestration’s execution will often begin on one node in your
BizTalk group and continue/complete on a different node. 

Of course, if you have many requests, and many nodes in your BizTalk group, it can
be a hassle to add everything to PerfMon.  20 requests * 3 servers = 60 interactions
with PerfMon’s UI.  You might remember to save an .msc file, but you might not. 
Here is where the QuickCounter viewer comes into play.  Since you’ve already
written an xml file that describes your requests, just let the viewer know about that
file –  and a comma separated list of servers you are running on:



(click)

(The viewer has benefitted substantially from John Thom’s contributions…)

Now, the WCF support in the library is…amazing, and completely the result of contributions
from Tomas Restrepo.  Keep an eye
on his blog for a full discussion, but the bottom line is that creation of the simple
xml format
and all API calls (BeginRequest/SetComplete/SetAbort) are handled for
you automatically simply by applying an attribute (or a behavior – your choice)
to your WCF service implementation.  How cool is that?

Our CodePlex home is here
Downloads and source code here
Contributors here.

I expect we’ll release about once a month or so for awhile – give it a spin and let
us know what you think.  It is being used by some very large BizTalk projects
(in production) with good success right now.

Introducing QuickCounters…

Have you ever wanted to get real-time metrics on:  How many times has a particular
orchestration executed on this node in the BizTalk group?  What does the average
execution time for my pipeline component looking like?  How many instances of
a particular orchestration are currently executing across the group, not including
dehydrated instances?

Questions like this can be tough to answer – the built-in BizTalk counters are too
coarse-grained, and Tracking & BAM are only partially useful here (particularly
if you want high frequency samples.)  What you would like is the ability to get
fine-grained “request metrics” on all of your BizTalk orchestrations and pipelines
in a way that is simple to implement and easy to consume.

I’ve been leading an open source project called QuickCounters for
quite some time now – though we’ve just recently moved into CodePlex.  It is
a library that will be of interest to BizTalk developers – but it can be used in much
broader settings.  I had a chance to present on this topic at the Twin
Cities .NET Users Group (on 11/2/2006 – presentation here.)

QuickCounters is a .NET library which makes it extremely easy to implement the common,
request-level performance counters that are interesting in just about any service
you might write.  You can use this library to instrument  general .Net components,
web services, BizTalk orchestrations, pipeline components, remoting interfaces, enterprise
service components…you get the idea.  There is special support in the library
for BizTalk and WCF scenarios, but it is quite easy to use in any case.

QuickCounters recognizes that the Windows Performance Counter technology that has
been with us for some time is often still the best choice for providing (and consuming)
detailed performance metrics.  QuickCounters also recognizes that for any given
service request there are several metrics which turn out to be interesting for performance
testing, operational health analysis, and historical trending. The idea with QuickCounters
is to raise the level of abstraction from that of an individual performance counter
up to the request itself, using a simple API.

Suppose you want to gather these metrics for each type of request in your system:

  • Requests Started
  • Requests Executing
  • Requests Completed
  • Requests Failed
  • Request Execution Time
  • Requests/Hour
  • Requests/Min
  • Requests/Sec

Although the .NET performance classes would give you a good start, it will still be
a chunk of work.  With QuickCounters, you describe your requests in a simple
xml format
, and include a simple code snippet in each request implementation:

void SampleRequest()
{
   RequestType someRequest = RequestType.Attach("MyApplication","someRequest");
   someRequest.BeginRequest();

   try
   {
      // Do useful work...
      someRequest.SetComplete();
   }
   catch
   {
      someRequest.SetAbort();
      throw;
   }
}
        

After a quick “install” of the xml that describes your requests (via the included
“QuickCounter viewer” utility) you are on your way.  You can see all eight metrics
described above for all of your requests (which each become a Performance object)
– in PerfMon, with MOM, or any other performance counter consumer. 

For a BizTalk orchestration, you will simply have a variable of type RequestType,
which you will assign in an expression shape at the top of your orchestration via
the static “RequestType.Attach” method, followed by a call to BeginRequest: 

quickCounterDemo = QuickCounters.RequestType.Attach(
    "QuickCounterDemo","demo",
    QuickCounterDemo(Microsoft.XLANGs.BaseTypes.InstanceId)); 
quickCounterDemo.BeginRequest();           

Successful completion should end with another expression shape that calls SetComplete,
otherwise SetAbort.  (Full sample referenced below.)  And yes, the library
understands that an orchestration’s execution will often begin on one node in your
BizTalk group and continue/complete on a different node. 

Of course, if you have many requests, and many nodes in your BizTalk group, it can
be a hassle to add everything to PerfMon.  20 requests * 3 servers = 60 interactions
with PerfMon’s UI.  You might remember to save an .msc file, but you might not. 
Here is where the QuickCounter viewer comes into play.  Since you’ve already
written an xml file that describes your requests, just let the viewer know about that
file –  and a comma separated list of servers you are running on:



(click)

(The viewer has benefitted substantially from John Thom’s contributions…)

Now, the WCF support in the library is…amazing, and completely the result of contributions
from Tomas
Restrepo
.  Keep an eye on his blog for a full discussion, but the bottom
line is that creation of the simple
xml format
and all API calls (BeginRequest/SetComplete/SetAbort) are handled for
you automatically simply by applying an attribute (or a behavior – your choice)
to your WCF service implementation.  How cool is that?

Our CodePlex home is here
Downloads and source code here
Contributors here.

I expect we’ll release about once a month or so for awhile – give it a spin and let
us know what you think.  It is being used by some very large BizTalk projects
(in production) with good success right now.

MTOM Interoperability between Oracle Application Server and Windows Communication Foundation Part1: From WCF to Oracle

By Jesus Rodriguez

This article is part of a series intended to illustrate WS-* protocols interoperability scenarios between Windows Communication Foundation (WCF) and different J2EE Web Services platforms. Particularly this article focus on MTOM interoperability between WCF and Oracle Application Server v10.1.3.1.

Optimizing XML data transmission: The need for interoperability

The use of XML vs. binary data has been the eternal argument since the early days of XML-RPC and the subsequent SOAP specifications. In some scenarios, XML data formats can add significant overhead compared with the processing of binary data.  This is cause for concern with many architects. However, the use of XML has made vendors agree for first time in a series of protocols and standards for data transmission.

To assume that binary messages are always smaller than XML messages is overlooking one of the most complex challenges in data transmission today. However, binary data is present in a significant percent of data exchange scenarios.

From the Web Services viewpoint, the standards to encode binary data as part of SOAP messages had been evolving for a few years now. However at every step of that evolution, the implementation of interoperable Web Services that can exchange binary data across multiple platforms had raised as an interesting and difficult challenge for developers. Finally, after WS-Attachments and DIME, MTOM has gained acceptance among the different vendors as the main protocol to represent optimized binary data in a SOAP envelope.

MTOM

MTOM, the acronym that identifies the unpronounceable Message Transmission Optimization Mechanism, enables you to send binary data as part of a SOAP message in an efficient and optimized manner.  The classic solution for embedding  binary data in a XML document is to use a base64 encoded representation. Although, the based64 format presents severe limitations in regards to optimizing a binary representation. These limitations are primarily due to the use of 6 out of every 8 bits to encode a base64 character. That factor alone represents a 4:3 extra encoding overhead plus the addition of extra characters. The result is a significant increase in the size of a base64 encoded string.

MTOM avoids this overhead, attaching the binary data to the XML file in the original format without any extra encoding. MTOM relies on the XML Optimized Processing (XOP) Standard as the serialization mechanism to represent binary data as a MIME/Multipart Related package. For more information about MTOM you should read this great post by John Evdemon.

Despite of the fact that MTOM is W3C Standard; the need for interoperable implementations is key for MTOM to become a widely adopted protocol. During the last year different vendors had provided MTOM implementations as part of their Web Services platforms making MTOM interoperability a need and a challenge for the industry. This is even more significant when dealing with Web Services interoperability between.NET and J2EE, given that both platforms have followed different evolution paths.

The following sections are intended to illustrate an interoperability scenario using Oracle App Server and Windows Communication Foundation (WCF).

MTOM and WCF

MTOM is supported in WCF as part of the encoder model. An encoder implements a set of basics rules to represent the message on the wire. Encoders exist at both the client and the service side and are typically configured using encoding binding elements.

Oracle App Server and MTOM

With the recent release of the Application Server 10.1.3.1, Oracle adds supports for MTOM in addition to the existent MIME and DIME attachments. MTOM is now supported at both the client and the service side, only however, using SOAP 1.2.

The following sections illustrate how to build an Oracle Web Service and a WCF client that exchange binary messages optimized using MTOM

Using MTOM in Oracle App Server

As the center of our example lets use the following Web Service that returns a byte[] representing a binary file.

public class FileTransmitter {

    public FileTransmitter() {

    }

    public byte[] GetImg(String path)

      {

          try {

                  ReadableByteChannel channel = new FileInputStream(path).getChannel();

                  ByteBuffer buf = ByteBuffer.allocateDirect(12000);

                  byte[] result= new byte[12000];

                  int numRead = 0;

                  while (numRead >= 0) {

                      buf.rewind();

                      numRead = channel.read(buf);

                      buf.rewind();

                      for (int i=0; i<numRead; i++) {

                          byte b = buf.get();

                          result[i]= b;

                      }

                    

                  }

                  return result;

              }

              catch (Exception e)

              {

                return null;

              }

      }

}

Without using MTOM the response of this Web Service looks like the following.

<env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ns0="http://wsprj/types/"><env:Body><ns0:GetImgResponseElement><ns0:result>a2V5dG9vbCAtZ2Vua2

V5IC1hbGlhcyB0ZXN0IC1rZXlhbGcgIlJTQSIgLXNpZ2F

sZyAiU0hBMXdpdGhSU0EiIC1kbmFtZSAiQ049dGVzdCwgQz1VUyIgLWtleXBhc3MgdGVzdDEyMyA

ta2V5c3RvcmUgdGVzdC5qa3MgLXN0b3JlcGFzcyB0ZXN0MTIzDQoNCmtle

XRvb2wgLWNlcnRyZXEgLWFsaWFzIHRlc3QgLXQxMjMAAAAAAAAA … binary file</ns0:result></ns0:GetImgResponseElement></env:Body></env:Envelope>

As you can see the entire file is serialized as a base64 string and embedded into the SOAP message.

In order to configure MTOM at the service side is necessary to set the mtom-support setting to true in the web services configuration file.

<oracle-webservices xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://xmlns.oracle.com/oracleas/schema/oracle-webservices-10_0.xsd">

    <webservice-description name="OraMtomWS">

        <port-component name="OraMtomWSSoap12HttpPort">

        <mtom-support>true</mtom-support>

            <operations>

                <operation name="GetImg" input="{http://wsprj/types/}GetImgElement"/>

            </operations>

        </port-component>

    </webservice-description>

</oracle-webservices>

With those two simple steps the Web Service is ready to deploy to the Oracle Application Server.

Building the WCF client

The first step to create a WCF client is to generate the proxy using the SvcUtil tool. The following code is a sample client that invokes the Oracle Web Service.

static void Main(string[] args)

        {

            GetImgClient();

        }

        private static void GetImgClient()

        {

            OraMtomWSClient proxy = new OraMtomWSClient();

                byte[] result = proxy.GetImg("c:\\temp\\test2.file");

                Console.ReadLine();

         }

In order to achieve interoperability with Oracle App Server, the WCF client needs to encode the messages using SOAP 1.2 and MTOM. The following code illustrates the client configuration file.

<configuration>

          <system.serviceModel>

                   <bindings>

                             <customBinding>

                                      <binding name="OraMtomWSSoap12Http">

                                                <mtomMessageEncoding  messageVersion="Soap12" />

                                                <httpTransport manualAddressing="false" maxBufferPoolSize="524288"

                        maxReceivedMessageSize="65536" allowCookies="false" authenticationScheme="Anonymous"

                        bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"

                        keepAliveEnabled="true" maxBufferSize="65536" proxyAuthenticationScheme="Anonymous"

                        realm="" transferMode="Buffered" unsafeConnectionNtlmAuthentication="false"

                        useDefaultWebProxy="true" />

                                      </binding>

                             </customBinding>

                   </bindings>

                   <client>

                             <endpoint address="http://server/WSInterop-WSPrj-context-root/OraMtomWSSoap12HttpPort"

                binding="customBinding" bindingConfiguration="OraMtomWSSoap12Http"

                contract="OraMtomWS" name="OraMtomWSSoap12HttpPort" />

                   </client>

          </system.serviceModel>

</configuration>

The above settings configure the server to receive MTOM messages using SOAP 1.2. Executing the client produces the following messages on the wire.

<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope"><s:Body><GetImgElement xmlns="http://wsprj/types/"><path>file path…</path></GetImgElement></s:Body></s:Envelope>

Figure SOAP request produced by the WCF client

<?xml version="1.0" encoding="utf-8" ?>

<env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ns0="http://wsprj/types/"><env:Body><ns0:GetImgResponseElement><ns0:result><xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include" href="cid:7cfa880ad11d4faa90eadbc06de797c5"/></ns0:result></ns0:GetImgResponseElement></env:Body></env:Envelope>

Figure SOAP response produced by the Oracle Web Service

As you can see there is notable size difference between this response and the message produced when MTOM is not used. Regardless of the size factor, containing the message in its original binary format makes the MTOM message optimal when processing the binary data.

Where are we?

MTOM provides a standard and optimized mechanism to represent binary data in a SOAP envelope. This article explained the required steps create a Windows Communication Foundation client that can achieve MTOM interoperability with an Oracle Application Server web Service.  The second part of this article will complement this scenario demonstrating interoperability between Oracle client and a Windows Communication Foundation service.

Community Content for WF, WCF, and BizTalk Server

In addition to the content posted earlier, here is some great content from the community that showcases integration with these technologies. Here are a few:



  • MSDN Webcast: Integrating BizTalk Server 2006 and Windows Workflow Foundation by Mick Badran (MVP)
    http://www.microsoft.com/events/EventDetails.aspx?CMTYSvcSource=MSCOMMedia&Params=%7eCMTYDataSvcParams%5e%7earg+Name%3d%22ID%22+Value%3d%221032298240%22%2f%5e%7earg+Name%3d%22ProviderID%22+Value%3d%22A6B43178-497C-4225-BA42-DF595171F04C%22%2f%5e%7earg+Name%3d%22lang%22+Value%3d%22en%22%2f%5e%7earg+Name%3d%22cr%22+Value%3d%22US%22%2f%5e%7esParams%5e%7e%2fsParams%5e%7e%2fCMTYDataSvcParams%5e

  • BizTalk Server 2004/2006 and Windows Workflow Foundation by Brian Loesgen (MVP)
    http://geekswithblogs.net/bloesgen/archive/2005/10/09/56481.aspx

  • WorkFlow and BRE Example by Matt Milner (Pluralsight)
    http://pluralsight.com/blogs/matt/archive/2006/11/03/41629.aspx

  • Invoking TCP-Hosted WCF Services using BizTalk Server R2
    http://weblogs.asp.net/gsusx/archive/2006/11/21/invoking-tcp-hosted-wcf-services-using-biztalk-server-r2.aspx

  • Invoking WCF services from BizTalk Server using the WSE 3.0 adapter
    http://weblogs.asp.net/gsusx/archive/2006/05/10/445991.aspx

 

Consuming (parsing) EXCEL documents into BizTalk – Using Farpoint Spread

When I attended the SOA and BP conference in OCT in Redmond, I stopped by the FarPoint http://www.fpoint.com/ booth to checkout their new EXCEL parsing adapter they had developed for BizTalk 2006. Essentially its a dissasembler component that consumes XLS documents and maps them to a predefined XML schema. It looked pretty good!

EXCEL parsing is actually a common topic of conversation here in Australia. Many companies have processes in place that require their trading partners or agencies to send EXCEL spreadsheets to them and that data is often re-keyed into enterprise systems. Whoa… talk about error prone. Anyway…that’s why I stopped by at the farpoint booth as it captured my attention.

They have called this  FarPoint Spread for Microsoft%u00ae BizTalk%u00ae Server 2006 http://www.fpoint.com/biztalk/default.aspx and you can download a trial version. I have not had the chance to use it yet, but I will plan to try it in the coming weeks.

Anyway, Tom Canter emailed me recently to highlight that farpoint have their first official case study http://www.fpoint.com/company/studies/CaseStudy-FpSpreadBizTalk-GCS.pdf – worth a read.

BTSTask.exe – MSI packaging and Deployment in BizTalk 2006…almost….one BIG Gotcha

On the whole – BTSTask.exe is a great improvement over BTSDeploy.exe in 2004.
A couple of things I wouldnt mind seeing in this tool – the ability to start/stop
deployed applications
.

Big Gotcha

Something I came across after my large scripting effort…..it goes something like
this….
(1) 3 biztalk applications – a ‘Core’ and Two others.
(2) For the ‘Core’ apart from Schemas, Maps, Pipelines + Orchestrations; I had 1
COM Assembly, 2 custom adapters, 4 custom functoids and 3 custom pipeline components

So a reasonable sized deployment that needed to easily be deployed into test/production.

My line of thinking was – “If I could get all the associated *.dlls deployed
into the BTS Core Application wthin Development…all would be good.”
So as we know we can go through the BTS Admin Console and add resources/files/bindings
etc. to our application (with various options), that way when we say “Export
to MSI…”
it’s self contained.

The PROBLEM is in the ‘Destination Location’….
Using the BTSTask AddResource….. setting the ‘-Destination:’ parameter works a treat
(IF your destination location exists within development environment!)

Let me ellaborate….
Development:

e:\projects\<project name>\BizTalk\
     – maps
     – schemas etc….
Associated *.dlls – found e:\projects\<project name>\CommonBin\Release

Testing/Production:
d:\Applications\<project name>\CommonBin…etc. etc.

So the drives are different and what’s more, there is NO d-drive in Development….hmmm….I
thought.

(I didnt have a ‘demo’ project to highlight this….so I’ve removed company info from
below)

Where I want to focus is the ‘Destination Location’

These 4 assemblies are deployed using VS.NET 2005 straight from the developers desktop.
(When using BTSTask AddResource….-Destination:<loc>   – loc has
to exist at time of adding and ‘exporting MSI’ – bts validates)

Export to MSI…fine MSI finally created.

Installing the MSI file in Production/Testing

Upon performing MSIEXEC /quiet /i Core.MSI FOLDER=d:\Applications\<Project
Name>\CommonBin\
I ended up getting a ‘msi package deployment’ – a guid as a foldername with
*.CAB files underneath. No *.dlls etc to be seen.

Importing Into BizTalk

Went to BTS Admin Console and did ImportApp – all looked good.

Then went to the D-Drive and found no new files?? where were my biztalk files? gac-ed
files etc?

The ones that needed Gac-ing – found copied to the GAC
The BizTalk ones Schemas, Maps etc – found in E:\projects\<project
name>\<development project path>\Bin\Deployment\…..
smooth! 🙁

As far as I can tell this is attributed to the already existing ‘destination location’
within the MSI on the BizTalk artifacts.

So the reason why it’s soooooo close is that if we could override this (i.e. the above
FOLDER= parameter takes effect) then all would be sweet in going from environment
to environment.

As it stands at the moment, I’m deploying all the files to Testing building
the MSIs there and then deploying to Production with
all the correct paths hopefully.

Thought I’d save you some tears.

Community Content for WF, WCF, and BizTalk Server

In addition to the content posted earlier, here is some great content from the community that showcases integration with these technologies. Here are a few:



  • MSDN Webcast: Integrating BizTalk Server 2006 and Windows Workflow Foundation by Mick Badran (MVP)
    http://www.microsoft.com/events/EventDetails.aspx?CMTYSvcSource=MSCOMMedia&Params=%7eCMTYDataSvcParams%5e%7earg+Name%3d%22ID%22+Value%3d%221032298240%22%2f%5e%7earg+Name%3d%22ProviderID%22+Value%3d%22A6B43178-497C-4225-BA42-DF595171F04C%22%2f%5e%7earg+Name%3d%22lang%22+Value%3d%22en%22%2f%5e%7earg+Name%3d%22cr%22+Value%3d%22US%22%2f%5e%7esParams%5e%7e%2fsParams%5e%7e%2fCMTYDataSvcParams%5e

  • BizTalk Server 2004/2006 and Windows Workflow Foundation by Brian Loesgen (MVP)
    http://geekswithblogs.net/bloesgen/archive/2005/10/09/56481.aspx

  • WorkFlow and BRE Example by Matt Milner (Pluralsight)
    http://pluralsight.com/blogs/matt/archive/2006/11/03/41629.aspx

  • Invoking TCP-Hosted WCF Services using BizTalk Server R2
    http://weblogs.asp.net/gsusx/archive/2006/11/21/invoking-tcp-hosted-wcf-services-using-biztalk-server-r2.aspx

  • Invoking WCF services from BizTalk Server using the WSE 3.0 adapter
    http://weblogs.asp.net/gsusx/archive/2006/05/10/445991.aspx