BizTalk 2009 Testing Features

Warning: This post is about a beta product. Details
might change before final release.

The beta of BizTalk Server 2009 brings a welcome new feature: Some basic support for
Unit Testing BizTalk artifacts; specifically for Pipelines, Maps and Schemas.

You can read the basics about how the new feature works in the product
documentation
; there’s not too much yet, but it’s enough to get a good idea of
how you use it.

I’d like to now share a few thoughts on this feature. Note: This is just my personal
opinion from playing with the API for a bit.

The Good

The first really good part about this feature is the fact that it even exists at all.
Facilities for creating and automating tests of BizTalk artifacts has been a sorely
lacking area of the product.

While Visual Studio had the Test option for maps and schemas since BizTalk 2004, and
it was sort of useful, it required way too much manual intervention and there was
no way to decently drive it through automation, making it useless for either unit
or regression testing.

The second thing I like about the unit testing feature is that, in it’s current incarnation,
it’s not intrinsically tied to MSTest (i.e. the Visual Studio Team System stuff).
This means you can use it with NUnit, xUnit, or your favorite testing tool.

The third interesting thing about the API is that it is a very simple API. In most
cases it really consists of just a single method call. Here’s an example of testing
a schema:

string fin = @"c:\temp\test1_in.xml";
Schema1 schema = new Schema1();
Assert.IsTrue(schema.ValidateInstance(fin, OutputInstanceType.XML));

The Not-so-Good

Not everything is rosy in BizTalk-land. Personally, I was somewhat underwhelmed by
the implementation of the Unit Testing features available in beta 1; particularly
since some of it could be addressed with fairly minor adjustments.

API Issues

Let’s start with the API: I mentioned before that the new testing API was simple.
However, it’s actually too simple. For example, consider the small test for
a schema I presented above: You simply create an input file to test with, call ValidateInstance() and
check the result. Great, no?

Well, if the test succeeds, then yes, it’s just fine. If it fails, however, because
the input file does not pass validation, then, it turns out, it’s missing one extremely
important functionality: There’s no way to get any details about why the
schema validation failed [1].

This means that diagnosing a failed test involves, at the very least, changing your
project settings in VS to use the failed input file as input to the schema in VS,
use the Validate Instance option in the IDE and examine the results in the output
window, where fortunately you do get more detailed error information.

This is extremely sub-optimal and requires way too much manual work. Can you imagine
if a change in a schema broke tens of tests and having to do this for each and every
single one of those? Ouch!

Another place where I feel the current API is not as nice as it could be is that it’s
strictly file-based. That is, you need to create actual files on disk to represent
inputs to the tests and store the resulting outputs. This is somewhat convenient for
many scenarios, and matches the existing functionality in BizTalk pretty well. It
does, however, make it very inconvenient to deal with dynamically generated inputs
or inputs stored someplace else.

The classic example here, would be, resources. For example, I very much like to keep
my unit tests as independent from the environment and as self-hosting as possible.
One very nice way of doing this is to store necessary input files as embedded resources
on my testing assembly. I do this all the time with PipelineTesting and
it works great. However, since you can’t provide streams instead of file paths to
the BizTalk 2009 testing features, you’d need to first save the embedded resource
to a temporary file, which adds a bit of friction to the process.

Compilation Changes

My biggest gripe with the Unit Testing features, is that enabling them implies a change
in the build process, that changes the generated code for the BizTalk artifacts themselves.

When you enable the Unit Testing option in Project -> Properties -> Deployment,
the compilation process will change the generated CLR types so that they inherit from
the TestableXXX classes defined in the Microsoft.BizTalk.TestTools.dll assembly,
instead of the normal BizTalk artifact classes.

These new Testable base classes, in turn, derive from the original BizTalk classes,
so the change isn’t all that big. However, it does smell wrong. Could it
introduce any problems/bugs if you were to enable the testing features for all builds?
Hopefully no, but no way to know until you run into them.

Of course, you could just enable Unit Testing for debug builds, but then you can’t
run your unit tests on the release builds of your BizTalk assemblies. It also introduces
yet another asymmetry between development and deployment builds which, to be honest,
makes me somewhat nervous.

Conclusion

The new Unit Testing features in BizTalk Server 2009 are a sorely needed and welcome
feature, if somewhat lacking in the first beta.

Will it improve? On one hand, I’m somewhat optimistic that some improvements might
make an appearance in forthcoming builds. However, I’m also not getting my hopes up,
as MS is notable for trying to avoid significant API changes during after beta 1 hits
the street (BizTalk, unfortunately, doesn’t use the CTP model, where significant API
change requests are more likely to be considered).

[1] TestableMap.TestMap() does throw exceptions on failure instead of
a Boolean return value, but said exceptions might not contain detailed error information.

2008 coming to a close…..so Just how is BizTalk doing out there??? Gartner Reports

I recently came across a Gartner report talking about all things to do with ‘App Integration
with Back End Systems’ (in a nutshell… the report goes into detail)

The end result of several pages within this report is a graph (we like graphs 🙂 showing
Microsoft as a leader with a high ability to execute.

The Microsoft Technologies that fell under the microscope here are:

  1. BizTalk Server
  2. Windows Communication Foundation
  3. SQL Service Broker
  4. SQL Integration Services
  5. Team Foundation Suite
  6. Oslo + Azure

The graph looks as follows (snipped from the report):

 

Get the whole report HERE

Screencast: Configuring MEX endpoints on WCF Services

Screencast: Configuring MEX endpoints on WCF Services

I recently published a free video on Configuring MEX endpoints on WCF Services.

configuring-mex-endpoints-200

In this short WCF screencast, you'll learn how to configure your WCF services with Metadata Exchange (MEX) endpoints – both over HTTP and TCP – allowing consumers to automatically retrieve metadata from your services at runtime.  Enjoy!

Check out our growing collection of free .NET screencasts and videos.  Subscribe to the Pluralsight feed to be notified when new screencasts are published.  Also, check out our growing library of online .NET training courses — see what you can learn with Pluralsight On-Demand!

Screencast: Configuring MEX endpoints

Screencast: Configuring MEX endpoints

In this short WCF screencast, you'll learn how to configure your WCF services with Metadata Exchange (MEX) endpoints – both over HTTP and TCP – allowing consumers to automatically retrieve metadata from your services at runtime.  Enjoy!

configuring-mex-endpoints-300 

Be sure to check out our growing collection of short screencasts on the Pluralsight screencast page.

Previous WCF Screencasts (RSS for all posts in the series)

Failed to create the transport adapter

I have received this error when testing my adapter I am creating:

The Messaging Engine failed to create the transport adapter “SQLBulkXML”.
OutboundAssemblyPath: “D:\Eric\Warehouse\Adapter\Run-Time\SQLBulkXMLTransmit\bin\Debug\SQLBulkXMLTransmitAdapter.dll”
OutboundTypeName: “StottIS.BizTalk.Adapters.Runtime.SQLBulkXMLTransmitter.SQLBulkXMLTransmitAdapter”
Exception Details: “Guid should contain 32 digits with 4 dashes (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx).”

The issue is using the adapter wizard, it had the following code in the SQLBulkXML.transmitter.cs:

public SQLBulkXMLTransmitAdapter() : base( "SQLBulkXML Transmit Adapter", "1.0", "Allows data to be inserted into database using bulk xml import", "SQLBulkXML", new Guid(""), SQLBulkXMLNamespace, typeof(SQLBulkXMLTransmitterEndpoint), 10)

The problem is the new Guid(“”) is not filled in.

Refactoring a common anti-pattern when publishing web services from BizTalk

Background

I was recently reviewing a BizTalk application where there were around a dozen orchestrations which had been exposed as either SOAP or WSE web services. The typical pattern that had been used in these orchestrations is illustrated in the below picture. You can see that there is a Request/Response port and in the orchestration initialises based of the request message then returns a very simple acknowledgement type message (just a message of .net type System.Int32 with a value of 0) to indicate the orchestration has the message and is going to do some additional work.

This is a pattern I’ve seen implemented particularly with inexperienced teams. Often when I’ve seen this pattern the code also had poor testing and the team did not have much confidence in their implementation. They ended up with this pattern to ensure that the caller knew they had the message. The main issue the customer had was that occasionally they would experience a timeout at the client end particularly during load, it wasn’t a major issue as it only happened occasionally and the client could recover from this ok, but although the client got a timeout the orchestration still processed anyway and would make the appropriate updates to the original application to update it on the progress.

The Challenge

The challenge in this refactoring exercise was as follows:

  • The application was already in use so we are not able to change any of the contracts used by client applications. This means the WSDL for the published WSE and SOAP web services can not change.
  • We wanted to reduce the number of receive instances waiting around for a meaningless response and as result reduce the load a little

Before we get into the details of the refactoring, we will first look at the code samples from the previously generated SOAP and WSE services.

Existing WSE Web Service

In the below code snippet you can see that firstly I’ve had to censor some of the code from this article, but secondly it’s a standard web service generated with the WSE adapter. It inherits from the WSEReceiver class and invokes the receive method and indicates that it’s a 2 way call, then returns the response.

Existing SOAP Web Service

The below picture shows a sample of one of the SOAP web services generated from an orchestration. Again pretty standard generated web service using the SOAP generating wizard.

Additional Notes

In addition to what I’ve mentioned above, some other things which affect this refactoring are:

  • The web services are generated every time we run a build and they are also versioned

The Refactoring

The following sections will describe how we refactored the solution to implement our desired improvements.

Manually Implementing the Web Service Projects

The first thing we needed to do was to change the web services from being automatically generated to being two projects within the solution. We creates these two projects but with the request and response types being generated from schemas within the solution we wanted to maintain the ability to update these web services automatically so that any changes to schema were reflected in the web services. In the msbuild for the new web services projects we extended the build process so that C# classes were generated in the web projects representing the schemas using xsd.exe. In the below picture you can see we are generating these files into a folder called Contract within the web services.

After this we basically copied the artefacts from the generated web services to the manually build web service projects. We also needed to ensure the correct references were added.

Changing the SOAP Web Service

In each of the web services we needed to make a few small changes from what was generated. The first change (the top red circle) was to change how we assign the bodyTypeAssemblyQualifiedName variable. In the generated web services this is hardcoded by inspecting the assembly, but when we want to do this manually I added a reference to the schemas assembly and then just used the highlighted code snippet to get that name.

The second change (the bottom red circle) is the change to the call into the BizTalk assemblies where we change the oneway parameter to indicate that we are not waiting for a response. Previously the zero response message came from the orchestration meaning the messages route had been into the message box, into an orchestration, back into the messagebox and then finally back to the waiting instance to reply to the caller. By doing this as a one way call the code will basically wait until the message has been persisted to the messagebox and then continue.

You can see I then return my own integer of value 0 to indicate it has worked.

Note I’ve also implemented some custom logging to log any errors.

Changing the WSE Web Service

The WSE Web service changes are very similar to the soap ones. In the picture you can again see the change to how the schema qualified name is obtained. We have also changed the oneway parameter when we call the invoke method (although I’ve missed this off the right of the screen shot)

Changing the Orchestration

Now we are managing the web services as projects within our solution we need to change the orchestrations. The picture below shows we now have a one way receive port which is no longer late bound like previously, but is now direct bound based on the message type. By the time we have initiated the orchestration the caller already knows its request hit the messagebox and it does not need to wait around for confirmation that the orchestration has started.

Changing the bindings

We finally needed to update our binding file template to change the receive locations to be one way.

Testing

We had already implemented a number of BizUnit tests and were able to use these to confirm everything that had worked before the refactoring still worked exactly as it did now and that none of the web service contracts had changed.

Summary

In this post we discussed a common anti-pattern you seen implemented with BizTalk. We were able to take a pragmatic approach to addressing some of the issues the customer was facing with this implementation while minimising the affects to the consumers of services published from this application.

BizTalk Server 2009 MSBuild Tasks

I’ve been looking around the new project system introduced in the BizTalk Server 2009
beta. As you might have heard before, it’s now based on MSBuild. This is an extremely
welcome change from the obscure compilation model in previous BizTalk versions that
caused so much frustration.

BizTalk Projects now look much like regular C# projects (they even have the same icon
in solution explorer) but they can contain BizTalk artifacts like schemas, orchestrations,
maps and pipelines.

Opening up the .btproj file reveals some of the differences and custom
MSBuild tasks used by the new BizTalk project system. Two things are initially important
to mark an MSBuild file as a BizTalk project.

The first one is the <ProjectTypeGuids/> value:

<ProjectTypeGuids>{EF7E3281-CD33-11D4-8326-00C04FA0CE8D};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

This means:

  • {EF7E3281-CD33-11D4-8326-00C04FA0CE8D}: This represents the
    BizTalk project factory, and tells VS where to find the item templates for BizTalk
    artifacts and so on.

  • {FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}: This represents the
    C# project system.

So, in fact, a BizTalk Project is a C# project. Right now you can’t add C#
artifacts to the project without editing the .btproj file by hand, but
one can hope this will be supported in upcoming builds.

The second part important to making a .btproj is importing the BizTalk
MSBuild tasks, which is accomplished by this line:

<Import Project="$(MSBuildExtensionsPath)\Microsoft\BizTalk\BizTalkC.targets" />

If we look at the BizTalkC.targets file, we can see that the BizTalk MSBuild tasks
are implemented in the Microsoft.VisualStudio.BizTalkProject.BuildTasks assembly,
which you’ll find in your ‘Microsoft BizTalk Server 2009\Developer Tools’ folder.

Global Project Properties

There are several properties that will get set in the project files depending on your
project settings that are BizTalk-specific:

  • <BpelCompliance/>: This is a Boolean (true, false) property
    that indicates if the generated assembly should be marked as BPEL compliant (I think).

  • <EnableUnitTesting/>: Corresponds to the Enable Unit Testing
    option in the Deployment tab of the project settings dialog, and controls whether
    the unit testing features for
    pipelines, schemas and maps are enabled.

Other settings are per-user settings and stored in the .btproj.user file:

  • <Server/>: The name of the SQL Server instance that has
    the BizTalk databases for deployment.

  • <ConfigurationDatabase/>: The name of the BizTalkMgmt
    database.

  • <ApplicationName/>: The name of the BizTalk application
    you want to deploy to.

  • <Redeploy/>: Boolean property that indicates if you want
    to allow redeployments from within Visual Studio.

  • <RestartHostInstance/>: If true, the BizTalk hosts will
    be restarted after each deployment.

  • <Register/>: If true, the generated assembly will be registered
    in the Global Assembly Cache (GAC).

Per-File Project Properties

The per-user project file (.btproj.user) can also include a bunch of item-specific
properties, which are visible when you select a file in Solution Explorer and open
the Properties window.

In this category you’ll find properties for setting paths for map input and output
files for testing/debugging as well as paths for input and output files for testing
schemas.

Here’s an example:

<File Path="E:\Projects\BizTalk\PipelineTesting\SampleSchemas\Map1.btm">

   <ValidateTestMapInput>True</ValidateTestMapInput>

   <ValidateTestMapOutput>True</ValidateTestMapOutput>

   <TestMapInputInstanceFilename></TestMapInputInstanceFilename>

   <TestMapOutputInstanceFilename></TestMapOutputInstanceFilename>

   <TestMapSourceType>0</TestMapSourceType>

   <TestMapTargetType>0</TestMapTargetType>

   <EditorOutputInstanceFilename></EditorOutputInstanceFilename>

   <EditorInputInstanceFilename></EditorInputInstanceFilename>

   <GenerateInstanceOutputType>0</GenerateInstanceOutputType>

   <ValidateInstanceInputType>0</ValidateInstanceInputType>

   <PropertySchemaFileName>PropertySchema.xsd</PropertySchemaFileName>

   <AutoRefreshSchema>0</AutoRefreshSchema>

</File>

<File Path="E:\Projects\BizTalk\PipelineTesting\SampleSchemas\NoNS.xsd">

   <ValidateTestMapInput>True</ValidateTestMapInput>

   <ValidateTestMapOutput>True</ValidateTestMapOutput>

   <TestMapInputInstanceFilename></TestMapInputInstanceFilename>

   <TestMapOutputInstanceFilename></TestMapOutputInstanceFilename>

   <TestMapSourceType>0</TestMapSourceType>

   <TestMapTargetType>0</TestMapTargetType>

   <EditorOutputInstanceFilename></EditorOutputInstanceFilename>

   <EditorInputInstanceFilename></EditorInputInstanceFilename>

   <GenerateInstanceOutputType>0</GenerateInstanceOutputType>

   <ValidateInstanceInputType>0</ValidateInstanceInputType>

   <PropertySchemaFileName>PropertySchema.xsd</PropertySchemaFileName>

   <AutoRefreshSchema>0</AutoRefreshSchema>

</File>

Building Pipelines

To build a pipeline, you can use the PipelineCompilerTask:

<ItemGroup>

   <Pipeline Include="CSV_FF_RecvPipeline.btp">

      <Namespace>SampleSchemas</Namespace>

      <TypeName>CSV_FF_RecvPipeline</TypeName>

   </Pipeline>

</ItemGroup>

Building Schemas

BizTalk Schemas can be compiled using the new SchemaCompiler task:

<ItemGroup>

   <Schema Include="Schema1_NPP.xsd">

      <Namespace>SampleSchemas</Namespace>

      <TypeName>Schema1_NPP</TypeName>

   </Schema>

</ItemGroup>

Building Maps

BizTalk Maps can be compiled using the new MapperCompiler task:

<ItemGroup>

   <Map Include="Map1.btm">

      <TypeName>Map1</TypeName>

      <Namespace>SampleSchemas</Namespace>

      <SubType>Task</SubType>

   </Map>

</ItemGroup>

I’m not sure yet what the <Subtype/> element does.

Building Orchestrations

BizTalk Orchestrations can be built with the new XLangTask:

<ItemGroup>

   <XLang Include="BizTalk
Orchestration1.odx"
>

      <TypeName>BizTalk_Orchestration1</TypeName>

      <Namespace>SampleSchemas</Namespace>

      <SubType>Task</SubType>

   </XLang>

</ItemGroup>

<ItemGroup>

PipelineTesting and BizTalk 2009

Now that the a public beta release of BizTalk Server 2009 is out, I thought I’d download
it and try building and testing PipelineTesting on
the new release.

After downloading and installing the beta, I proceeded to import the existing Visual
Studio 2005 project file into Visual Studio 2008 with the new BizTalk project system
(a huge improvement over the old one, if I may say so [1]). The import worked just
fine overall.

The only change I had to make was remove the reference to PipelineObjects.dll from
the projects and add it again with the new version in BizTalk 2009. The new version
seems to be slightly different than the one in R2 (it’s 3KB smaller), but I haven’t
looked closely to see what the differences might be.

In either case, PipelineTesting builds perfectly with the new version, and all Unit
Tests are still passing in the green!

[1] I did hit a small snag with the project system: When building the imported solution,
only the SampleSchemas project gets built, though all are selected to build in the
configuration manager. Had to manually build the other projects from the project context
menu.