Warning: This post is about a beta product. Details
might change before final release.
The beta of BizTalk Server 2009 brings a welcome new feature: Some basic support for
Unit Testing BizTalk artifacts; specifically for Pipelines, Maps and Schemas.
You can read the basics about how the new feature works in the product
documentation; there’s not too much yet, but it’s enough to get a good idea of
how you use it.
I’d like to now share a few thoughts on this feature. Note: This is just my personal
opinion from playing with the API for a bit.
The first really good part about this feature is the fact that it even exists at all.
Facilities for creating and automating tests of BizTalk artifacts has been a sorely
lacking area of the product.
While Visual Studio had the Test option for maps and schemas since BizTalk 2004, and
it was sort of useful, it required way too much manual intervention and there was
no way to decently drive it through automation, making it useless for either unit
or regression testing.
The second thing I like about the unit testing feature is that, in it’s current incarnation,
it’s not intrinsically tied to MSTest (i.e. the Visual Studio Team System stuff).
This means you can use it with NUnit, xUnit, or your favorite testing tool.
The third interesting thing about the API is that it is a very simple API. In most
cases it really consists of just a single method call. Here’s an example of testing
string fin = @"c:\temp\test1_in.xml";
Schema1 schema = new Schema1();
Not everything is rosy in BizTalk-land. Personally, I was somewhat underwhelmed by
the implementation of the Unit Testing features available in beta 1; particularly
since some of it could be addressed with fairly minor adjustments.
Let’s start with the API: I mentioned before that the new testing API was simple.
However, it’s actually too simple. For example, consider the small test for
a schema I presented above: You simply create an input file to test with, call
check the result. Great, no?
Well, if the test succeeds, then yes, it’s just fine. If it fails, however, because
the input file does not pass validation, then, it turns out, it’s missing one extremely
important functionality: There’s no way to get any details about why the
schema validation failed .
This means that diagnosing a failed test involves, at the very least, changing your
project settings in VS to use the failed input file as input to the schema in VS,
use the Validate Instance option in the IDE and examine the results in the output
window, where fortunately you do get more detailed error information.
This is extremely sub-optimal and requires way too much manual work. Can you imagine
if a change in a schema broke tens of tests and having to do this for each and every
single one of those? Ouch!
Another place where I feel the current API is not as nice as it could be is that it’s
strictly file-based. That is, you need to create actual files on disk to represent
inputs to the tests and store the resulting outputs. This is somewhat convenient for
many scenarios, and matches the existing functionality in BizTalk pretty well. It
does, however, make it very inconvenient to deal with dynamically generated inputs
or inputs stored someplace else.
The classic example here, would be, resources. For example, I very much like to keep
my unit tests as independent from the environment and as self-hosting as possible.
One very nice way of doing this is to store necessary input files as embedded resources
on my testing assembly. I do this all the time with PipelineTesting and
it works great. However, since you can’t provide streams instead of file paths to
the BizTalk 2009 testing features, you’d need to first save the embedded resource
to a temporary file, which adds a bit of friction to the process.
My biggest gripe with the Unit Testing features, is that enabling them implies a change
in the build process, that changes the generated code for the BizTalk artifacts themselves.
When you enable the Unit Testing option in Project -> Properties -> Deployment,
the compilation process will change the generated CLR types so that they inherit from
TestableXXX classes defined in the
instead of the normal BizTalk artifact classes.
These new Testable base classes, in turn, derive from the original BizTalk classes,
so the change isn’t all that big. However, it does smell wrong. Could it
introduce any problems/bugs if you were to enable the testing features for all builds?
Hopefully no, but no way to know until you run into them.
Of course, you could just enable Unit Testing for debug builds, but then you can’t
run your unit tests on the release builds of your BizTalk assemblies. It also introduces
yet another asymmetry between development and deployment builds which, to be honest,
makes me somewhat nervous.
The new Unit Testing features in BizTalk Server 2009 are a sorely needed and welcome
feature, if somewhat lacking in the first beta.
Will it improve? On one hand, I’m somewhat optimistic that some improvements might
make an appearance in forthcoming builds. However, I’m also not getting my hopes up,
as MS is notable for trying to avoid significant API changes during after beta 1 hits
the street (BizTalk, unfortunately, doesn’t use the CTP model, where significant API
change requests are more likely to be considered).
TestableMap.TestMap() does throw exceptions on failure instead of
a Boolean return value, but said exceptions might not contain detailed error information.