by community-syndication | Apr 24, 2006 | BizTalk Community Blogs via Syndication
As many of you know, there are basically two
ways to test custom Pipelines and custom Pipeline Components in BizTalk Server:
-
To create them and test them by using them “for real” in biztalk messaging scenarios:
i.e. configure ports to use your custom pipelines and feed messages through them.
-
Use the pipeline.exe tool included in the BizTalk SDK to run them standalone.
>
Option 1 is required, definitely a needed option, but it is inconvenient, at
the least, for agile development: it is slow, cumbersome, and hard to automate. Option
2 is easier and faster, but it’s also inconvinient to automate.
So I had been looking at ways to automate the testing of pipelines and pipeline
components, and ran across this
article, explaining how to use mocks to unit test pipeline components. Neat, but
inconvenient as well. Furthermore, it does not allow you to test pipeline components
inside a real pipeline, which is sometimes necessary to ensure your component works
well with the built-in components in BizTalk (disassembler in particularly can be
pretty picky).
I realized then that MS already provided most of what you needed to test pipeline
components in a better way, and one which was possible to embed inside NUnit tests
(or whatever unit testing framework you use): Pipeline.exe relies on a set of helper
components in the PipelineObjects.dll assembly included with the SDK, which does the
heavy work of mocking the most important BizTalk objects that execute pipelines, such
as IPipelineContext, IBaseMessage and IBaseMessageFactory and so on. I figured that,
with a nicer API built on top of this, I could come up with something that really
made it easier to test your pipeline components in a more agile manner, and that complemented
the core unit tests you are creating for internal functionality.
So I’ve been working on this for the past couple of days and I’ve already have a working
implementation. Here’s an example of what it might look like:
/// <summary>
/// Tests
that we can execute successfully a loaded pipeline
/// with
a flat file as input
/// </summary>
[Test]
public void Test_ExecuteOK_FF()
{
ReceivePipelineWrapper pipeline =
PipelineFactory.CreateReceivePipeline(typeof(CSV_FF_RecvPipeline));
//
Create the input message to pass through the pipeline
Stream stream = DocLoader.LoadStream(“CSV_FF_RecvInput.txt”);
IBaseMessage inputMessage = MessageHelper.CreateFromStream(stream);
inputMessage.BodyPart.Charset = “UTF-8”;
//
Add the necessary schemas to the pipeline, so that
//
disassembling works
pipeline.AddDocSpec(typeof(Schema3_FF));
//
Execute the pipeline, and check the output
MessageCollection outputMessages = pipeline.Execute(inputMessage);
Assert.IsNotNull(outputMessages);
Assert.IsTrue(outputMessages.Count > 0);
}
The code above does the following:
-
Loads a receive pipeline, which in this case contains a Flat File Disassembler
-
Creates a new input message with a Flat File loaded from a resource stream
-
Loads the Flat File Schema the FFDasm will use to parse the message and makes it available
to the pipeline
-
Execute the pipeline and checks the output.
>>>
It’s cetainly not a lot of code, and personally I think the API is looking pretty
nice right now. So far, I got it working with the following scenarios:
-
Support for both receive and send pipelines
-
You can create pipelines programatically, by adding components to any stage, or load
an existing biztalk pipeline (the preffered method) using its type.
-
You can make schemas known to the pipeline before executing it by loading the necessary
biztalk schemas out of a biztalk assembly. I’ve tested it already with both XML and
Flat File schemas, schemas with one or multiple roots, schemas with promoted properties,
and envelope schemas.
-
Both debatching receive pipelines and batching send pipelines are supported through
the API.
>>>
Here’s another example, this time of doing multiple document batching into an envelope
using the XML Assembler:
/// <summary>
/// Tests
we can execute a send pipeline with
/// multiple
input messages and an envelope
/// </summary>
[Test]
public void Test_ExecuteOK_MultiInput()
{
SendPipelineWrapper pipeline =
PipelineFactory.CreateSendPipeline(typeof(Env_SendPipeline));
//
Create the input message to pass through the pipeline
string body =
@”<o:Body
xmlns:o=’http://SampleSchemas.SimpleBody’>
this
is a body</o:Body>”;
MessageCollection inputMessages = new MessageCollection();
inputMessages.Add(MessageHelper.CreateFromString(body));
inputMessages.Add(MessageHelper.CreateFromString(body));
inputMessages.Add(MessageHelper.CreateFromString(body));
//
Add the necessary schemas to the pipeline, so that
//
assembling works
pipeline.AddDocSpec(typeof(SimpleBody));
pipeline.AddDocSpec(typeof(SimpleEnv));
//
Execute the pipeline, and check the output
//
we get a single message batched with all the
//
messages grouped into the envelope’s body
IBaseMessage outputMessage = pipeline.Execute(inputMessages);
Assert.IsNotNull(outputMessage);
using ( StreamReader reader = new StreamReader(outputMessage.BodyPart.Data)
)
{
//
d contains the entire output message
string d = reader.ReadToEnd();
}
}
One important difference with working with the API I’m creating and pipeline.exe is
that the API is meant to be used with existing biztalk artifacts, which you can create
with regular biztalk projects (though they don’t need to be deployed to use them).
Pipeline.exe, on the other hand, works with the raw *.btp and *.xsd files, which is
simpler for some things, but far more inconvinient for testing, at least in my opinion.
I should be ready to post this in a few days, as I still need to test it more throughly
and fix a few things, but if anyone is interested, let
me know and I’ll pass on what code I have right now. Do note I’m implementing
this for BTS06 only, though I see no reason why you couldn’t port it to BTS04!
by community-syndication | Apr 24, 2006 | BizTalk Community Blogs via Syndication
Often when developing a generic business process, using generic documents makes the most sense.
Consider the core functionality in the ESB project I’m on, we need to take a document of ANY type (type will not be known at development time), and apply a map that will result in a document whose type is not known at development time. Without using generic documents, the only way to do this would be to have a specific mapping process for each combination of input and output documents. However, this is not a workable approach as the numbers of schemas and maps start to grow. Generic documents solve this problem. In fact, for the ESB project I am working on, generic documents are the cornerstone that functionality is being upon.
With BizTalk 2004, you would specify a generic document by creating a new message, and using the type browser to navigate to the System.Xml.XmlDocument class. In BizTalk 2006, System.Xml.XmlDocument has been elevated to a first class citizen status and is now available without needing to browse to it, alongside System.String and System.Boolean.
So, in theory, for my generic mapping requirement, I would receive an inbound System.Xml.XmlDcoument, and apply a map to create an outbound System.Xml.XmlDcoument.
This all works great, and in most cases is enough. However, there is one critical difference between using generic documents like this and using strongly typed documents. In the generic ESB case I was dealing with, the difference is enough to be a show-stopper. The problem is that promoted properties are not promoted in the outbound document if the document is not strongly typed (schema specified at design time).
To understand why, consider when property promotion happens. It can happen in a receive pipeline, once the document type is identified. Promoted properties for a known schema will have been defined in a property schema associated with that known schema. When you create a document of a known type in an orchestration, BizTalk is aware of the property schema associated with the new message and will promote the properties.
However, with a generic document, this does not happen. If you use direct-bound ports (which in my opinion is what all BizTalk developers should be doing in order to create extensible, loosely-coupled and scalable processes, see my previous post) you will find that an exception occurs when the outbound message is persisted to the MessageBox. The exception message is that BizTalk was unable to route the message because no subscribers were found. This will happen even if you have a valid subscription set up that is filtering on a promoted property in the message. This happens despite the fact that the associate property schema was deployed, and you can assign and read the properties in question in your orchestration. If you examine the message that got suspended, and look at the context properties, you’ll see that your values were assigned (so the subscription should have been found), but the property is in fact not promoted, so BizTalk has no way to match the subscription.
Pity, SOOOOO close to generic nirvana. So the question now becomes: is there a way to force property promotion for generic documents? The answer, very fortunately for those of us working with generic documents, is yes, it can be done.
In order to do this, you need to create a correlation set, and initialize it in the send port that is bound to the MessageBox. Although it may seem that you are creating a correlation set that you have no intention of following, but it does make sense as the subscriber(s) you have set up that are looking for those promoted properties are in effect following the correlation. You can create a single correlation type that references one or more message context properties, and they will all be promoted and become available to the messaging engine to do routing.
The punch line: in BizTalk 2006, this is actually now documented in the section dealing with correlation types. However, unless you know and understand the technique, you’d never go looking there. This is the reason I put this post together, enjoy, and happy travels as you do off on your loosely-coupled generic way!
by community-syndication | Apr 23, 2006 | BizTalk Community Blogs via Syndication
Yes, it’s still alive and rolling. This is a famous comet discovered in 1930 that broke apart on it’s return in 1995. It keeps disintegrating and there are over 20 parts identified. The brightest ones are already visible in amateur instruments as they about 9th magnitude now. In the middle of May when it passes near Earth the comet may reach 4th magnitude which is in naked eye range given clear sky and away from city lights. Comet updates and finding charts can be found on NASA and at Sky and Telescope site. Got to plan a night out in mountains. 🙂
by community-syndication | Apr 22, 2006 | BizTalk Community Blogs via Syndication
Here’s a cool code snippet (functoid) written by Carlos that can retrieve custom context properties for use inside a BizTalk Map.
http://biztalkblogs.com/carlosmedina/archive/2006/03/23/355.aspx
by community-syndication | Apr 21, 2006 | BizTalk Community Blogs via Syndication
If you didn’t read Understanding Application Deployment, be sure to view the archive. This explains the new features in BizTalk Server 2006 which assist with Application Deployment.
Now that you understand how to get applications deployed and running in production, at some point they’ll probably need to be upgraded. Perhaps a partner’s schema will change so your schema and map will have to be changed accordingly. Or maybe the business will require a new auditing API to be called during one part of the business process’s execution. And for as good as our BizTalk developers are, there are always code defects in application code that will need to be fixed.
So how does BizTalk Server accommodate these upgrades? Well, there are a couple of different ways to upgrade your running applications, and as always, different points to consider.
Simple Upgrade
One scenario we’ll deem the “Simple Upgrade Scenario”. This scenario requires no downtime, requires no code changes, and can be performed by an IT Pro or Business Analyst (BA) single-handedly.
Some examples include:
- An additional partner requests access to all orders of a certain type
- A business rule changes
- A queried web server will be phased out and replaced with a new one
Changes in this scenario are typically as simple as adding an additional send port to a send port group, adding an additional send port with appropriate subscription filters, changing the URI of a send port, updating a rule in the BRE, et cetera.
However, there may be more involved cases of application upgrade.
Patching Scenario
One such scenario is called the “Patching Scenario” in which existing application binaries in production must be edited and swapped with updated ones.
Typical examples include:
- Patching an orchestration with a code change
- Changing a schema
- Updating a map
In this case, the customer scenario must meet the following conditions:
- System downtime can be scheduled for application upgrade
- Customer does not have long-running business processes
- Dehydrated or suspended instances can be quickly resumed and completed, or alternately terminated
If the customer is updating an orchestration, say, and meets these criteria, the following steps might be their typical marching orders. First, the application binaries will be updated and recompiled with name and version unchanged. Then, downtime must be scheduled and new instances should be prevented from starting up. During this period, running instances will have to be stopped and unenlisted, which requires all dehydrated or suspended instances to be either manually resumed and completed, or terminated. After service instances are in the stopped and unenlisted state, the new application binaries can be deployed. First, the group’s MgmtDb should be updated by performing a Deploy operation using the overwrite flag. Second, each and every server in the group must be updated by GAC’ing the changed assembly. Finally, all BizTalk host instances should be restarted. At this point, the new orchestrations can be re-enlisted and started, and message flow can be resumed.
However, some customers may have more stringent requirements for upgrade. For example, they may not be able to schedule downtime or may have very long-running instances which cannot be terminated. In these cases, side-by-side versioning may be required.
Side-by-side (SxS) Versioning
This scenario, “Side-by-side Versioning”, allows two versions of the same application to be running side-by-side. The .NET runtime inherently allows for same-named but different versioned assemblies to be deployed and running. BizTalk also allows for this, although some discretion is advised.
BizTalk artifacts like maps are typically chosen by FQSN (fully-qualified strong name) which means the bindings are mindful of the version used. Making the code change in the map, upping the version number (major and minor builds only), compiling, and deploying the additional assembly to the group (and all machines) would then allow users to simply select the new map for inbound or outbound mapping. However, calling maps from orchestration may require code changes to the orchestration itself if the map reference is hard-coded.
Making changes to orchestrations can be a bit more involved. If you have short-lived orchestrations, then the “Patching Scenario”, previously described, may be sufficient. But if you have long-running orchestrations or cannot terminate existing instances, then side-by-side versioning will be your only alternative. Typically, the SxS story for orchestrations would go something like this.
Mortgage Company X needs to update their orchestration, but has many existing orchestration instances in flight that aren’t expected to terminate for weeks or months. They’d like for their existing instances to culminate on the live DLLs, but have new instances start up on the rev’d assemblies. To do this, they begin by having the developer increase the orchestration’s version number (major and minor builds only) and make the code changes to the orchestration. Next, they’ll deploy this new assembly to the group and GAC it on all runtime machines. Then the customer has the option to create new receive ports and locations for the new version or use existing ports. In the former is chosen, simply binding to the new ports and enlisting/starting the new artifacts will probably be sufficient. If the latter is chosen, some additional steps apply. The customer will have to (a) bind the new version of the orchestration to existing ports, (b) unenlist (but do not stop) the old orchestration version, and (c) enlist and start the new orchestration version. The documentation includes a script which help you to do steps (b) and (c) in one transaction so that messages are not missing subscriptions in between manual clicking.
The end result is that future publications will be directed to the new orchestration version and already running orchestration instances will complete on the older version. The product documentation should be consulted and followed closely for this kind of procedure, or for scenarios that depart from the “norm”, such as orchestration use of role links, direct binding, Specify Now binding, BAM activities, et cetera.
Upgrading pipelines has several alternatives as well. The simple solution is to simply choose the newly deployed pipeline version in the send port or receive location. This will replace the old pipeline with the new one. However, if true side-by-side functionality is required for backwards-compatibility purposes, then new send ports and receive locations will have to be created and bound to with the new pipeline version specified.
Finally, updating schemas also has some peculiarities. Side-by-side versioning is absolutely allowed, but the schema resolution behavior for the XML disassembler should be examined closely (described well in the docs). In certain cases, customers may want to hard-code references in the DASM properties to specific versions of the schemas to avoid dynamic resolution behavior. This will allow for more “pure” side-by-side scenarios.
How can MSIs be leveraged for upgrade scenarios?
Upgrading applications are typically a deliberate and precise operation in production. Because of this, following a manual checklist is advised. However, certain steps may be streamlined by using MSIs.
MSIs are a great way to wrap up your application artifacts into a distributable package. This may help when rolling out updated DLLs to multiple runtime boxes or assist with the group-level deploy. Precaution should be used when creating the MSIs to exclude all other unchanged resources and bindings from the package.
If conducting a “Patching Scenario”, stopping/unenlisting/re-enlisting/starting steps will need to be done manually before and after Importing/Installing the MSI. Similarly, numerous steps outlined above will have to be performed in the “Side-by-side Versioning Scenario” before and after using the MSI.
Still, MSIs can be useful for auditing purposes alone, since they leave a trace of the application’s upgrade installation on the machine in the Add/Remove Programs list.
More Information
In general, application upgrade is a cerebral activity and any change that you’ll ever make to your production environment should always be well tested and practiced in a pre-production environment. Having a non-production environment which mirrors the live environment will allow you to dry-run this and other production changes.
Needless to say, you should also carefully consult the product documentation before making any changes to your production environment, but this is especially true in the case of application upgrading. For more information on this topic, be sure to check out the documentation article entitled “Managing Artifacts”.
HTH,
Doug Girard
Note: This posting is provided “AS IS” with no warranties, and confers no rights.
by community-syndication | Apr 20, 2006 | BizTalk Community Blogs via Syndication
Adrian Hamza has posted 4 great demo videos covering the BizTalk Adapter for Windows SharePoint Services. These videos cover everything from setup & config, content based routing, InfoPath & Office Integration, and using SharePoint properties inside Orchestrations.
-Kris
by community-syndication | Apr 20, 2006 | BizTalk Community Blogs via Syndication
A common customer request has been to be able to easily print the BizTalk documentation. Now you can! Download the entire documentation in PDF format by visiting this link:
http://www.microsoft.com/downloads/details.aspx?FamilyId=8F3EC693-2865-4B85-8455-745511EA4267&displaylang=en
We’re listening, so please keep the great suggestions coming. And be sure to thank the documentation team!
by community-syndication | Apr 20, 2006 | BizTalk Community Blogs via Syndication
Kevin has released BizUnit 2006, a framework for unit testing BizTalk solution artifacts. This is a must have for any serious BizTalk developer’s toolbox.
-Kris
by community-syndication | Apr 20, 2006 | BizTalk Community Blogs via Syndication
When setting up WSS so you can do a full install of BizTalk, you may get an error similar to “System Error 1057 while trying to query service SPTimer”. For me, this occured while using the Configure Administrative Virtual Server section of Central Administration. I was using a local (non AD) service account that I had created. The problem is corrected if you make sure you enter “domain\user name” or “machine name\user name” format. Not easy to correlate this error description to the actual issue.
Cheers,
Todd
by community-syndication | Apr 19, 2006 | BizTalk Community Blogs via Syndication
Having done 2006 Installs/Configurations for all (public) versions of BizTalk Server 2006 from CTP to Beta2, I was surprised that the process was not much different for the RTM release.
There was so much hype as to how much the process had improved from 2004 to 2006, and to some extent I have to agree that the configuration after the install is now much easier than 2004 however …
The install part especially the check for prerequisites is still not there yet. Yes, there are so many prereqs that are still missing!
What would be nice for the Biztalk Server 2008 install is that ALL POSSIBLE recommended prereqs are available from a single location, ie prepackaged within a single CAB, and not spread here and there on subwebs on msdn etc…
For example for a vanilla machine install from bare metal, excluding the core essentials such as OS, Visual Studio, Sql Server which are understandably available from separate installs, all the supplementary filesets should be prepackaged in a cab file as part of the iso image.
One classic example is ADOMD.Net 8.0 + Patch + yes you guessed it not ADOMD.Net 9.0 but SqlServer2005_ADOMD.msi!
Please could they get the monikers correct in the prereqs check list!
Wish List, automate an install via script that takes params(such as credentials for automatic reboots etc) via XML config or whatever, initiate the msi to install all prereqs if necessary prepackaged from a cab file, and come back after a long lunch break, to see the post install configuration completed as well while you were away. Now that’s something..
Besides Install/Config and moving onto Integration itself, here’s a Biztalk Server 2008 Wish List done a while back last year 🙂
Oh by the way I was tempted to get/publish the Biztalk 2008 photos from flickr but instead I will just post the link.
I love artifact explorer/solution designer, as it allows an architect to model the end to end solution of a biztalk integration from a source system using biztalk receive artefacts that publish to the message box and likewise those processing artefacts that subscribe to the message box and eventually the send artifacts to the destination systems. For a deeper look at the end to end design of a future biztalk system have a look here and take time out to listen to Channel 9 Eddie Churchill’s Solution Designer Video, also take a preview into Eddie Churchill’s Mapper Video for Biztalk Server 2008
