Schema Design Patterns: Garden of Eden

This is the fourth of five entries talking about schema design patterns.  In previous entries the Russian Doll, the Salami Slice and the Venetian Blind approaches were discussed.


The Garden of Eden approach implements characteristics of the Venetian Blind and Salami Slice approach.  Like the Salami Slice the Garden of Eden uses a modular approach by defining all elements globally and like the Venetian Blind approach all type definitions are declared globally.  Each element is globally defined as an immediate child of the <schema> node and its type attribute can be set to one of the named complex types.  Since the elements are globally declared, the namespace must be exposed no matter what the settings of the elementFormDefault attribute is.  Therefore, if the elementFormDefault attribute is set to qualified, this does not change the content and structure of the instance document but if the attribute is set to unqualified then the local elements in the instance document must not be qualified with the prefix of the namespace prefix.


 


<?xml version=”1.0″ encoding=”UTF-8″?>
<xs:schema targetNamespace=”TargetNamespace” xmlns:TN=”TargetNamespace” xmlns:xs=”http://www.w3.org/2001/XMLSchema”
elementFormDefault=”qualified” attributeFormDefault=”unqualified”>
    <xs:element name=”BookInformation” type=”BookInformationType”/>
    <xs:complexType name=”BookInformationType”>
        <xs:sequence>
            <xs:element ref=”Title”/>
            <xs:element ref=”ISBN”/>
            <xs:element ref=”Publisher”/>
            <xs:element ref=”PeopleInvolved” maxOccurs=”unbounded”/>
        </xs:sequence>
    </xs:complexType>
    <xs:complexType name=”PeopleInvolvedType”>
        <xs:sequence>
            <xs:element name=”Author”/>
        </xs:sequence>
    </xs:complexType>
    <xs:element name=”Title”/>
    <xs:element name=”ISBN”/>
    <xs:element name=”Publisher”/>
    <xs:element name=”PeopleInvolved” type=”PeopleInvolvedType”/>
</xs:schema>


 


The advantage of this approach is that the schemas are reusable.  Since both the elements and types are defined globally both are available for reuse.  This approach offers the maximum amount of reusable content.


The disadvantages are the that the schema is verbose.  Since everything is globally declared and then referenced the size of the schema is increased.  Also, the schema is not self contained and can be coupled with other schemas thus possibly affecting other schemas with any changes made.  Lastly, there is no option to hide the namespace since they are always exposed on globally defined items.


This type of approach is good when need the flexibility of both global elements and global types.  This is also a good approach when you need to allow other schema authors to extend or restrict your types or you want to allow them to compose custom types by picking constituent types from your schema.  This would be an appropriate design when you are creating general libraries in which you can afford to make no assumptions about the scope of the schema elements and types and their use in other schemas particularly in reference to extensibility and modularity.

Orchestration Error handling: Fix-and-Log

Abstract: An exception has been thrown, what should I do? –> At least two actions: Fix the situation and Log it


Orchestrations tend to fail. Sure. This is because the nature of integration: Orchestrations deal with external applications, and expect behaviors that are not always as expected.
Exception handlers should be everywhere where errors can happen. The rule of thumb is that “Suspended/Non Resumable state should never occur”.
Each exception handler should perform at least two actions: fix the situation and log it.


Fix the situation
Fixing means not allowing the orchestration fail into a suspended state, and notify all the caller process. This is usually done by creating an error response message, or valid response that contains no data, but some detail on the error, so the flow can continue in some way.
Note that using the same response message or a different and specialized fault message usually depends on the message schemas and other requirements.


Log
A common practice when handling with errors is to log full message or context data (or both!) using the Windows Event Log or any other log framework. Typically, developers log data about the exception, the state of the orchestration, the message that raised the error, etc.
With this practice is very easy to duplicate a lot of data, because BizTalk already stores all this data for you. This information can be found in the tracking database, using HAT. The problem with HAT is that is difficult to find useful info if you are not sure what you are looking for.
So what kind of information should you log in case of error? -> Enough info to:


1. know what kind of error happen.
2. know where it has happen
3. be able to find the details using HAT


Nothing more. This is the tip: don’t log data that is logged by BizTalk. If there is an exception, BizTalk has the details. If message tracking is enabled, do not log message contents. Learn to use HAT efficiently, and you’ll save time and logging code.
Here is an easy sample of parallel Fix-and-Log:


 

Upcoming Events…

Just a few notes on what I’ll be up to in the next month or two…

Do you use the Deployment
Framework for BizTalk?  Or just have general questions on automating BizTalk
deployments?  You can attend my session at TechEd
2005
– Friday June 10th, 10:45am to 12:00pm in the BPI Cabana, where
I’ll be doing a talk on those topics…

I’ll also be speaking at the Microsoft
DevCon event on my “other favorite” topic, Windows Mobile.  The focus will
be on what is new for Visual Studio 2005 in the mobile arena…The conference will
also be covering the new Team System offerings, ASP.NET 2.0, and quite a few other
VS2005 topics.  Check it out…

Upcoming Events…

Just a few notes on what I’ll be up to in the next month or two…

Do you use the Deployment
Framework for BizTalk?  Or just have general questions on automating BizTalk
deployments?  You can attend my session at TechEd
2005
– Friday June 10th, 10:45am to 12:00pm in the BPI Cabana, where
I’ll be doing a talk on those topics…

I’ll also be speaking at the Microsoft
DevCon event on my “other favorite” topic, Windows Mobile.  The focus will
be on what is new for Visual Studio 2005 in the mobile arena…The conference will
also be covering the new Team System offerings, ASP.NET 2.0, and quite a few other
VS2005 topics.  Check it out…

A Few More Features for the Deployment Framework

(Update: See the latest on the Deployment Framework here.)

Well, its time for another revision of the BizTalk/NAnt deployment tools… 

For folks recently joining us, these tools allow you to quickly create both developer
workstation deployments (inside Visual Studio) and MSI-based deployments (for servers)
for BizTalk 2004 solutions.  Just about all aspects of BizTalk deployments are
handled, including items such as virtual directories, SOAP proxies, and WinXP/2k/2003
differences.

I get quite a few emails from people who are using these tools (which is great!) 
A couple folks have suggested that I start referring to the whole thing as a” p framework?. <
deployment>

That sounds about right – like all frameworks, there is a core set of functionality
that takes care of quite a bit of work, and a given project can supply what needs
customization or what is missing. 

In this case, the core NAnt scripts, custom NAnt tasks, WiX generation, and other
custom tools form the framework itself – and a given BizTalk project can supply project-specific
details, what needs to be customized, and any missing functionality.   So
I’ll refer to this work as the “Deployment Framework for BizTalk” from now on… 

What has changed in this rev? 

  • When last
    posting on this topic, I should have been even clearer that as of the last release, the
    deployment framework requires NAnt .85.
  • The biggest improvement is the ability to use binding files as they are emitted by
    the BizTalk Deployment Wizard directly.  In the past, the framework had
    insisted that binding files be split between orchestration bindings and messaging
    bindings – this is no longer required.  I had several requests for this fix…Also,
    note that having multiple binding files is supported (like one for each of
    several assemblies.)  Now, if you wish to continue to use the “old style”
    split bindings, define the orchBindings property in your project NAnt file as shown
    below:
  • <property name="orchBindings" value="${nant.project.name}.OrchBindings.xml"
    overwrite="false" />

  • You can now set the value of NAnt properties from the SettingsFileGenerator spreadsheet.
     Recall that this spreadsheet is used (in conjunction with XmlPreProcess) to
    manage environment-relative configuration – that is, to manage those values that vary
    depending on whether you are in a development, QA, or production environment.  Until
    now, it has been difficult to use those configuration values within the NAnt deployment
    file!  You typically just used them in your binding files or other application-level
    XML files.  Now, however, you can define a property as shown below to make these
    values available as NAnt properties.  A good example usage is the case where
    you want the actual names of the SSO Application Users group and SSO Administrators
    group to be defined in the SettingsFileGenerator spreadsheet (because the group names
    may vary by environment you deploy to.)  Just define a property in your project
    NAnt file called “propsFromEnvSettings” with a comma-separated list of values you
    want pulled in.  (The sample has been updated to show this, too.)
  • <property name="propsFromEnvSettings" value="ssoAppUserGroup,ssoAppAdminGroup"/>

  • Many people had asked for the DeployResults.txt file (which is created in the DeployResults
    subdirectory during a deployment) to be available in a history log, so that it would
    be easy to walk up to a production server and determine what deploy/undeploy activity
    had taken place (especially useful for diagnosing any failures.)  To that end,
    the deployment framework will preserve past DeployResult.txt files with this convention:
    `DeployResults TRACESATURN_05032005_0922.txt’  Note that the machine name, date,
    and time are preserved – so you can gather these files across all the servers in a
    BizTalk group, if need be.
  • SSOSettingsFileReader class improvements: In the last
    release, I introduced the ability to manage settings within the BizTalk SSO using
    the SettingsFileGenerator spreadsheet that is also used for managing environment-relative
    configuration.  Part of that functionality included a class called SSOSettingsFileReader,
    used to retrieve settings at runtime.  This class now has some strongly typed
    accessors, and no longer sits inside a lengthy namespace, which means that in an orchestration
    expression shape you can simply do:
  • someVar = SSOSettingsFileReader.ReadString("BizTalkSample","SomeAppConfigItem")
    // or ReadInt32

  • Ben Cops had found that on large-scale
    deployments, it was useful to restrict which orchestrations were deployed/started
    to speed the development cycle.  So, he modified the orchestration NAnt task
    accordingly.  To use (if you need to), just set the orchestrationsToIgnore property
    in your project NAnt file with a comma-separated list that reflects what you are currently
    working on.  Make sure to copy BizTalk.NAnt.Tasks.dll to your program files\nant\bin
    directory again!
    (you’ll find it in the DeployTools directory in the download.)
  • A bug was fixed with undeployment for the case where an SSO affiliate app had already
    been deleted…
  • All references in NAnt files to xmlns=”http://nant.sf.net/schemas/nant-0.84.win32.net-1.0.xsd”
    were removed, because they were simply causing heartache (for many reasons.)  Remove
    this namespace in your project NAnt file when upgrading.
  • Rather than trying to keep the source for the C# tools (NAnt tasks, SetEnvUI, DeployBTRules,
    SSOSettingsFileImport\SSOSettingsFileReader, etc.) current within the GotDotNet workspace,
    I’m making them available as a zip. 

Remember, you do not need to piece together the current functionality set from past
blog entries – the complete documentation (in Word form) is in the download and separately here.

Some last thoughts:

  • Some folks have been reluctant to use the deployment framework because of its reliance
    on NAnt, and a worry over using NAnt in a server environment.  One thing to remember
    (that might allay this concern) is that a subset of NAnt is delivered with the generated
    MSI, and NAnt will not require a separate installation procedure by your production
    operations team.  Moreover, NAnt doesn’t create any kind of persistent service
    or footprint on the production machine (i.e. no automatic file associations that would
    cause scripts to run with a double-click or anything like that…)
  • Add a Visual Studio External Tool for the updateOrchestration target if you haven’t
    already!  This can speed your development cycle significantly.  See the tools
    zip for a script that creates all relevant external tools.
  • When using the WiX-based MSI generation, make sure your project NAnt file (for WiX
    generation, like BizTalkSample.WiXSetup.Build) is choosing “debug” or “release” based
    on what you want to put in the MSI.  See comments in that file for more detail,
    and Jon Flanders’ post
    on this topic
    .

How do I upgrade?

  1. With a safe copy of your BizTalk solution, expand the “Core” zip contents right on
    your project directory, such that the core scripts will overwrite what you have now
    (but your project-specific scripts will be preserved.)
  2. Manually copy the WiX-related updates to your *.WiXSetup directory, since they will
    get unzipped to BizTalkSample.WiXSetup
  3. Remove xmlns=”http://nant.sf.net/schemas/nant-0.84.win32.net-1.0.xsd” from your project
    NAnt file
  4. Copy DeplyTools\BizTalk.NAnt.Tasks.dll to your program files\nant\bin directory
  5. Do a deployment on a developer workstation and a server to test things out…

Downloading

  • You can always use the download links on this blog.
  • Full sample here,
    core scripts here,
    extra tools here. 
  • (But always check the download links to see if there has been an
    update…)

  • Standalone documentation here (and
    diagram here.)
  • GotDotNet workspace here. 

Automating Creation of VS.NET External Tools

(Update: See the latest on the Deployment Framework here.)

In the latest release of the Deployment Framework for BizTalk “extra
tools”, you will find a script that will automate the creation of “external
tools” in Visual Studio that are useful for working with BizTalk 2004 and the
Deployment Framework.  Modify the script to include all the tools that your team
relies on, too.

The idea that this should be automated came from Ben
Cops
(though I’ve heard the complaint before!)  Manually creating all
the useful external tools was somewhat painful for a large team of BizTalk developers.

After you run the script, your Tools menu in Visual Studio should include entries
for deploying and un-deploying the current solution, as well as for updating just
the orchestrations, .NET components, and SSO configuration data (one menu entry.)
 In addition, there will be entries for running NAnt on a highlighted target
name (useful if you have your build script open), and for running HAT and the Subscription
Viewer.  Enjoy…

 

Automating Creation of VS.NET External Tools

(Update: See the latest on the Deployment Framework here.)

In the latest release of the Deployment Framework for BizTalk “extra
tools”, you will find a script that will automate the creation of “external
tools” in Visual Studio that are useful for working with BizTalk 2004 and the
Deployment Framework.  Modify the script to include all the tools that your team
relies on, too.

The idea that this should be automated came from Ben
Cops
(though I’ve heard the complaint before!)  Manually creating all
the useful external tools was somewhat painful for a large team of BizTalk developers.

After you run the script, your Tools menu in Visual Studio should include entries
for deploying and un-deploying the current solution, as well as for updating just
the orchestrations, .NET components, and SSO configuration data (one menu entry.)
 In addition, there will be entries for running NAnt on a highlighted target
name (useful if you have your build script open), and for running HAT and the Subscription
Viewer.  Enjoy…

 

Controlling the CLR Thread pool hosted by BizTalk

The CLR thread pool is global to the process, meaning that all components using the thread pool in a given process share the same thread pool and therefore compete for it. This for the most part is a good thing as it is more efficient in terms of system resources. When you do performance testing/tuning there are scenarios when you need to control the thread pool, for example when using the SOAP Adapter.


I came across this little gem last week, I hadn’t realized that Q886966 added the registry keys MinCompletionPortThreads, and MinWorkerThreads and then SP1 added the MaxWorkerThreads key below. These all exist under:


 


HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc{Guid}\CLR Hosting


MinWorkerThreads – the minimum number of worker threads that are available in the thread pool


MaxWorkerThreads – the number of worker threads in the thread pool


MinCompletionPortThreads – default 150


 


MinWorkerThreads can be tuned to address the problem due to the slow speed at which the thread pool can sometimes add new threads, this typically happens during service startup or during a burst of new load after a lull period. MaxWorkerThreads can be used to increase the number of threads in the thread pool, and this can be used to address the “out of threads issue” while using the SOAP adapter.


Note, as with most BizTalk registry keys they don’t exist by default, you’ll need to create them in order to use their non-default values.

Suddenly SOA does not exist!…

…or at least that’s what Clemens Vaster thinks. A little bit radical just to explain that SOA is an Orientation, not an Architecture itself… anyways, I agree with him, and I’ve added him to my blogroll…


It’s true that there’s too much hype with SO and SOA, so I’ll try to explain my point of view of SO in 3 over-simplifyed steps:


I want to benefit of Service Orientation. What should I do?



  1. Design different elements of the architecture to be independant each other (stateless, autonomous)
  2. For each element, separate interface from implementation.
  3. For each interface, use a technology that provides platform-independence and location-independence (SOAP over HTTP is a good example, but it’s not the only one)

…and that’s all, you are Service Oriented!


Well, we can discuss that platform-independance may not be required, but that’s another post…