by community-syndication | May 10, 2005 | BizTalk Community Blogs via Syndication
So, I’m looking at my blog today, and I see I haven’t posted anything in 2 months and a day. Shameful!!! Anyway, I’m climbing back in the saddle.
In what little free time I had before, I was working on a whitepaper for MSDN for the BizTalk Adapter for SQL Server. I’m glad to say the paper is complete, and should be on the MSDN site very shortly. I will post some examples from the paper here very shortly.
The question came up at work today about debatching in Orchestrations. This is a very common scenario in business processing, or at least the issue of dealing with batch style messages is. Stephen Thomas has done a great job in writing up some examples of how to deal with this. So good, that I won’t bother to recap. Anyway, there are a couple of important things to keep in mind when considering your options.
- Is it important that “good“ messages must be processesed, and “bad“ messages be dealt with?
- Are the file sizes significant?
- Does the service level agreement have specific time constraints?
- Does the trailer and header (if flat file messages) of the message have to be dealt with?
Many times you can’t let one single bad record cause an entire file not to be processed. If this is the case, you will have to deal with your messages in another fashion. One option that is common is to debatch, or “shred“ your messages in the orchestration using XPATH. You will notice from Stephen’s examples that when using XPATH to debatch messages inside your orchestration, peformance starts to degrade as the file size grows. I suspect what is happening is that as the XPATH iteration gets deeper in the nest, it is still starting at the beginning of the file, so it slows over time.
If you have larger files and have a time constraint, there are still a few options available to you. Since XPATH seems to be slow with large files due to it’s apparent scanning of the entire message each time, you may want to try Streaming XPATH. This option has all the benefits of XPATH, whithout the negative side effects. Look at this information to see if it might work for you.
There are still a couple of other options that may be available to you. I will start to post some more thoughts and examples on this topic in the next several weeks.
Cheers,
Todd
by community-syndication | May 10, 2005 | BizTalk Community Blogs via Syndication
At one of our customers we installed a complete new BizTalk infrastructure to improve the performance (throughput).
The following machines are involved:
1 x SQL Server dedicated to master message box processing wit publication disabled, single default instance.
Software
Microsoft SQL Server 2000 with SP3a and BizTalk SQL Hotfix SQL2000-KB810185-8.00.0878-ENU.exe
Microsoft Windows Server 2003 SP1
Database Instance BizTalkMsgBoxDb
2 x SQL Servers dedicated to message box publication, each 1 named instance.
Software
2 x Microsoft SQL Server 2000 with SP3a and BizTalk SQL Hotfix SQL2000-KB810185-8.00.0878-NU.exe
Microsoft Windows Server 2003 SP1
Database Instance BizTalkMsgBoxPubDb1 , BizTalkMngmtDb , EDIDb , SSO , etc
Database Instance BizTalkMsgBoxPubDb2 , BizTalkDTADb
1 x BizTalk 2004 Server Machine DL380 3 GB of RAM
Software
Microsoft BizTalk Enterprise Edition 2004 SP1
Microsoft Windows Server 2003 SP1
MDAC 2.8
MSXML / SQLXML
When BizTalk uses one singe MessageBox on the Master Message Box Machine there are no errors generated, BizTalk works fine. When we enable the one or two extra Publication MessageBoxes (via the BizTalk Server Administration Console), the BizTalk Engine starts with Network Errors.
“An attempt to connect to “BizTalkMsgBoxPubDb2“ SQL Server Database on Server “<SERVERNAME>“ failed with error: “[DBNETLIB][ConnectionWrite(send())]. General network error. Check your network documentation.“.
This has something to do with ACK’s and FIN’s I read in some discussion groups.
What I saw on the SQL Server where the Publication Message Box is configured were many SQL Server Locks.
BizTalk Server 2004 Engine starts up the connection with his own message boxes one minute later.
Any idea how we can fix this network problem or anyone got an idea ?
Many thanks for your input.
by community-syndication | May 10, 2005 | BizTalk Community Blogs via Syndication
This is the fourth of five entries talking about schema design patterns. In previous entries the Russian Doll, the Salami Slice and the Venetian Blind approaches were discussed.
The Garden of Eden approach implements characteristics of the Venetian Blind and Salami Slice approach. Like the Salami Slice the Garden of Eden uses a modular approach by defining all elements globally and like the Venetian Blind approach all type definitions are declared globally. Each element is globally defined as an immediate child of the <schema> node and its type attribute can be set to one of the named complex types. Since the elements are globally declared, the namespace must be exposed no matter what the settings of the elementFormDefault attribute is. Therefore, if the elementFormDefault attribute is set to qualified, this does not change the content and structure of the instance document but if the attribute is set to unqualified then the local elements in the instance document must not be qualified with the prefix of the namespace prefix.
<?xml version=”1.0″ encoding=”UTF-8″?>
<xs:schema targetNamespace=”TargetNamespace” xmlns:TN=”TargetNamespace” xmlns:xs=”http://www.w3.org/2001/XMLSchema”
elementFormDefault=”qualified” attributeFormDefault=”unqualified”>
<xs:element name=”BookInformation” type=”BookInformationType”/>
<xs:complexType name=”BookInformationType”>
<xs:sequence>
<xs:element ref=”Title”/>
<xs:element ref=”ISBN”/>
<xs:element ref=”Publisher”/>
<xs:element ref=”PeopleInvolved” maxOccurs=”unbounded”/>
</xs:sequence>
</xs:complexType>
<xs:complexType name=”PeopleInvolvedType”>
<xs:sequence>
<xs:element name=”Author”/>
</xs:sequence>
</xs:complexType>
<xs:element name=”Title”/>
<xs:element name=”ISBN”/>
<xs:element name=”Publisher”/>
<xs:element name=”PeopleInvolved” type=”PeopleInvolvedType”/>
</xs:schema>
The advantage of this approach is that the schemas are reusable. Since both the elements and types are defined globally both are available for reuse. This approach offers the maximum amount of reusable content.
The disadvantages are the that the schema is verbose. Since everything is globally declared and then referenced the size of the schema is increased. Also, the schema is not self contained and can be coupled with other schemas thus possibly affecting other schemas with any changes made. Lastly, there is no option to hide the namespace since they are always exposed on globally defined items.
This type of approach is good when need the flexibility of both global elements and global types. This is also a good approach when you need to allow other schema authors to extend or restrict your types or you want to allow them to compose custom types by picking constituent types from your schema. This would be an appropriate design when you are creating general libraries in which you can afford to make no assumptions about the scope of the schema elements and types and their use in other schemas particularly in reference to extensibility and modularity.
by community-syndication | May 9, 2005 | BizTalk Community Blogs via Syndication
Abstract: An exception has been thrown, what should I do? –> At least two actions: Fix the situation and Log it
Orchestrations tend to fail. Sure. This is because the nature of integration: Orchestrations deal with external applications, and expect behaviors that are not always as expected.
Exception handlers should be everywhere where errors can happen. The rule of thumb is that “Suspended/Non Resumable state should never occur”.
Each exception handler should perform at least two actions: fix the situation and log it.
Fix the situation
Fixing means not allowing the orchestration fail into a suspended state, and notify all the caller process. This is usually done by creating an error response message, or valid response that contains no data, but some detail on the error, so the flow can continue in some way.
Note that using the same response message or a different and specialized fault message usually depends on the message schemas and other requirements.
Log
A common practice when handling with errors is to log full message or context data (or both!) using the Windows Event Log or any other log framework. Typically, developers log data about the exception, the state of the orchestration, the message that raised the error, etc.
With this practice is very easy to duplicate a lot of data, because BizTalk already stores all this data for you. This information can be found in the tracking database, using HAT. The problem with HAT is that is difficult to find useful info if you are not sure what you are looking for.
So what kind of information should you log in case of error? -> Enough info to:
1. know what kind of error happen.
2. know where it has happen
3. be able to find the details using HAT
Nothing more. This is the tip: don’t log data that is logged by BizTalk. If there is an exception, BizTalk has the details. If message tracking is enabled, do not log message contents. Learn to use HAT efficiently, and you’ll save time and logging code.
Here is an easy sample of parallel Fix-and-Log:
by community-syndication | May 9, 2005 | BizTalk Community Blogs via Syndication
Just a few notes on what I’ll be up to in the next month or two…
Do you use the Deployment
Framework for BizTalk? Or just have general questions on automating BizTalk
deployments? You can attend my session at TechEd
2005 – Friday June 10th, 10:45am to 12:00pm in the BPI Cabana, where
I’ll be doing a talk on those topics…
I’ll also be speaking at the Microsoft
DevCon event on my “other favorite” topic, Windows Mobile. The focus will
be on what is new for Visual Studio 2005 in the mobile arena…The conference will
also be covering the new Team System offerings, ASP.NET 2.0, and quite a few other
VS2005 topics. Check it out…
by community-syndication | May 9, 2005 | BizTalk Community Blogs via Syndication
Just a few notes on what I’ll be up to in the next month or two…
Do you use the Deployment
Framework for BizTalk? Or just have general questions on automating BizTalk
deployments? You can attend my session at TechEd
2005 – Friday June 10th, 10:45am to 12:00pm in the BPI Cabana, where
I’ll be doing a talk on those topics…
I’ll also be speaking at the Microsoft
DevCon event on my “other favorite” topic, Windows Mobile. The focus will
be on what is new for Visual Studio 2005 in the mobile arena…The conference will
also be covering the new Team System offerings, ASP.NET 2.0, and quite a few other
VS2005 topics. Check it out…
by community-syndication | May 6, 2005 | BizTalk Community Blogs via Syndication
(Update: See the latest on the Deployment Framework here.)
Well, its time for another revision of the BizTalk/NAnt deployment tools…
For folks recently joining us, these tools allow you to quickly create both developer
workstation deployments (inside Visual Studio) and MSI-based deployments (for servers)
for BizTalk 2004 solutions. Just about all aspects of BizTalk deployments are
handled, including items such as virtual directories, SOAP proxies, and WinXP/2k/2003
differences.
I get quite a few emails from people who are using these tools (which is great!)
A couple folks have suggested that I start referring to the whole thing as a” p framework?. <
deployment>
That sounds about right – like all frameworks, there is a core set of functionality
that takes care of quite a bit of work, and a given project can supply what needs
customization or what is missing.
In this case, the core NAnt scripts, custom NAnt tasks, WiX generation, and other
custom tools form the framework itself – and a given BizTalk project can supply project-specific
details, what needs to be customized, and any missing functionality. So
I’ll refer to this work as the “Deployment Framework for BizTalk” from now on…
What has changed in this rev?
-
When last
posting on this topic, I should have been even clearer that as of the last release, the
deployment framework requires NAnt .85.
-
The biggest improvement is the ability to use binding files as they are emitted by
the BizTalk Deployment Wizard directly. In the past, the framework had
insisted that binding files be split between orchestration bindings and messaging
bindings – this is no longer required. I had several requests for this fix…Also,
note that having multiple binding files is supported (like one for each of
several assemblies.) Now, if you wish to continue to use the “old style”
split bindings, define the orchBindings property in your project NAnt file as shown
below:
<property name="orchBindings" value="${nant.project.name}.OrchBindings.xml"
overwrite="false" />
-
You can now set the value of NAnt properties from the SettingsFileGenerator spreadsheet.
Recall that this spreadsheet is used (in conjunction with XmlPreProcess) to
manage environment-relative configuration – that is, to manage those values that vary
depending on whether you are in a development, QA, or production environment. Until
now, it has been difficult to use those configuration values within the NAnt deployment
file! You typically just used them in your binding files or other application-level
XML files. Now, however, you can define a property as shown below to make these
values available as NAnt properties. A good example usage is the case where
you want the actual names of the SSO Application Users group and SSO Administrators
group to be defined in the SettingsFileGenerator spreadsheet (because the group names
may vary by environment you deploy to.) Just define a property in your project
NAnt file called “propsFromEnvSettings” with a comma-separated list of values you
want pulled in. (The sample has been updated to show this, too.)
<property name="propsFromEnvSettings" value="ssoAppUserGroup,ssoAppAdminGroup"/>
-
Many people had asked for the DeployResults.txt file (which is created in the DeployResults
subdirectory during a deployment) to be available in a history log, so that it would
be easy to walk up to a production server and determine what deploy/undeploy activity
had taken place (especially useful for diagnosing any failures.) To that end,
the deployment framework will preserve past DeployResult.txt files with this convention:
`DeployResults TRACESATURN_05032005_0922.txt’ Note that the machine name, date,
and time are preserved – so you can gather these files across all the servers in a
BizTalk group, if need be.
-
SSOSettingsFileReader class improvements: In the last
release, I introduced the ability to manage settings within the BizTalk SSO using
the SettingsFileGenerator spreadsheet that is also used for managing environment-relative
configuration. Part of that functionality included a class called SSOSettingsFileReader,
used to retrieve settings at runtime. This class now has some strongly typed
accessors, and no longer sits inside a lengthy namespace, which means that in an orchestration
expression shape you can simply do:
someVar = SSOSettingsFileReader.ReadString("BizTalkSample","SomeAppConfigItem")
// or ReadInt32
-
Ben Cops had found that on large-scale
deployments, it was useful to restrict which orchestrations were deployed/started
to speed the development cycle. So, he modified the orchestration NAnt task
accordingly. To use (if you need to), just set the orchestrationsToIgnore property
in your project NAnt file with a comma-separated list that reflects what you are currently
working on. Make sure to copy BizTalk.NAnt.Tasks.dll to your program files\nant\bin
directory again! (you’ll find it in the DeployTools directory in the download.)
-
A bug was fixed with undeployment for the case where an SSO affiliate app had already
been deleted…
-
All references in NAnt files to xmlns=”http://nant.sf.net/schemas/nant-0.84.win32.net-1.0.xsd”
were removed, because they were simply causing heartache (for many reasons.) Remove
this namespace in your project NAnt file when upgrading.
-
Rather than trying to keep the source for the C# tools (NAnt tasks, SetEnvUI, DeployBTRules,
SSOSettingsFileImport\SSOSettingsFileReader, etc.) current within the GotDotNet workspace,
I’m making them available as a zip.
Remember, you do not need to piece together the current functionality set from past
blog entries – the complete documentation (in Word form) is in the download and separately here.
Some last thoughts:
-
Some folks have been reluctant to use the deployment framework because of its reliance
on NAnt, and a worry over using NAnt in a server environment. One thing to remember
(that might allay this concern) is that a subset of NAnt is delivered with the generated
MSI, and NAnt will not require a separate installation procedure by your production
operations team. Moreover, NAnt doesn’t create any kind of persistent service
or footprint on the production machine (i.e. no automatic file associations that would
cause scripts to run with a double-click or anything like that…)
-
Add a Visual Studio External Tool for the updateOrchestration target if you haven’t
already! This can speed your development cycle significantly. See the tools
zip for a script that creates all relevant external tools.
-
When using the WiX-based MSI generation, make sure your project NAnt file (for WiX
generation, like BizTalkSample.WiXSetup.Build) is choosing “debug” or “release” based
on what you want to put in the MSI. See comments in that file for more detail,
and Jon Flanders’ post
on this topic.
How do I upgrade?
-
With a safe copy of your BizTalk solution, expand the “Core” zip contents right on
your project directory, such that the core scripts will overwrite what you have now
(but your project-specific scripts will be preserved.)
-
Manually copy the WiX-related updates to your *.WiXSetup directory, since they will
get unzipped to BizTalkSample.WiXSetup
-
Remove xmlns=”http://nant.sf.net/schemas/nant-0.84.win32.net-1.0.xsd” from your project
NAnt file
-
Copy DeplyTools\BizTalk.NAnt.Tasks.dll to your program files\nant\bin directory
-
Do a deployment on a developer workstation and a server to test things out…
Downloading
by community-syndication | May 6, 2005 | BizTalk Community Blogs via Syndication
(Update: See the latest on the Deployment Framework here.)
In the latest release of the Deployment Framework for BizTalk “extra
tools”, you will find a script that will automate the creation of “external
tools” in Visual Studio that are useful for working with BizTalk 2004 and the
Deployment Framework. Modify the script to include all the tools that your team
relies on, too.
The idea that this should be automated came from Ben
Cops (though I’ve heard the complaint before!) Manually creating all
the useful external tools was somewhat painful for a large team of BizTalk developers.
After you run the script, your Tools menu in Visual Studio should include entries
for deploying and un-deploying the current solution, as well as for updating just
the orchestrations, .NET components, and SSO configuration data (one menu entry.)
In addition, there will be entries for running NAnt on a highlighted target
name (useful if you have your build script open), and for running HAT and the Subscription
Viewer. Enjoy…
by community-syndication | May 6, 2005 | BizTalk Community Blogs via Syndication
(Update: See the latest on the Deployment Framework here.)
In the latest release of the Deployment Framework for BizTalk “extra
tools”, you will find a script that will automate the creation of “external
tools” in Visual Studio that are useful for working with BizTalk 2004 and the
Deployment Framework. Modify the script to include all the tools that your team
relies on, too.
The idea that this should be automated came from Ben
Cops (though I’ve heard the complaint before!) Manually creating all
the useful external tools was somewhat painful for a large team of BizTalk developers.
After you run the script, your Tools menu in Visual Studio should include entries
for deploying and un-deploying the current solution, as well as for updating just
the orchestrations, .NET components, and SSO configuration data (one menu entry.)
In addition, there will be entries for running NAnt on a highlighted target
name (useful if you have your build script open), and for running HAT and the Subscription
Viewer. Enjoy…
by community-syndication | May 4, 2005 | BizTalk Community Blogs via Syndication
The CLR thread pool is global to the process, meaning that all components using the thread pool in a given process share the same thread pool and therefore compete for it. This for the most part is a good thing as it is more efficient in terms of system resources. When you do performance testing/tuning there are scenarios when you need to control the thread pool, for example when using the SOAP Adapter.
I came across this little gem last week, I hadn’t realized that Q886966 added the registry keys MinCompletionPortThreads, and MinWorkerThreads and then SP1 added the MaxWorkerThreads key below. These all exist under:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc{Guid}\CLR Hosting
MinWorkerThreads – the minimum number of worker threads that are available in the thread pool
MaxWorkerThreads – the number of worker threads in the thread pool
MinCompletionPortThreads – default 150
MinWorkerThreads can be tuned to address the problem due to the slow speed at which the thread pool can sometimes add new threads, this typically happens during service startup or during a burst of new load after a lull period. MaxWorkerThreads can be used to increase the number of threads in the thread pool, and this can be used to address the “out of threads issue” while using the SOAP adapter.
Note, as with most BizTalk registry keys they don’t exist by default, you’ll need to create them in order to use their non-default values.
