BizTalk 2004 "Commando" Training course – More posts to come…
Done for the night….I’ll post a lot more later this week…
BizTalk 2004 "Commando" Training course – App Domains
Remember, during the course we went over App Domain configuration for BizTalk. The key to creating App Domains with BizTalk, and hence being able to use configuration files with your project is to edit the BTSNTSvc.exe.config file located in the BizTalk program directory. The next key is to make sure you put the right sections in the right spot within the config file, else the BizTalk service will never start again (I ran into this myself once).
Why would you use them? Well, here’s a couple of reasons:
1) It gives you the ability to configure them separately. This is useful for orchestrations that use 3rd party component that read their configuration for the config file and you have different requirements for these 3rd party components when they are used from different orchestrations. You can also specify an assembly search-path on a per-app-domain basis.
2) The unit of code-unloading in .Net is app-domain. If you put separate orchestrations into app-domains there is an opportunity to unload some of them while the rest are still in use. For example, you can put you frequently used orchestrations into a “hot” app-domain and the less-used orchestrations into a “cold” app-domain. The hot app-domain will be continually used. The cold app-domain will get recycled after 25 minutes of idle time. You can control the unloading times with app-domain configuration (see MSDN).
3) It gives you some level of isolation. For example, if you have a singleton object—you’ll actually get one per app-domain—so one orchestration cannot mess the other orchestration’s singleton.
4) Internally, XLANG can use app-domains to host different versions of the XLANG runtime (e.g., Voyager vs. Pathfinder)
Here’s a sample of the edits to the BizTalk config file. Remember, the configSections comes first, and the sections it defines comes last….all the new additions are in RED.
<?xml version=”1.0″ ?>
<configuration>
<configSections>
<section
name=”xlangs”
type=”Microsoft.XLANGs.BizTalk.CrossProcess.XmlSerializationConfigurationSectionHandler, Microsoft.XLANGs.BizTalk.CrossProcess” />
</configSections>
<runtime>
<assemblyBinding xmlns=”urn:schemas-microsoft-com:asm.v1″>
<probing privatePath=”BizTalk Assemblies;Developer Tools;Tracking;Tracking\interop” />
</assemblyBinding>
</runtime>
<system.runtime.remoting>
<channelSinkProviders>
<serverProviders>
<provider id=”sspi” type=”Microsoft.BizTalk.XLANGs.BTXEngine.SecurityServerChannelSinkProvider,Microsoft.XLANGs.BizTalk.Engine” securityPackage=”ntlm” authenticationLevel=”packetPrivacy” />
</serverProviders>
</channelSinkProviders>
<application>
<channels>
<channel ref=”tcp” port=”0″ name=””>
<serverProviders>
<provider ref=”sspi” />
<formatter ref=”binary” typeFilterLevel=”Full”/>
</serverProviders>
</channel>
</channels>
</application>
</system.runtime.remoting>
<xlangs>
<Configuration>
<AppDomains AssembliesPerDomain=”10″>
<DefaultSpec SecondsIdleBeforeShutdown=”5″ SecondsEmptyBeforeShutdown=”5″>
</DefaultSpec>
<AppDomainSpecs>
<AppDomainSpec Name=”MyAppDomain”>
<BaseSetup>
<ConfigurationFile>C:\Projects\Projects\AppDomainConfig\MyAppDomainConfiguration.config</ConfigurationFile>
</BaseSetup>
</AppDomainSpec>
</AppDomainSpecs>
<PatternAssignmentRules>
<PatternAssignmentRule AssemblyNamePattern=”AppDomainConfig.*” AppDomainName=”MyAppDomain” />
</PatternAssignmentRules>
</AppDomains>
</Configuration>
</xlangs>
</configuration>
Lastly, here’s the sample we walked through in class. Yossi Levanoni put this together back in September of 03′.
The sample demonstrates how to instruct the orchestration engine to execute top-level (i.e., not-called) orchestrations from a particular assembly in a particular app domain; then how to specify configuration for this domain. In this sample, a configuration file is associated with the domain. The config file contains a custom configuration section. The test orchestration calls into user code and consumes this custom configuration section. The data is inserted into a flat-file-like XLANG message which is sent to an output folder.
To test it:
1) Compile.
2) Gac/deploy/modify binding.
3) Copy the BTSNTSvc.exe.config to your program files dir. You might want to edit this file so that it points to the correct location for MyAppDomainConfiguration.Config on your machine.
4) Drop any file (it’s just used to start the orchestration) with .xml extension into DataFolders\in.
5) You should see a file with a name like ConfigOut_<guid>.txt generated in DataFolders\out.
6) The contents of the file are:
key=”setting1″ val=”Value1″
key=”setting2″ val=”value two”
key=”setting3″ val=”third value”
Which corresponds to the contents of MyAppDomainConfiguration.config:
<?xml version=”1.0″?>
<configuration>
<configSections>
<section name=”sampleSection”
type=”System.Configuration.SingleTagSectionHandler” />
</configSections>
<sampleSection setting1=”Value1″ setting2=”value two”
setting3=”third value” /> </configuration>
Compiled BizTalk 2004 Help File (.chm)
I find this very handy for my laptop that does NOT have BizTalk Server 2004 installed on it. Its a compiled version of the online docs….a
convenient chm file. Its updated to SP1. Enjoy.
BizTalk Server 2004 SP1 should be MANDATORY for production environments
I can’t believe I see more and more recent BizTalk implemenations in production WITHOUT SP1. With all the issues that we fixed in SP1, any new BizTalk Server 2004 project should simply never go into production without SP1. I can’t stress that enough. The product team did phenomenal work raising the bar on the quality of this service pack. It can be downloaded here.
Additionally, I would also recommend that you contact premier support and ask for the following POST SP1 fixes, if you encounter any of the issues described below…which you undoubtedly will.
lDatabase Functoid Caching – you’ll see this if you’re using our database functoid…just get it if you are trying to use our database functoids.
lDelivery Notification forcing dehydration of orchestration – you’ll see this if using DN in orchestration….another “just get it” if you are using DN.
lSide by Side Deployment
lPerformance degradation with Singleton Orchestration – under certain conditions, using transactions is one of them, you’ll run into this.
lLeak in WMI provider
lBody Tracking before receive
lOOM Errors with MQSeries Adapter ****
lPipeline versions Side by Side
lEDI adapter update (KB894785) – Fixes a lot of things…
lMicrosoft.XLANGs.Core.PersistenceException – I’ve seen this almost every time….I would say just get it as well.
File Adapter losing files if SQL shuts down – this is a bad one…..another “just get it” fix.
BizTalk 2004 "Commando" Training course – Decks and Samples
Here’s the powerpoint decks we used for some of the extended sessions. Some of these have added sections on all the perf tuning recommendations we layed out. There’s quite a bit that you have to consider when working in a farm scenario. Also, you’ll find some best practice around flat file, and the Flat File project I walked you through building. In all you’ll find the following:
1. Advanced Orchestration
2. Rules Engine
3. Building Pipelines
4. Messaging and Flat File best practice
5. Installation of farms and best practice and tuning parameters
Here are the decks, and here is the project we walked through to demonstrate parsing Flat Files…
Messaging-Database Aggregator Pattern
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
al
al
2
282
2005-06-06T16:37:00Z
2005-06-06T16:37:00Z
1
1651
9414
Know IT
78
18
11561
9.3821
21
It’s been a
while since I blogged an article, and a long while since I blogged about the
“Sequential Convoy Aggregator” pattern. Back then I was fairly new to BizTalk,
I was aware of persistence points, and orchestration state, but did not have a
really clear understanding of how badly they could affect an orchestration
design.
Since then
I have been thinking about this more, and wondering how well my aggregator
implementation would stand up in a high load scenario. After running a few
tests, I discovered the design pattern does not scale well to aggregate large
batches of messages. The problem then was to come up with a design that would
Pattern Implementation
Using an Orchestration
The
sequential convoy aggregator pattern is implemented by building a sequential
convoy orchestration with two receive shapes correlated to receive all the
messages that are to be aggregated. The orchestration maintains the state of
the aggregation, by creating a .NET XmlDocument object, and adding each message
as a node to this document. When the aggregation is complete, the XmlDocument
object is assigned to a BizTalk message, which is then sent from the
orchestration to a subscribing send port.
Sequential Convoy
Aggregator Performance
The
performance tests are using the aggregator pattern I created and blogged about
last August. The orchestration was modified slightly to use a message count for
the completeness criteria so it could be tested with different numbers of
messages. The tests were then run, starting with 100 messages then increasing
the size of the messages being aggregated.
After each
test, the “Backup BizTalk Server” SQL Server Agent job was manually started to
create log backups for the processing of the message aggregation. The results
of the test are shown below.
|
Msg |
Log Size /KB |
KB/Msg |
|
100 |
7,038 |
70.38 |
|
200 |
16,446 |
82.23 |
|
300 |
29,999 |
100.00 |
|
400 |
47,703 |
119.26 |
|
600 |
96,976 |
161.63 |
|
800 |
171,584 |
214.48 |
|
1,000 |
262,825 |
262.83 |
|
Msg |
The |
|
Log |
The size |
|
KB/Msg |
The size |
From the
results of the tests, it can be seen that the size of the message box database
log increases dramatically as the number of messages being aggregated
increases. As the log file is a log of the database activity on the message box
database, this implies that the aggregator implementation does not scale well
as the number of messages increases.
If there
were thousands of messages to aggregate over a 24-hour period for example, the
size of the message box database log files would become a significant problem.
The Problem or Persistence
In order to
provide reliable messaging, the BizTalk Server orchestration engine saves its
state to the message box database at specific points during it’s execution.
This serialization of the message state is known as persistence, and each
saving of the orchestration state is known as a persistence point. Whilst these
orchestration persistence is great from the reliability perspective, it can
have some detrimental effects in terms of performance if orchestrations are
designed without paying a thought to the mechanics of the orchestration engine.
In the
aggregator pattern, as the orchestration starts to aggregate messages, the size
of the XmlDocument object that aggregated the messages is small. At each
persistence point, this XmlDocument object is serialized to the message box
database to allow recovery form a possible failure. As the orchestration
aggregates more and more messages, the size of the XmlDocument object
increases.
This object
is persisted to the database (three times for each message aggregation loop in this
case), every time a message is received, so when the orchestration is holding
the stare of a few hundred messages, the load on the message box database
starts to get heavy. If the patterns is used to aggregate thousands of
messages, and the messages themselves are large, the pattern will quickly
become unusable.
Building a Scalable
Aggregator
The problem
affecting the scalability of the aggregator was caused by maintaining the state
of the aggregated message as the orchestration looped to receive the other
messages in the convoy. If a way can be found to escape the state persistence
problem, it should be possible to design a scaleable aggregator pattern.
Aggregator Design Using
Messaging and a Database
One of the
possible designs would be to abandon the convoy orchestration, and create a
solution that was based purely on messaging. At present, in BizTalk Server
2004, there is no way to aggregate messages in the pipeline of a send port in a
similar manor to splitting in a receive pipeline. This pattern uses a SQL
Server database to store the messages as they are aggregated, and uses the SQL
Server adapter to add the orders to the database, and to receive the aggregated
list when the aggregation is complete. The completeness criteria is determined
in the database, and in this case, a message count is used.
Messaging-Database
Aggregator Pattern
|
In File |
The order |
|
Order |
The |
|
Add Order (Map |
Port with |
|
Add Order |
Stored |
|
Order |
Database |
|
Get Order |
Stored |
|
Get Order SQL |
Receive |
|
Aggregated (Map SQL |
Send port SQL |
|
Aggregated |
File |
Test Results
The tests
were repeated using the messaging-database aggregator pattern, and the
following results obtained. The Messaging log size is a sum of the message box
database log file, and the order aggregation log file.
|
Msg |
Convoy Log |
Convoy |
Messaging |
Messaging |
% Log |
|
100 |
7,038 |
70.38 |
3,064.00 |
30.64 |
56.46% |
|
200 |
16,446 |
82.23 |
5,984.00 |
29.92 |
63.61% |
|
300 |
29,999 |
100.00 |
8,676.00 |
28.92 |
71.08% |
|
400 |
47,703 |
119.26 |
10,977.00 |
27.44 |
76.99% |
|
600 |
96,976 |
161.63 |
16,866.00 |
28.11 |
82.61% |
|
800 |
171,584 |
214.48 |
21,755.00 |
27.19 |
87.32% |
|
1,000 |
262,825 |
262.83 |
26,593.00 |
26.59 |
89.88% |
|
Msg |
The |
|
Convoy Log Size /KB |
The size |
|
Convoy |
The size |
|
Messaging |
The size |
|
Messaging |
The size |
|
% Log |
The |
KB Log File Size for
Aggregation / Message Count
The effect
of the batch size on the sequential convoy orchestration can clearly be seen in
the above graph. As the batch size increases, the messaging-database aggregator
offers significantly better performance.
KB Log File Size per Message
/ Message Count
As the
batch size increases, the messaging-database aggregator becomes slightly more
efficient, producing less log file activity per message. With the orchestration
based aggregator, continues persistence of the increasing aggregation state
makes the pattern less efficient.
Why is the Message Box
Database Log File Size so Important?
The log
files created when performing a transaction log backup on the message box
database are an indication of the load that has been placed on the database
since the last transaction log backup. As a BizTalk server installation is
scaled up, and out, the load on the message box database is often a bottleneck.
Adding another SQL cluster to handle increased message box activity is costly.
It’s much better to design a solution that uses the message box in an efficient
way if there it is anticipated there will be a high load on the system.
Conclusions
From the
results it can be seen that the messaging-database aggregator provides a 90%
reduction in message box database load compared with the sequential convoy
aggregator in this scenario. If the message count were to increase, this design
could perform 20 or ever 100 times better than the orchestration based design,
in a production system, this could result in significant cost reductions. Your
database administrator would also appreciate the difference in log file sizes
used for the backups :-p.
I tried to
keep the tests as scientific as possible, and had planned to include execution
times in the results as well. Unfortunately, Virtual PC is not a great platform
for performance testing, and does not seem to give the speed that running on
bare metal would do, so I have not included those figures. The log file sizes
should provide an accurate indication of performance, (and I did remember to
add the log file sizes for the order aggregation database as well ;-).
I think the
option of using a messaging-database aggregator should be considered when
building an aggregator in BizTalk. Apart from the performance advantages, it
also provides the option to recover better from failure, as the messages are
present in a database, and the aggregation can be triggered manually.
There’s a
few improvements that could be made to the design, I’m not happy with creating
a database table that contains the fields in the schema, with a hierarchical
schema, this would involve multiple tables, and references. It would be much
better to save the XML content of the message in one column.
I’ve herd a
few people discussing the option of using a pure messaging based solution to
improve performance on high load systems, and it would be great to see more
examples of this.
To sum it up
-
Sequential Convoy Aggregators
are fine for handling aggregations with a low number of messages. -
Sequential Convoy Aggregators
do not scale well with higher message traffic if the aggregation state is
held within the orchestration. -
Building a messaging-database
aggregator provides a design that scales well with large numbers of messages. -
It is also possible to retain
the sequential convoy orchestration design, and use a database to store
the aggregation state if the need arises.
BizTalk 2004 "Commando" Training course in Santa Monica, CA
Hi Everyone…….well…this is a first! I never thought I’d actually post something to my own Blog. I think I set this up 6 months ago or so, but after seeing all the wonderful content already being posted to blogs, I thought to myself, what could I probably add. However, I think this is going to be a great avenue to distribute all the goodies we cover in our training courses, as well as other things of interest…
Last week Richard Seroter and I tried to to kill about 60 people by putting them through the extended 5 day MOC course. What does that mean? 14 to 16 hours a day for 5 days, first going through the the typical 5 day MOC (Microsoft Official Curriculum) lab training (this was done in about 3 days), followed by about 2+ days of optional deep dive training.
This was trial by fire…..as the air conditioning in the Santa Monica office was something to be desired …. 🙂
During the last 2 days, we had everyone build a fully functional BizTalk 2004 Farm with high security. This consisted of setting up all the users and groups on an Active Directory box, and installing BizTalk 2004 on two servers with a remote SQL Server….4 servers in our farm. We then went thru all the Perf Tuning stuff, deploying an application and seeing for ourselves all the cool things you can do around availablity with Host Instances. You can find the performance white paper here, Performance WhitePaper.
After that I think we installed the MSMQT and MSMQ adapter side by side and got those bad boys running and functional. Then it was off to the following:
1. Using the Business Rules Engine API to automate our deployment and execution policies from C#.
2. Creating some custom code so that we could call BizTalk maps dynamically from C#.
3. Configuring App Domains and using config files from within our BizTalk projects.
4. Briefly going over Convoys, which I’ll post some good working samples….
5. Parsing REAL messy flat files that I got a thrill doing on a project once..
6. Developing a custom pipeline component to copy context properties into our messages…
I’ve set up an FTP site which I’ll be posting many of these materials to, code samples and powerpoints….so stay tuned. Apparently I can’t VPN out when logged onto the Microsoft Network….so off to starbucks I go for unfettered connectivity….
Facts and Slots: XPath Selector and XPath Field
The following is an extract from a thread at msnusers.com/biztalkserverstuff. Charles Young explains with great clarity the subtle differences in the rules engine parsing of facts expressed using XPath Selector and XPath Field combinations (or “facts” and “slots”);
This causes so much confusion.
The rules engine works by asserting ‘facts’ into its working memory. A ‘fact’ is some object, and therefore contains fields, commonly called ‘slots’ in the world of rule engines. The exception is thrown because the rule engine is attempting to access a slot in a fact in order to evaluate a condition within a rule. The engine is attempting to do this evaluation because the relevant fact has been asserted. However, when it looks for the slot, it can’t find it.
This translates into the world of XML as follows. When you create a vocabulary definition for a node in your schema, there are two properties that are set. These are ‘XPath Selector’ and ‘XPath Field’. Thinks of these as the technical terms that refer to some data item. The vocabulary definition is mapping these to business-friendly terms defined by the Name and Display Name properties.
The XPath Selector is used to define and select a ‘fact’. Vocabulary definitions can refer to a fact, rather than a slot, and in this case the XPath Field property is empty. However, in your case, there will be an additional XPath expression in the XPath Field property. This will typically be used to select a descendant node of the fact. The exception is happening because the ‘fact’ (the programId element) exists (i.e., the XML Selector addresses at least one programId element – if it addresses more than one programId element, multiple facts will be asserted into memory), but the ‘slot’ (the id element) does not.
If the programId element did not exist, no error would occur. Very simply, the engine would not be able to assert the programId fact and would therefore realise that it could not evaluate any rule conditions that depends on this fact. Because the programId element exists, the engine assumes that the id child element exists, and throws an error when it finds it doesn’t.
One way to deal with this is to edit your XPath Selector so that it only selects programId elements that contain an id element. XPath support filters, so you could amend your selector as follows (addition shown in bold):
“/*[local-name()=’SOA_Message’ and namespace-uri()=’http://schemas.test.com/20050307/SoaMessageSchema’]/*[local-name()=’products’ and namespace-uri()=”]/*[local-name()=’clients’ and namespace-uri()=”]/*[local-name()=’client’ and namespace-uri()=”]/*[local-name()=’programs’ and namespace-uri()=”]/*[local-name()=’program’ and namespace-uri()=”]/*[local-name()=’programId’ and namespace-uri()=”][id]”
You might want to further improve on this with the following filter [id = “”]. This would only select programId nodes as facts if they contain an id child element with a non-empty text node.
The key to running the rules engine effectively over XML is to understand XPath and the difference between ‘facts’ and ‘slots’, and to edit your XPath Selectors and XPath Fields accordingly to meet your needs.
Rolling Thunder
It’s been a busy week. I’ve been on holiday for only a week and it’s going to take the best part of a day to catch up with voicemails, emails, and blogs via bloglines. Apart from almost breaking my foot in NJ (long story) and the Yankees going down to their biggest defeat at the hands of the Red Sox, it was a nice break.
Some essential catch ups;
- If you haven’t subscribed to the biztalkserverstuff group over at msnusers.com, get going. Scott W has posted a whole load of preview stuff on Biztalk 2006 (Rolling Thunder).
- Rob at synBoogaloo has posted a summary of the new and improved stuff, or you can get the full brief from MS at biztalkserverstuff at msnusers.com
Get ready for TechEd 2005 in Orlando!
Make sure you pick up a copy of BizTalk Server 2006 CTP (Community Technology Preview). Starting next week, I will start a serie of deep dive posts on some of the new BAM (Business Activity Monitoring) features … Staty tuned!