Understanding the BizTalk Mapper: Part 12 – Performance and Maintainability

In this section:
Performance

   Summary of Tests

   Testing performance in isolation
(non-BizTalk)

   Performance Test Results

   Measuring Memory Usage in BizTalk

   BizTalk Memory Test Results

   Byte Arrays

   Analysing the performance results

Maintainability

   External XSLT

   Serializable Classes

   Why is it so difficult
to edit code in the Script functoid?

   Documentation



Any large BizTalk project will likely have had the inevitable conversations about
performance and maintainability: will it be fast/sustainable enough, and will the
tech support team (or whoever looks after the code once the developers have finished)
be able to maintain it?

In this post I want to look at the performance of the Mapper, and also look at how
maintainable maps are generally.

In order to do this, I want look at the different options you have for executing XSLT
with the Mapper, and compare this to the most common non-Mapper mechanism for performing
transformation: using serializable classes.

In BizTalk, your options for transformations using the Mapper are:

• Built-in functoids
• Inline XSLT script
• Inline C# / VB.NET / JScript.NET
• Classes/methods in external assemblies
• Custom functoids
• External XSLT file

Normally you’d use a combination of these in a complex BizTalk solution.

So how do you decide which to use?
Which gives you the best performance?
Which option(s) are the easiest to maintain?

The answer is: it depends!

In this section I’m going to try and give you some hard data you can use to try and
answer these questions. In the next post, I’ll try and answer the questions.

Performance

Performance is a very subjective subject. You could spend weeks getting your maps
to execute in under 10ms, but if your performance requirement is "anything less
than 100ms" then why bother?

What’s more important is that performance is "good enough" and is sustainable.

Sustainable performance is the key – its one thing to be able to perform a single
transformation in less than 10ms, but what about performing 40 simultaneous transformations
continuously for over an hour?
And what about the memory footprint?

When we talk about performance, we also need to look at memory/resource usage: If
you have a very fast system that for each transformation uses 1MB memory, then at
a certain point under sustained load your memory usage will start to affect performance
as memory is paged out to disk (assuming that garbage collection can’t keep up with
the number of transformations you’re performing).

In order to find out the performance and memory footprints of the various options,
I put together some tests.

I ran two suites of tests:

• Using a stand-alone test harness with the XslTransform class
  to perform transformation in isolation (this gives the comparative performance)
• Using a BizTalk application to measure memory usage (this gives the comparative memory
  usage)

Summary of Tests

I simplified the six options above into four separate XSLT tests, and then added two
tests involving serializable classes for comparison:

1. Standard Map
   
   A map using default built-in functoids, with a mix of functoids that emit XSLT and
   functoids that emit C#
2. Map using external XSLT
   
   A map which used an external XSLT file consisting of pure XSLT i.e. no external assemblies
   or inline script
3. Map using inline C#
   
   A map using inline C# code in a Script functoid
4. Map using external assembly
   
   A map using an external assembly via a Script functoid
5. Transformation using Classes
   
   De-serializing the input message into a class, transforming to a new class, and serializing
   the class back into the output message
6. Transformation using classes and serializers
   
   Same as 5. above but using the sgen.exe tool
   to pre-generate a Serialization class

My test scenario was: Transforming a message containing 20 employee records into a
new message which contained separate Manager and Staff records:
i.e. this:


becomes this:

Each of the maps performed the transformation in the same way, as much as was possible.

Testing performance in isolation (non-BizTalk)

For the XSLT tests, I tested using both the XslTransform class
(used by BizTalk) and the newer XslCompiledTransform class
(for comparison).

For both of these class tests I used static and non-static instances of the classes:

• In the static test a transform class was created once and then re-used for each iteration
• In the non-static test a new transform class was created for each iteration

I ran each test twice: once with 20 iterations, and once with 500 iterations.

I also measured the amount of memory in use before and after each test – from this
I calculated a rough "memory used" for each test (this is before garbage
collection kicked in).

For each test I show:

1. Total
   
   This is the total time (in ms) that the test took to run
2. Average
   
   This is the average time for each iteration
3. Average without first
   
   This is the average time without the first iteration
4. Average setup
   
   This is the average time it took to create the transform object in each iteration
   (will be 0 for the static tests as creation of transform is performed once and not
   included in the timings)
5. Average transform
   
   This is the average time it took to perform the transformation
6. Memory Used
   
   This is the amount of memory used to perform the entire test

Performance Test Results

The results of the 500 iteration tests are:

	Standard Map	Map with External Assembly	Map with Inline Script	Map with External XSLT	Class Transform	Class Transform with Serializer
Iterations	500	500	500	500	500	500
XslTransform – NonStatic
Total (ms)	82402	78815	79507	1622	1749	728
Average (ms)	164	157	159	3	3	1
Average without first (ms)	164	157	158	3	2	1
Average setup (ms)	151	151	152	1	1	0
Average transform (ms)	12	5	5	1	1	1
Memory Used	504MB	452MB	456MB	62MB	90MB	86MB
XslTransform – Static
Total (ms)	4392	2234	1564	720	769	735
Average (ms)	8	4	3	1	1	1
Average without first (ms)	8	4	3	1	1	1
Average setup (ms)	0	0	0	0	0	0
Average transform (ms)	8	4	3	1	1	1
Memory Used	64MB	40MB	40MB	30MB	88MB	88MB
XslCompiledTransform – NonStatic
Total (ms)	101214	88181	89608	16325
Average (ms)	202	176	179	32
Average without first (ms)	202	176	179	32
Average setup (ms)	156	145	147	3
Average transform (ms)	45	30	30	28
Memory Used	192MB	141MB	145MB	82MB
XslCompiledTransform – Static
Total (ms)	95	131	70	55
Average (ms)	0	0	0	0
Average without first (ms)	0	0	0	0
Average setup (ms)	0	0	0	0
Average transform (ms)	0	0	0	0
Memory Used	14MB	15MB	13MB	13MB

(the lowest result on each row is highlighted in green)

And for comparison, the results from the 20 iteration tests:

	Standard Map	Map with External Assembly	Map with Inline Script	Map with External XSLT	Class Transform	Class Transform with Serializer
Iterations	20	20	20	20	20	20
XslTransform – NonStatic
Total (ms)	3820	3931	3718	78	1213	82
Average (ms)	191	196	185	3	60	4
Average without first (ms)	171	194	184	3	4	1
Average setup (ms)	175	189	178	1	56	1
Average transform (ms)	15	7	6	1	4	2
Memory Used	20MB	18MB	17MB	2MB	4MB	3MB
XslTransform – Static
Total (ms)	169	100	66	29	122	83
Average (ms)	8	5	3	1	6	4
Average without first (ms)	8	4	3	1	2	1
Average setup (ms)	0	0	0	0	2	1
Average transform (ms)	8	4	3	1	3	2
Memory Used	4MB	1MB	1MB	1MB	3MB	3MB
XslCompiledTransform – NonStatic
Total (ms)	4628	3758	5094	1190
Average (ms)	231	187	254	59
Average without first (ms)	224	183	254	50
Average setup (ms)	180	154	211	11
Average transform (ms)	50	32	42	48
Memory Used	8MB	6MB	6MB	3MB
XslCompiledTransform – Static
Total (ms)	46	39	67	81
Average (ms)	2	1	3	4
Average without first (ms)	0	0	1	1
Average setup (ms)	0	0	0	0
Average transform (ms)	2	1	3	4
Memory Used	642KB	647KB	583KB	542KB

(the lowest result on each row is highlighted in green)

Measuring Memory Usage in BizTalk

Although the Memory Used amount from the performance tests was useful, I wanted
to know exactly how much memory BizTalk used for performing transformations – and
what objects were in memory.
In order to measure this I used SciTech Software’s .NET
Memory Profiler.
This tool attaches to the BizTalk service (BTSNTSvc.exe) and can create a snapshot
of all the object instances currently in use, including how many there are and how
much memory they’re using.

I created a BizTalk application which contained a separate map for each of the tests
above, and created orchestrations to execute the maps (and to call the C# code to
perform the transform using the classes).

I performed a memory snapshot before and after running the orchestrations, and restarted
the BizTalk service between each test.

I ran each test twice: once with a single message, and once with 50 messages.

For each test I show:

1. Byte[] Instances
   
   This is the count of Byte arrays in use (the relevance of Byte arrays is explained
   below)
2. Byte[] Instances Size (MB)
   
   The is the total size of all Byte arrays (i.e. the size of all the data they contain)
3. Total Instances
   
   The is the count of all .NET objects in use
4. Total Instances Size(MB)
   
   This is the total size of all .NET objects in use

Note that any sizes measured are after garbage collection has occurred i.e.
these are objects which are still classed as being in-use.

BizTalk Memory Test Results

The results I measured were:
Single Message:

Test – 1 Iteration	Byte[] Instances	Byte[] Instances Size (MB)	Total Instances	Total Instances Size (MB)
Standard	5,652	0.63	17,277	1.78
External XSLT	5,752	0.59	19,038	2.11
Inline Script	5,763	0.63	20,522	2.23
Referenced Assembly	5,757	1.25	19,897	2.42
Class	30	0.03	7,476	0.74

(the lowest result in each column is highlighted in green)

50 Messages:

Test – 50 Iterations	Byte[] Instances	Byte[] Instances Size (MB)	Total Instances	Total Instances Size (MB)
Standard	146	1.63	18,354	3.24
External XSLT	126	0.80	14,359	1.87
Inline Script	5,780	1.89	22,590	3.13
Referenced Assembly	5,757	1.36	20,100	3.00
Class	146	0.85	14,202	1.81

(the lowest result in each column is highlighted in green)

Byte Arrays

I can let Microsoft explain this in their own words (this is taken from a knowledge
base article here):

The System.Policy.Security.Evidence object is often used
in transforms and can consume a lot of memory. Whenever a map contains a scripting
functoid that uses inline C# (or any other inline language), the assembly is created
in memory. The System.Policy.Security.Evidence object uses the object of the actual
calling assembly. This situation creates a rooted object that is not deleted until
the BizTalk service is restarted.

 

Most of the default BizTalk functoids are implemented
as inline script. These items can cause System.Byte[] objects to collect in memory.
To minimize memory consumption, we recommend that you put any map that uses these
functoids into a small assembly. Then, reference that assembly. Use the chart below
to determine which functoids use inline script and which functoids do not use inline
script.

In the second column, “Yes” means that this functoid is
implemented as inline script, and that it will cause System.Byte[] objects to collect
in memory. “No” means that this functoid is not implemented as inline script, and
that it will not cause System.Byte[] objects to collect in memory.

Functoids	Inline script?
All String Functoids	Yes
All Mathematical Functoids	Yes
All Logical Functoids except IsNil	Yes
Logical IsNil Functoid	No
All Date/Time Functoids	Yes
All Conversion Functoids	Yes
All Scientific Functoids	Yes
All Cumulative Functoids	Yes
All Database Functoids	No
All Advanced Functoids (apart from Script functoids using Inline C#/VB/Jscript)	No

Assemblies created by compiling inline script in an XSLT are temporary assemblies
and are loaded into the appropriate AppDomain – they will remain in memory until the
AppDomain is unloaded i.e. the BizTalk Host Instance is restarted.

So if you keep all your maps, orchestrations, schemas, etc in one big assembly, then
that assembly will stay loaded in memory until the BizTalk Host Instance that loaded
it is restarted.
Solution? Keep your maps (especially maps using inline C#) in a separate project/assembly
– and try and keep that assembly as small as possible!

Analysing the performance results

The results shouldn’t really come as any surprise.
What they say in a nutshell is: pure XSLT is much faster than XSLT which uses inline
script or referenced assemblies. How much faster? Well, my tests show an average 5500%
speed increase over using the default functoids (i.e. 55 times faster)!
Additionally, using pure XSLT uses 1/8 the memory.
Of course, your mileage will vary.

What’s interesting though is how fast using serializable classes is i.e. de-serializing
a message into a class, performing operations to create a new class, and then serializing
the new class into a message.
When used with a pre-generated serialization assembly, this mechanism chases closely
behind using pure XSLT (and actually beat it in one of the tests).

Maintainability

One of BizTalk’s trump cards is the BizTalk Mapper: you can create maps which can
be easily maintained – what’s more, because the Mapper is a visual tool you can see
at a glance how the mapping works.
At least, that’s the theory.
If you have a relatively simple map which uses no script functoids, and has less then,
say, 50 connections then Yes, I’d say this was true: it’s easy to see what the map
does, and probably easy to maintain it.

But if you have a map with 1000 connections, or with a whole smattering of Script
functoids (or a bird’s nest of Logical functoids) then No, I don’t think it’s easy
to see what the map does or to maintain it.

At what point do you have to admit defeat with a Map and say that it’s got as bit
too complex? or that the next developer to come along will have problems maintaining
it?
In that case, would you be better off using external XSLT or serializable classes?

External XSLT

One of the main complaints I hear about using external XSLT with the Mapper is that
it’s difficult to maintain. This can be true – if your editing tool is notepad! But
there are great tools around for maintaining XSLT – have a look at Altova’s
MapForce for an example of one.
The other complaint is that XSLT is difficult, or hard to learn. Well, so is C# if
you’ve never used it before.
Go buy a book on it, or do a course!

Truth be told, if you work for a company which uses XSLT for other projects, then
you’re more likely to find support for using it as external XSLT in BizTalk.
Some companies have teams in IT which do nothing but create XSLT.

My point here is that although there’s a myth that external XSLT is hard to maintain,
it’s a lot easier to maintain than a complicated map. And if you use the right tool,
you can get a graphical view of what it does as well.

Serializable Classes

In my experience, it’s very very common for developers to create a whole slew of transform
and utility classes for handling transformations.
Sometimes this is because it’s the best way to do it.
Other times it’s because they simply didn’t know how to achieve something in the Mapper.
One of the best features of BizTalk (the ability to call out to C# classes/methods
from an orchestration or map) can also be its worst: Just because you *can* create
C# code, doesn’t mean that you *should*.

Maintaining poorly written C# code is a nightmare.
So if you’re using serializable classes to perform transformations make sure they’re
well written and well documented – but most importantly: understand *why* you’re using
them over XSLT.

Why is it so difficult to edit code
in the Script functoid?

Ever wondered why you can’t resize the Script functoid code window? Or do a Ctrl-A
to select all the code in it? It turns out there is actually a reason for it…

Scott Woodgate (former Product Manager for BizTalk) had
this to say…
The article is correct when it points out we discourage the use of .NET objects
directly inside the map. While this is possible, we encourage good developer design
which is encapsulating code in external assemblies. This turns out to be much better
because you have a single assembly with code shared across multiple maps that can
be versioned once and managed more easily

Unfortunately, this restriction also means that it’s difficult to use inline XSLT,
which is a shame.

Documentation

You can’t get away from the fact that code that is easy to maintain is either self-documenting,
or is accompanied by excellent documentation.
Regardless if you use maps, external XSLT, or serializable classes you should really
document your transforms: explain what they do, how they do it, why they do it – and
most importantly, give some context: explain why you chose to do it that particular
way.
A developer who has to maintain your code in 2 years time might not have your background
of development and political issues to understand why you made your choices.

The next post is going to look at the performance/maintainability of the different
transformation options and attempt to help you to decide when to use which option.