by community-syndication | Jun 23, 2010 | BizTalk Community Blogs via Syndication
Introduction
This is the first in a series of articles that will introduce and explore a couple a patterns to transfer and process large messages using WCF-Adapters. Generally speaking, an end to end BizTalk solution adopts the FTP Adapter or one of its variations (SFTP/FTPS) when it needs to exchange large messages with external parties over the internet, while it uses the FILE adapter for moving large batch files over the intranet. Large input files can be debatched by a receive pipeline into smaller discrete messages. This technique removes the need to worry about a large message and enables processing to be parallelized. The main recommendation to bear in mind when dealing with large messages might seem obvious, but it is often overlooked by developers: you must avoid any operations that require the entire message to be loaded into memory and adopt a streaming approach, especially in those cases where the application can concurrently receive and process multiple large messages in the same time. BizTalk itself adopts a streaming approach whereby the message is read through the use of a stream, meaning that the entire message doesn’t have to be present in memory at any point in time. This streaming approach is used by all the built-in pipeline components such as the Flat File Disassembler, XML Disassembler, etc. Therefore, it’s extremely important that you adopt the same approach when designing and developing custom pipeline components, orchestrations or helper classes for processing large messages.
The problems associated with large messages can be divided into the following categories:
-
Out of memory errors: Certain types of message processing can require the entire message to be loaded into memory. BizTalk Server operates a streaming approach whereby the message is read through the use of a stream, meaning that the entire message doesn’t have to be present in memory at any point in time. For example, BizTalk Server 2006 introduced the VirtualStream class in the Messaging Runtime and if the message size is over a certain, configurable threshold (1MB is the default), the document is written to a temporary file, thus keeping the memory usage flat during mapping. Of course, this swapping incurs performance overhead, but this prevents the host process from running out of memory. Any custom pipeline components, orchestrations, or helper classes should follow the same streaming approach and avoid using an XmlDocument to load the entire message into memory.
-
Performance problems for messages that are not loaded into memory: Messages that are not required to be loaded into memory are streamed to the MessageBox database using an .NET XmlReader object. While they are not subject to any size limitations, there are some important factors that impact how BizTalk Server processes messages that are streamed to the MessageBox database.
The original message size, message format, and type of message processing are the main factors that affect how BizTalk Server processes large messages.
-
Original message size: The size of the message received by BizTalk Server is the most visible indication of how large the message will be when it is processed by BizTalk Server. The original size of a message has a much greater impact on performance if the entire message is loaded into memory than if the message is streamed to the MessageBox database.
-
Message format: Messages are received into BizTalk Server in one of two primary formats: XML files or flat files.
-
XML files: In order for BizTalk Server to perform any processing on a message other than pass through routing, the message must be in the XML file format. If the files to be processed are received in XML format then they will retain their original size for the most part.
-
Flat files: Flat files must be parsed into an XML format before BizTalk Server can perform any processing (for instance message transformation) other than pass through routing. Parsing a flat file into an XML file can significantly increase the size of the file. Flat files do not contain XML tags with descriptive information about their data. By contrast, XML files wrap all of their data in descriptive XML tags. In some scenarios parsing can increase the size of a flat file by a factor of 10 or more, depending on how much descriptive data is contained in the XML tags for the file.
-
Type of message processing: In BizTalk Server, there are two types of message processing: Routing only and Mapping. The performance variables tied to the type of message processing that is performed are message size and whether the message is loaded into memory.
-
Routing: If BizTalk Server is only used only for routing messages based upon promoted properties, then the Message Agent uses the stream opened by the receive adapter to publish the message to the BizTalkMsgBoxDb database using the .NET XmlReader interface, and message parts are not individually loaded into memory. In this scenario, out of memory errors are not an issue and the primary consideration is the amount of time that is required to write very large messages (over 100 MB) into the BizTalkMsgBoxDb database. In fact, this operation can have a severe impact on overall performance, especially when the same BizTalk installation is used to host also low latency applications. The BizTalk Server development team has successfully tested the processing of messages up to 1 GB in size when performing routing only.
-
Mapping: Transforming a document with a map is a memory-intensive operation. When a document is transformed by a map, BizTalk Server passes the message stream to the .Net
XslTransform class, which then loads the document into a .NET
XPathDocument object for processing. Loading the document into the .NET
XPathDocument can potentially expand the original file size in memory by a factor of 10 or more. This expansion may be more pronounced when mapping flat files because flat files must be parsed into XML before they can be transformed. Almost 4 months ago I wrote an
article on how boosting message transformation using the
XslCompiledTransform class in place of the
XslTransform.
For more information about large message transfer and processing with BizTalk Server, you can read the following topics:
Problem Statement
In this article we’ll try to provide an answer to the following questions:
-
How can I use BizTalk and WCF to transfer a large message in a reliable and performant way?
-
What is the best Message Encoder (Binary, MTOM, Text) and Transfer Mode (Buffered, Streamed) that I can use to transmit a large message to a WCF Receive Location?
Besides, in this series of articles I’ll discuss the following topics:
-
How to use the
InboundHeaders context property exposed by WCF Adapters to access the SOAP headers of incoming WCF messages.
-
How to use the TransferMode = Streamed.
-
How to process in a streaming fashion within a pipeline component.
-
How to transfer a large message in a transactional way using a client-side initiated transaction.
-
How to stage the incoming document to a folder or SQL.
-
How to use a table with a column of type “varbinary(max) FILESTREAM” to stage a large file on SQL Server 2008.
-
How to use the API of the Transaction NTFS (TxF) to write a large file to a shared folder in reliable, transactional way.
-
How to hook the distributed transaction used by the Message Agent within a pipeline component.
-
How to use a custom stream on the send port to wrap the original stream and execute closing tasks after message transmission.
-
How to initiate delivery notifications within a pipeline component and handle ACK/NACK messages within an orchestration.
WCF Message Encoders and Transfer Modes
When I started to design this demo, my will was comparing the performance of the various message encoders and transfers modes provided out of the box by WCF and find the best combination to adopt when transferring large message over the network. WCF includes three types of encoding for SOAP messages:
- Text Message Encoding: The text encoding represented by the TextMessageEncodingBindingElement is the most interoperable, but the least efficient encoder for XML messages. Web service or Web service client can generally understand textual XML. However, transmitting large blocks of binary data as text is not efficient.
- Binary Message Encoding: The BinaryMessageEncodingBindingElement is the binding element that specified the .NET Binary Format for XML should be used for encoding messages, and has options to specify the character encoding and the SOAP and WS-Addressing version to be used. Binary encoding is most efficient but least interoperable of the encoding options.
- MTOM Message Encoding: The MTOMMessageEncodingBindingElement represents the binding element that specifies the character encoding and message versioning and other settings used for messages using a Message Transmission Optimization Mechanism (MTOM) encoding. (MTOM) is an efficient technology for transmitting binary data in WCF messages. The MTOM encoder attempts to create a balance between efficiency and interoperability. The MTOM encoding transmits most XML in textual form, but optimizes large blocks of binary data by transmitting them as-is, without conversion to their base64 encoded format.
WCF transport channels support two modes for transferring messages in each direction:
- Buffered: transmissions hold the entire message in a memory buffer until the transfer is complete. On the service side message processing cannot start until the entire message has been received.
- Streamed: transmissions only buffer the message headers and expose the message body as a stream, from which smaller portions can be read at a time.
The TransferMode property exposed by transport protocol channel (e.g.HttpTransportBindingElement, TcpTransportBindingElement, etc.) and bindings (BasicHttpBinding, NetTcpBinding, etc.) allows to indicate whether messages are sent buffered or streamed. Streamed transfers can improve the scalability of a service by eliminating the need for large memory buffers. Whether changing the transfer mode actually improves scalability in practice depends on the size of the messages being transferred. Improvements in scalability should be most evident when large messages use streamed instead of buffered transfers. Therefore we’ll exploit the streamed transfer mode in for transferring large messages using BizTalk and WCF Adapters.
Note: By default, the HTTP, TCP/IP and Named Pipe transports use buffered message transfers. If you plan to exploit the streamed transfer mode when receiving a large message through a WCF Receive Location, you have to use the WCF-Custom/WCF-CustomIsolated Adapter. In fact, these latter allow developer to select a certain binding (e.g. BasicHttpBinding, NetTcpBinding, NetNamedPipeBinding, etc.) and then change the value of its TransferMode property to Streamed. The other WCF Adapters (e.g. WCF-BasicHttp, WCF-NetTcp, etc.) do not provide this possibility.
For more information on the WCF Transfer Modes, you can read the following topic:
-
“TransferMode Enumeration” on
MSDN.
Client Application
I created a WinForm application that is capable to upload multiple documents at the same time. The UI of the driver application allows to select a list of files to be sent in parallel to BizTalk and which service endpoint to use for this purpose. Six different WCF Receive Locations have been created, one for each possible combination of message encoders and transfer modes. Therefore, the configuration file of the client application contains a different service endpoint, one for each WCF Receive Location. When you push the Submit button on the main form, the click event handler method (btnSubmit_Click) creates a a list of calls and then using the ThreadPool it start an asynchronous thread for each file selected on the UI. At this point, if the selected service endpoint uses a transaction-enabled binding, the application will call the SubmitFileWithTxn method, otherwise the SubmitFileWithoutTxn method.
The code of the 2 functions is quite similar, the only difference between the two is that the SubmitFileWithTxn method initiates a transaction that will be flowed to the WCF Receive Location. Let’s analyze the code of this latter method. The SubmitFileWithTxn starts opening a FileStream to read the selected file from the disk and I wrap this latter with a custom stream called ProgressStream. As the name suggests, the main scope of this custom stream is to update a progress bar on the UI as the file is being transmitted. This feature is particularly interesting when we use a binding which TransferMode is equal to Streamed. The ProgressStream class exposes an event called ProgressChanged that can be used to invoke a method that updates the progress bar. Since the progress bar control is owned by the main thread, while the application uses a ThreadPool worker thread to transmit the message to BizTalk, in order to change the value of the progress bar is necessary to call the Invoke method exposed by the ProgressForm object with a delegate. This technique enables the worker thread to ask the main thread to execute the delegate.
Then, the method creates an UploadMessage. If you look at the code of the class, you will note that I decorated the class with the MessageContractAttribute. Then I decorated all the properties representing transmission metadata like the name of the sender or the file size with the MessageHeaderAttribute attribute. This way I declare that this data will be transmitted as custom headers in the SOAP envelope.
Instead, the property called Data that will contain the content of the source file stream has been decorated with the MessageBodyMember attribute. Hence data will be transmitted within the body of the SOAP envelope.
Finally, the method creates a ChannelFactory using the IUploadMessageTxn interface as service contract. If you look at the code, you will note that this interface has been decorated with the TransactionFlowAttribute. In particular, the TransactionFlowOption.Mandatory value indicates that the service endpoint requires the client application to start and flow a distributed transaction.
For this reason, the SubmitFileWithTxn method invoke the underlying WCF Receive Location within a transaction scope. The driver application allows to control the outcome of the transaction selecting one between the Commit or Abort option buttons on the UI. Selecting abort, you can simulate a transaction failure. This possibility is extremely useful when you configure the WCF receive location to use a transactional binding. In fact, in this case the Message Agent will use the transaction initiated by the client application to post a placeholder document to the BizTalkMsgBoxDb (as you will see in the remainder of the article, the original message will be staged in a folder or SQL database and replaced by a placeholder document which contains its location and metadata). Therefore, if the client application aborts the transaction, no message will be published to the BizTalkMsgBoxDb. Transferring a large message in the context of WS-AT or OleTransactions transaction greatly improves transmission reliability, but it has a severe impact on overall performance, so this technique is recommended only in those cases where resilience is the primary concern.
For sake of readability, I omitted most of the code and I reported below only the classes and methods that can be helpful to understand the application.
MainForm class
#region Copyright
//-------------------------------------------------
// Author: Paolo Salvatori
// Email: paolos@microsoft.com
// History: 2009-01-14 Created
//-------------------------------------------------
#endregion
#region Using Directives
using System;
using System.IO;
using System.Collections.Generic;
using System.Configuration;
using System.ComponentModel;
using System.Transactions;
using System.Data;
using System.Drawing;
using System.Text;
using System.Xml;
using System.Xml.XPath;
using System.Threading;
using System.Windows.Forms;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.ServiceModel.Configuration;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client
{
public partial class MainForm : Form
{
...
private void btnSubmit_Click(object sender, EventArgs e)
{
try
{
if (shellView.SelectedItems != null &&
shellView.SelectedItems.Length > 0)
{
FileInfo fileInfo = null;
List<CallParameters> callList = new List<CallParameters>();
for (int i = 0; i < shellView.SelectedItems.Length; i++)
{
if (File.Exists(shellView.SelectedItems[i].FileSystemPath))
{
fileInfo = new FileInfo(shellView.SelectedItems[i].FileSystemPath);
callList.Add(new CallParameters(i,
(int)fileInfo.Length,
shellView.SelectedItems[i].FileSystemPath,
cboEndpoint.Text));
}
else
{
WriteToLog(string.Format(FileDoesNotExist, shellView.SelectedItems[i].FileSystemPath));
}
}
count = callList.Count;
completed = 0;
progressForm = new ProgressForm(callList);
progressForm.Show();
for (int i = 0; i < shellView.SelectedItems.Length; i++)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(SubmitFile), new StartState(callList[i], this));
}
}
else
{
WriteToLog(NoFilesSelected);
}
}
catch (Exception ex)
{
WriteToLog(ex.Message);
}
}
private void SubmitFile(Object stateInfo)
{
StartState state = stateInfo as StartState;
if (TxnBinding.ContainsKey(state.Parameters.Endpoint) &&
TxnBinding[state.Parameters.Endpoint])
{
SubmitFileWithTxn(state);
}
else
{
SubmitFileWithoutTxn(state);
}
}
private void SubmitFileWithTxn(StartState startState)
{
ChannelFactory<IUploadMessageTxn> channelFactory = null;
try
{
if (startState != null &&
startState.Parameters != null &&
!string.IsNullOrEmpty(startState.Parameters.Endpoint) &&
!string.IsNullOrEmpty(startState.Parameters.Filename) &&
File.Exists(startState.Parameters.Filename))
{
FileInfo fileInfo = new FileInfo(startState.Parameters.Filename);
using (FileStream fileStream = File.Open(startState.Parameters.Filename,
FileMode.Open,
FileAccess.Read,
FileShare.Read))
{
ProgressStream progressStream = new ProgressStream(startState.Parameters.Index, fileStream);
progressStream.ProgressChanged +=
new EventHandler<ProgressChangedEventArgs>(progressForm.ProgressStream_ProgressChanged);
UploadMessage uploadMessage = new UploadMessage(Guid.NewGuid().ToString(),
txtSender.Text,
fileInfo.Name,
DateTime.Now,
fileInfo.Length,
progressStream);
channelFactory = new ChannelFactory<IUploadMessageTxn>(startState.Parameters.Endpoint);
channelFactory.Endpoint.Contract.SessionMode = SessionMode.Allowed;
IUploadMessageTxn channel = channelFactory.CreateChannel();
DateTime start = DateTime.Now;
bool committed = true;
string transactionId = string.Empty;
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required))
{
channel.UploadMessageTxn(uploadMessage);
transactionId = Transaction.Current.TransactionInformation.DistributedIdentifier.ToString();
WriteToLog(string.Format(TransactionIdFormat, transactionId));
if (commitRadioButton.Checked)
{
scope.Complete();
}
else
{
committed = false;
Thread.Sleep(200);
}
}
DateTime end = DateTime.Now;
TimeSpan time = end.Subtract(start);
channelFactory.Close();
double seconds = time.TotalSeconds;
WriteToLog(string.Format(MessageSuccessfullyUploaded, fileInfo.Name,
startState.Parameters.Endpoint, seconds));
if (committed)
{
WriteToLog(string.Format(TransactionCommittedFormat, transactionId));
}
else
{
WriteToLog(string.Format(TransactionAbortedFormat, transactionId));
}
}
}
else
{
WriteToLog(string.Format(FileDoesNotExist, startState.Parameters.Filename));
}
}
catch (FaultException ex)
{
WriteToLog(ex.Message);
if (channelFactory != null)
{
channelFactory.Abort();
}
}
catch (CommunicationException ex)
{
WriteToLog(ex.Message);
if (channelFactory != null)
{
channelFactory.Abort();
}
}
catch (TimeoutException ex)
{
WriteToLog(ex.Message);
if (channelFactory != null)
{
channelFactory.Abort();
}
}
catch (Exception ex)
{
WriteToLog(ex.Message);
if (channelFactory != null)
{
channelFactory.Abort();
}
}
finally
{
try
{
if (startState.Form != null)
{
startState.Form.ThreadTerminated();
}
if (channelFactory != null)
{
channelFactory.Close();
}
}
catch (Exception)
{
}
}
}
...
}
}
|
ProgressStream class
#region Copyright
//-------------------------------------------------
// Author: Paolo Salvatori
// Email: paolos@microsoft.com
// History: 2009-06-30 Created
//-------------------------------------------------
#endregion
#region Using Directives
using System;
using System.IO;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client
{
public class ProgressStream : Stream
{
#region Private Fields
private Stream stream;
private int index;
private long bytesRead;
private long totalLength;
#endregion
#region Public Handler
public event EventHandler<ProgressChangedEventArgs> ProgressChanged;
#endregion
#region Public Constructor
public ProgressStream(int index, Stream file)
{
this.index = index;
this.stream = file;
this.totalLength = file.Length;
this.bytesRead = 0;
}
#endregion
#region Public Properties
public override bool CanRead
{
get
{
return this.stream.CanRead;
}
}
public override bool CanSeek
{
get
{
return this.stream.CanSeek;
}
}
public override bool CanWrite
{
get
{
return this.stream.CanWrite;
}
}
public override void Flush()
{
this.stream.Flush();
}
public override long Length
{
get
{
return this.stream.Length;
}
}
public override long Position
{
get
{
return this.stream.Position;
}
set
{
this.stream.Position = value;
}
}
#endregion
#region Public Methods
public override int Read(byte[] buffer, int offset, int count)
{
int result = stream.Read(buffer, offset, count);
bytesRead += result;
if (ProgressChanged != null)
{
try
{
ProgressChanged(this, new ProgressChangedEventArgs(index, bytesRead, totalLength));
}
catch (Exception)
{
ProgressChanged = null;
}
}
return result;
}
public override long Seek(long offset, SeekOrigin origin)
{
return this.stream.Seek(offset, origin);
}
public override void SetLength(long value)
{
this.stream.SetLength(value);
}
public override void Write(byte[] buffer, int offset, int count)
{
this.stream.Write(buffer, offset, count);
}
#endregion
}
public class ProgressChangedEventArgs : EventArgs
{
#region Private Fields
private int index;
private long bytesRead;
private long totalLength;
#endregion
#region Public Constructor
public ProgressChangedEventArgs(int index, long bytesRead, long totalLength)
{
this.index = index;
this.bytesRead = bytesRead;
this.totalLength = totalLength;
}
#endregion
#region Public properties
public int Index
{
get
{
return this.index;
}
set
{
this.index = value;
}
}
public long BytesRead
{
get
{
return this.bytesRead;
}
set
{
this.bytesRead = value;
}
}
public long TotalLength
{
get
{
return this.totalLength;
}
set
{
this.totalLength = value;
}
}
#endregion
}
}
|
IUploadMessageTxn interface
#region Copyright
//-------------------------------------------------
// Author: Paolo Salvatori
// Email: paolos@microsoft.com
// History: 2009-01-14 Created
//-------------------------------------------------
#endregion
#region Using Directives
using System;
using System.IO;
using System.ServiceModel;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client
{
[ServiceContract(Namespace = "http://microsoft.biztalk.cat/10/samples/ssisintegration/uploadservice")]
public interface IUploadMessageTxn
{
#region Contract Operations
[OperationContract(Action = "UploadMessageTxn", ReplyAction = "*")]
[TransactionFlow(TransactionFlowOption.Mandatory)]
[XmlSerializerFormat()]
void UploadMessageTxn(UploadMessage message);
#endregion
}
}
|
UploadMessage class
#region Copyright
//-------------------------------------------------
// Author: Paolo Salvatori
// Email: paolos@microsoft.com
// History: 2009-01-14 Created
//-------------------------------------------------
#endregion
#region Using Directives
using System;
using System.IO;
using System.ServiceModel;
using System.Net.Security;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client
{
[MessageContract]
public class UploadMessage
{
#region Private Fields
private string id;
private string sender;
private string filename;
private DateTime dateTime;
private long size;
private Stream data;
#endregion
#region Public Constructors
public UploadMessage()
{
this.id = null;
this.sender = null;
this.filename = null;
this.dateTime = DateTime.Now;
this.size= 0;
this.data = null;
}
public UploadMessage(string id,
string sender,
string filename,
DateTime dateTime,
long size,
Stream data)
{
this.id = id;
this.sender = sender;
this.filename = filename;
this.dateTime = dateTime;
this.size = size;
this.data = data;
}
#endregion
#region Public Properties
[MessageHeader(Name = "id",
Namespace = "http://microsoft.biztalk.cat/10/samples/ssisintegration/uploadservice",
ProtectionLevel = ProtectionLevel.None)]
public string Id
{
get
{
return this.id;
}
set
{
this.id = value;
}
}
[MessageHeader(Name = "sender",
Namespace = "http://microsoft.biztalk.cat/10/samples/ssisintegration/uploadservice",
ProtectionLevel = ProtectionLevel.None)]
public string Sender
{
get
{
return this.sender;
}
set
{
this.sender = value;
}
}
[MessageHeader(Name = "filename",
Namespace = "http://microsoft.biztalk.cat/10/samples/ssisintegration/uploadservice",
ProtectionLevel = ProtectionLevel.None)]
public string Filename
{
get
{
return this.filename;
}
set
{
this.filename = value;
}
}
[MessageHeader(Name = "dateTime",
Namespace = "http://microsoft.biztalk.cat/10/samples/ssisintegration/uploadservice",
ProtectionLevel = ProtectionLevel.None)]
public DateTime DateTime
{
get
{
return this.dateTime;
}
set
{
this.dateTime = value;
}
}
[MessageHeader(Name = "size",
Namespace = "http://microsoft.biztalk.cat/10/samples/ssisintegration/uploadservice",
ProtectionLevel = ProtectionLevel.None)]
public long Size
{
get
{
return this.size;
}
set
{
this.size = value;
}
}
[MessageBodyMember(Name = "data",
Namespace = "http://microsoft.biztalk.cat/10/samples/ssisintegration/uploadservice",
ProtectionLevel = ProtectionLevel.None)]
public Stream Data
{
get
{
return this.data;
}
set
{
this.data = value;
}
}
#endregion
}
}
|
App.Config
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<system.diagnostics>
<sources>
<source name="System.ServiceModel.MessageLogging" switchValue="Warning, ActivityTracing">
<listeners>
<add type="System.Diagnostics.DefaultTraceListener" name="Default">
<filter type="" />
</add>
<add name="ServiceModelMessageLoggingListener">
<filter type="" />
</add>
</listeners>
</source>
</sources>
<sharedListeners>
<add initializeData="C:\ssisintegration_client.svclog"
type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089"
name="ServiceModelMessageLoggingListener"
traceOutputOptions="Timestamp">
<filter type="" />
</add>
</sharedListeners>
</system.diagnostics>
<system.serviceModel>
<diagnostics>
<messageLogging logEntireMessage="false"
logMalformedMessages="false"
logMessagesAtServiceLevel="false"
logMessagesAtTransportLevel="false"
maxSizeOfMessageToLog="1000000" />
</diagnostics>
<bindings>
<customBinding>
<binding name="netTcpMtomStreamedBinding"
receiveTimeout="01:00:00"
sendTimeout="01:00:00">
<mtomMessageEncoding maxBufferSize="1048576"
maxReadPoolSize="1024"
maxWritePoolSize="1024">
<readerQuotas maxDepth="1048576"
maxStringContentLength="1048576"
maxArrayLength="104857600"
maxBytesPerRead="1048576"
maxNameTableCharCount="1048576" />
</mtomMessageEncoding>
<tcpTransport maxBufferPoolSize="1048576"
maxReceivedMessageSize="104857600"
connectionBufferSize="1048576"
maxBufferSize="1048576"
maxPendingConnections="200"
maxPendingAccepts="200"
transferMode="Streamed"
listenBacklog="200">
<connectionPoolSettings maxOutboundConnectionsPerEndpoint="200" />
</tcpTransport>
</binding>
<binding name="netTcpBinaryStreamedBinding"
receiveTimeout="01:00:00"
sendTimeout="01:00:00">
<binaryMessageEncoding maxSessionSize="1048576">
<readerQuotas maxDepth="1048576"
maxStringContentLength="1048576"
maxArrayLength="104857600"
maxBytesPerRead="1048576"
maxNameTableCharCount="1048576" />
</binaryMessageEncoding>
<tcpTransport maxBufferPoolSize="1048576"
maxReceivedMessageSize="104857600"
connectionBufferSize="1048576"
maxBufferSize="1048576"
maxPendingConnections="200"
maxPendingAccepts="200"
transferMode="Streamed"
listenBacklog="200">
<connectionPoolSettings maxOutboundConnectionsPerEndpoint="200" />
</tcpTransport>
</binding>
<binding name="netTcpTextStreamedBinding"
receiveTimeout="01:00:00"
sendTimeout="01:00:00">
<textMessageEncoding maxWritePoolSize="64">
<readerQuotas maxDepth="1048576"
maxStringContentLength="1048576"
maxArrayLength="104857600"
maxBytesPerRead="1048576"
maxNameTableCharCount="1048576" />
</textMessageEncoding>
<tcpTransport maxBufferPoolSize="1048576"
maxReceivedMessageSize="104857600"
connectionBufferSize="1048576"
maxBufferSize="1048576"
maxPendingConnections="200"
maxPendingAccepts="200"
transferMode="Streamed"
listenBacklog="200">
<connectionPoolSettings maxOutboundConnectionsPerEndpoint="200" />
</tcpTransport>
</binding>
<binding name="netTcpMtomBufferedBinding"
receiveTimeout="01:00:00"
sendTimeout="01:00:00">
<mtomMessageEncoding maxBufferSize="1048576">
<readerQuotas maxDepth="1048576"
maxStringContentLength="1048576"
maxArrayLength="104857600"
maxBytesPerRead="1048576"
maxNameTableCharCount="1048576" />
</mtomMessageEncoding>
<tcpTransport maxBufferPoolSize="1048576"
maxReceivedMessageSize="1048576"
connectionBufferSize="1048576"
maxBufferSize="104857600"
maxPendingConnections="200"
maxPendingAccepts="200"
transferMode="Buffered"
listenBacklog="200">
<connectionPoolSettings maxOutboundConnectionsPerEndpoint="200" />
</tcpTransport>
</binding>
<binding name="netTcpBinaryBufferedBinding"
receiveTimeout="01:00:00"
sendTimeout="01:00:00">
<binaryMessageEncoding maxSessionSize="1048576">
<readerQuotas maxDepth="1048576"
maxStringContentLength="1048576"
maxArrayLength="104857600"
maxBytesPerRead="1048576"
maxNameTableCharCount="1048576" />
</binaryMessageEncoding>
<tcpTransport maxBufferPoolSize="1048576"
maxReceivedMessageSize="104857600"
connectionBufferSize="1048576"
maxBufferSize="104857600"
maxPendingConnections="200"
maxPendingAccepts="200"
transferMode="Buffered"
listenBacklog="200">
<connectionPoolSettings maxOutboundConnectionsPerEndpoint="200" />
</tcpTransport>
</binding>
<binding name="netTcpTextBufferedBinding"
receiveTimeout="01:00:00"
sendTimeout="01:00:00">
<textMessageEncoding maxWritePoolSize="64">
<readerQuotas maxDepth="1048576"
maxStringContentLength="1048576"
maxArrayLength="104857600"
maxBytesPerRead="1048576"
maxNameTableCharCount="1048576" />
</textMessageEncoding>
<tcpTransport maxBufferPoolSize="1048576"
maxReceivedMessageSize="104857600"
connectionBufferSize="1048576"
maxBufferSize="104857600"
maxPendingConnections="200"
maxPendingAccepts="200"
transferMode="Buffered"
listenBacklog="200">
<connectionPoolSettings maxOutboundConnectionsPerEndpoint="200" />
</tcpTransport>
</binding>
<binding name="netTcpMtomStreamedTxnBinding"
receiveTimeout="01:00:00"
sendTimeout="01:00:00">
<transactionFlow transactionProtocol="OleTransactions"/>
<mtomMessageEncoding maxBufferSize="1048576"
maxReadPoolSize="1024"
maxWritePoolSize="1024">
<readerQuotas maxDepth="1048576"
maxStringContentLength="1048576"
maxArrayLength="104857600"
maxBytesPerRead="1048576"
maxNameTableCharCount="1048576" />
</mtomMessageEncoding>
<tcpTransport maxBufferPoolSize="1048576"
maxReceivedMessageSize="104857600"
connectionBufferSize="1048576"
maxBufferSize="1048576"
maxPendingConnections="200"
maxPendingAccepts="200"
transferMode="Streamed"
listenBacklog="200">
<connectionPoolSettings maxOutboundConnectionsPerEndpoint="200" />
</tcpTransport>
</binding>
</customBinding>
<netNamedPipeBinding>
<binding name="netNamedPipeBinding"
transferMode="Streamed"
maxBufferPoolSize="1048576"
maxBufferSize="1048576"
maxConnections="200"
maxReceivedMessageSize="104857600">
<security mode="None" />
</binding>
</netNamedPipeBinding>
</bindings>
<client>
<endpoint address="net.tcp://localhost:11550/ssisintegration"
binding="customBinding"
bindingConfiguration="netTcpMtomStreamedBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessage"
name="netTcpMtomStreamedEndpoint" />
<endpoint address="net.tcp://localhost:11551/ssisintegration"
binding="customBinding"
bindingConfiguration="netTcpBinaryStreamedBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessage"
name="netTcpBinaryStreamedEndpoint" />
<endpoint address="net.tcp://localhost:11552/ssisintegration"
binding="customBinding"
bindingConfiguration="netTcpTextStreamedBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessage"
name="netTcpTextStreamedEndpoint" />
<endpoint address="net.tcp://localhost:11553/ssisintegration"
binding="customBinding"
bindingConfiguration="netTcpMtomBufferedBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessage"
name="netTcpMtomBufferedEndpoint" />
<endpoint address="net.tcp://localhost:11554/ssisintegration"
binding="customBinding"
bindingConfiguration="netTcpBinaryBufferedBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessage"
name="netTcpBinaryBufferedEndpoint" />
<endpoint address="net.tcp://localhost:11555/ssisintegration"
binding="customBinding"
bindingConfiguration="netTcpTextBufferedBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessage"
name="netTcpTextBufferedEndpoint" />
<endpoint address="net.tcp://localhost:11556/ssisintegration"
binding="customBinding"
bindingConfiguration="netTcpMtomStreamedTxnBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessageTxn"
name="netTcpMtomStreamedTxnEndpoint" />
<endpoint address="net.tcp://localhost:11557/ssisintegration"
binding="customBinding"
bindingConfiguration="netTcpBinaryStreamedBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessage"
name="netTcpSynchronousSSISPackageEndpoint" />
<endpoint address="net.tcp://localhost:11558/ssisintegration"
binding="customBinding"
bindingConfiguration="netTcpBinaryStreamedBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessage"
name="netTcpAsynchronousSSISPackageEndpoint" />
<endpoint address="net.pipe://localhost/ssiintegration"
binding="netNamedPipeBinding"
bindingConfiguration="netNamedPipeBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessage"
name="netNamedPipeStreamedEndpoint" />
<endpoint address="net.tcp://localhost:11557/ssisintegration"
binding="customBinding"
bindingConfiguration="netTcpMtomStreamedBinding"
contract="Microsoft.BizTalk.CAT.Samples.SSISIntegration.Client.IUploadMessage"
name="directServiceEndpoint" />
</client>
</system.serviceModel>
</configuration>
|
For more information on how using the streamed transfer mode along with WCF, you can read the following topic:
-
“How to: Enable Streaming” article on
MSDN.
BizTalk Application
In order to compare the performance of the various message encoders and transfer modes when transferring large messages over the network, I created six different WCF-Custom Receive Locations, one for each possible combination of these properties:
-
SSISIntegration.WCF-Custom.NetTcp.Mtom.Streamed.ReceiveLocation: TransferMode = Streamed, Message Encoder = MTOM.
-
SSISIntegration.WCF-Custom.NetTcp.Binary.Streamed.ReceiveLocation: TransferMode = Streamed, Message Encoder = Binary.
-
SSISIntegration.WCF-Custom.NetTcp.Text.Streamed.ReceiveLocation: TransferMode = Streamed, Message Encoder = Text.
- SSISIntegration.WCF-Custom.NetTcp.Mtom.Buffered.ReceiveLocation: TransferMode = Buffered, Message Encoder = MTOM.
-
SSISIntegration.WCF-Custom.NetTcp.Binary.Buffered.ReceiveLocation: TransferMode = Buffered, Message Encoder = Binary.
-
SSISIntegration.WCF-Custom.NetTcp.Text.Buffered.ReceiveLocation: TransferMode = Buffered, Message Encoder = Text.
Each of the above Receive Locations use a CustomBinding and the TcpTransportBindingElement. Let’s compare the Binding configuration of the Receive Locations #2 and #5. They both use the Binary message encoder but they are configured to use a different transfer mode, respectively streamed and buffered. If you look at the WCF-Custom Adapter configuration of the SSISIntegration.WCF-Custom.NetTcp.Binary.Streamed.ReceiveLocation receive location (see the picture below), you will note that:
-
The value of the MaxReceivedMessageSize property has been increased to almost 1 GB to handle large messages.
-
The value of the MaxBufferSize property has been set to 1 MB. You can eventually arrange a larger memory buffer.
-
The value of the TransferMode property has been set to Streamed.
-
I increased the value of several properties like
MaxPendingAccepts (default value = 1),
MaxPendingConnections (default value = 10),
ListenBacklog (default value = 10),
MaxOutboundConnectionsPerEndpoint (default value = 10). 200 is probably too high, but it’s important to note here that when using the WCF-Custom/WCF-CustomIsolated Adapter in conjunction with the
CustomBinding, you have can fine tune the value of all the properties exposed by the individual binding elements which compose your binding.
By contrast, the SSISIntegration.WCF-Custom.NetTcp.Binary.Buffered.ReceiveLocation receive location has been configured as follows:
-
The value of the MaxReceivedMessageSize property has been increased to almost 1 GB to handle large messages.
-
The value of the MaxBufferSize property is identical to the value of the MaxReceivedMessageSize. When using the Buffered transfer mode, the incoming message must be entirely received in a memory buffer before message processing can start. Therefore, the value of the MaxBufferSize property must necessarily be equal to the value of the MaxReceivedMessageSize property. If you try to specify a different value for the 2 properties, the WCF-Custom Receive Location will be immediately disabled and the following error message will be written in the Application Log:
The Messaging Engine failed to add a receive location “<Receive Location Name>” with URL “<Receive Location URL>” to the adapter “WCF-Custom”. Reason: “System.ArgumentException: For TransferMode.Buffered, MaxReceivedMessageSize and MaxBufferSize must be the same value.
-
The value of the TransferMode property has been set to Streamed.
-
The value of other performance related has been tuned up.
All the above Receive Locations have been configured to use the ServiceThrottlingBehavior. The default value of the 3 properties exposed by the service behavior have been increased from the default value to 200. Indeed, as I explained in my article called Customizing and Extending the BizTalk WCF Adapters, all incoming messages to a certain WCF Receive Location are received and processed by a singleton instance of the BizTalkServiceInstance class. Thus, the value you specify for the MaxConcurrentInstances property is irrelevant as a single service instance will be used for each WCF Receive Location. Also the value you assign to the MaxConcurrentSessions property is no relevant if you don’t use a session-aware binding. As a consequence, the only property that you should properly set is the MaxConcurrentCalls that specifies the maximum number of messages concurrently processed by the WCF Receive location.
If we enable WCF tracing on the client application and we try to send a file to BizTalk, we can note that the SOAP message sent by the client application to BizTalk has the following format (the namespace of custom headers have been removed for simplicity).
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing">
<s:Header>
<a:Action s:mustUnderstand="1">UploadMessage</a:Action>
<h:dateTime>2010-05-17T17:14:47.0058023+02:00</h:dateTime>
<h:filename>config.xml</h:filename>
<h:id>93204315-aeee-48d0-8922-09ef521c27f2</h:id>
<h:sender>Paolo Salvatori</h:sender>
<h:size>3617</h:size>
<a:MessageID>urn:uuid:d8f87456-cef5-4302-b760-f1765483e4f1</a:MessageID>
<a:ReplyTo>
<a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
</a:ReplyTo>
<a:To s:mustUnderstand="1">net.tcp://localhost:11553/ssisintegration</a:To>
</s:Header>
<s:Body>
<UploadMessage xmlns="http://microsoft.biztalk.cat/10/samples/ssisintegration/uploadservice">
<data>...</data>
</UploadMessage>
</s:Body>
</s:Envelope>
|
As a consequence, in order to extract data from the Body element, it’s necessary to properly configure the Inbound BizTalk message body section of all the WCF Receive Locations:
- The expression “/*[local-name()=’UploadMessage’ and namespace-uri()='<targetNamespace’’]/*[local-name()=’data’ and namespace-uri()=’targetNamespace’]” must be assigned to the Body path expression field.
- Base64 must be selected as Node encoding.
Scenario
To avoid publishing the incoming large message to the MessageBox, I created a custom receive pipeline and in particular a custom pipeline component called StageFilePipeComponent that receives and stage the incoming file a folder or to a custom SQL database using a streaming approach and finally replace the original file with a placeholder document which contains the location of the staged file and its metadata. When published to the MessageBox, the placeholder document is consumed by a Send Port and processed by a custom send pipeline that reads the location of the original message and substitutes the placeholder document with this latter. Besides, the custom pipeline running within the Receive Location promotes the AckRequired context property. This will cause the send port to generate ACK and NACK messages when the adapter reports a message as successfully delivered or in the case of a NACK, when the adapter has failed and a message will be suspended. These messages are special and will only be published to the MessageBox, if there is someone subscribing to them. In my case, ACK and NACK messages are consumed by a special orchestration which role is to remove the original message from the staging repository (Folder, SQL Server) once the original message has been successfully transmitted by the send port or to eventually resubmit a suspended resumable message when the send port runs out of retries. In the next article of the series, I’ll show the code of the pipeline components and orchestration. For now I will only describe the data flow of the demo and discuss the results in terms of performance:
Non Transactional Case
-
The user selects one or multiple files on the UI of the driver application, then selects a service endpoint that corresponds to a given WCF Receive Location and finally click the Submit button. For simplicity, let’s assume that the user chose to send a single file. This latter is received by one of the six WCF Receive Locations.
-
The custom StageFileReceivePipeline reads the message metadata from the custom headers using the InboundHeaders context property and then writes the original file to a staging repository that can be a folder or a table in custom SQL database which contains a varbinary(max) FILSTREAM column. Besides, the pipeline promotes the AckRequired context property to enable delivery notifications and substitutes the original file with a small placeholder document which contains the metadata and actual location of the original message.
-
The placeholder document is published to the BizTalkMsgBoxDb by the Message Agent.
-
The inbound request is consumed by a FILE Send Port.
-
The RestoreFileSendPipeline reads the content of the placeholder document and replaces this latter with the original message. In particular, the RestoreFilePipelineComponent pipeline assign a different type of stream to the outbound IBaseMessage depending on the location of the original message:
-
Folder: the component assign a FileStream object to the Data property of the message body part.
-
SQL Server: the component opens a SqlFileStream to read the content of the original file from SQL Server and wraps this object within a WrapperStream object. The pipeline component opens a connection and initiate a SQL transaction to initiate the SqlFileStream. The content of this latter is not meant to be read by the pipeline component, but by the send adapter, therefore the connection and transaction need to remain open. The scope of the WrapperStream object is committing the transaction and closing the connection when the adapter has successfully transmitted the message. In fact, when the adapter has finished processing, the BizTalk messaging runtime invokes the Close method of the WrapperStream object which contains the code for committing the transaction and closing the connection.
-
The WCF-Custom Send Port write the original message to the target folder (Out).
-
The BizTalk Messaging Runtime generates an ACK/NACK message depending on whether the transmission was successful or not.
-
The ACK/NACK message is consumed by a special orchestration called NotificationHandler. This latter executes a different action depending on the type of the inbound document:
-
ACK: the orchestration deletes the original file from the staging repository.
-
NACK: the orchestration invokes a BRE policy to decide how to proceed. Depending on the information contained in the policy, the orchestration can execute one of the following actions:
Results
The two tables below report the performance data in terms of latency and memory consumption that I measured during my tests. The size of messages used during tests was around 50 MB. The following findings emerge from the analysis of the figures:
- Binary Encoder > Mtom Encoder > Text Encoder, where “>” stands for faster.
- TransferMode= Streamed is faster than the TransferMode= Buffered (the default) when transmitting large messages.
- If the WCF Receive Location uses the TransferMode= Streamed the Working Set of the BizTalk process hosting the Receive Location remains pretty low when receiving multiple large messages at the same time.
- If the WCF Receive Location uses the TransferMode= Buffered the Working Set of the BizTalk process hosting the Receive Location grows almost linearly with the number of concurrent incoming messages.
Conclusions
This article introduced a powerful technique to transfer and handle large files and provided a comparison in terms of performance and memory consumption between the message encoders and transfer modes supplied by WCF. In the next article, I’ll provide more details on the solution implementation and I’ll explain how to configure a WCF-Custom Receive Location and the StageFileReceivePipeline to stage the incoming file to the file system (using the Transactional NTFS or TxF) or to SQL Server in a transactional way using the same transaction utilized by the Message Agent to post the placeholder document to the MessageBox. In the meantime, you can download the code from my site in 2 flavors:
-
-
SSISIntegration: this is a sort of experimental version where the
StageFilePipelineComponent has been extended to allow the possibility to synchronously or asynchronously invoke a SSIS Package against the staged file. To this purpose, I extended the
Helpers library with a class called
PackageHelper that provides the possibility to call an SSIS Package from a custom component.
As always, please provide feedbacks, problems and suggestions.
by community-syndication | Jun 23, 2010 | BizTalk Community Blogs via Syndication
I had a BizTalk Webservice that was generated by the WebService Wizard from BizTalk. When I exported the MSI the WebDirectory came along. I thought this webdirectory was added during the export. But this is not the case. The WebService Wizard adds the…(read more)
by community-syndication | Jun 23, 2010 | BizTalk Community Blogs via Syndication
Scenario
Last year I wrote a post on how using BizTalk Server 2006 R2/2009 and Protocol Transition to impersonate the original caller when invoking a downstream service that uses the Windows Integrated Security. Recently, one customer posed the following question to my colleague, Tim Wieman:
Can I create a WCF Send Port that is able to impersonate a statically defined domain user, other than then the service account of the host instance process, when calling a downstream WCF service that exposes a BasicHttpBinding/WsHttpBinding endpoint configured to use the Transport security mode ?
The answer is:
-
Yes, when using the Basic or the Digest authentication scheme.
-
No, when using the Windows or NTLM client credential type.
To verify this constraint, you can proceed as follows:
-
Open the BizTalk Administration Console.
-
Create a new WCF-BasicHttp or WCF-WsHttp Static Solicit-Response Send Port
-
Click the Configure button.
-
Select the Security tab.
-
Choose the Transport security mode.
At this point, if you select the Basic or Digest transport client credential type from the corresponding drop-down list:
Instead, if you select the Ntlm or Windows transport client credential type from the corresponding drop-down list, the Edit button in the User name credentials section is greyed out, as shown in the picture below:
So at this point some of you might ask yourselves:
How can I impersonate a statically-defined user, different from the service account of the host process running my WCF Send Port, when invoking an underlying WCF service that uses the Transport security mode along with the Ntlm or Windows authentication scheme?
The answer is straightforward, you can achieve this objective using the WCF-Custom adapter and writing a custom WCF channel. Indeed, I didn’t create a new component from scratch, I just used grabbed some code from MSDN, and extended the component I wrote one year ago for my previous post on BizTalk and Protocol Transition . In particular, I made the following changes:
-
I extended the following components:
-
to expose two additional properties:
-
Then I extended the WindowsUserPositionEnum type and therefore the WindowsUserPosition property to include a new mode called Static. As a consequence, the custom channel at runtime will retrieve the client credentials in a different way depending on the value of the WindowsUserPosition property exposed by the InspectingBindingExtensionElement component:
-
Context: username will be read from the message context property identified by the
ContextPropertyName and
ContextPropertyNamespace properties exposed by the InspectingBindingExtensionElement. In this case, the environment must be properly configured to use Protocol Transition. See my previous
post for more details.
-
Message: username will be read from the message using the XPath expression contained in the
WindowsUserXPath property. Even in this case, the environment must be properly configured to use Protocol Transition. See my previous
post for more details.
-
Static: the custom channel will use the client credentials contained in the
WindowsUserName and
WindowsUserPassword to impersonate the corresponding domain account before invoking the downstream WCF service.
Note: this pattern requires the component to uses the client credentials to invoke the
LogonUser function at runtime, but it does not require to configure the BizTalk environment for Protocol Transition.
-
I finally extended the InspectingRequestChannel and InspectingHelper classes to support the new Static mode. In particular, the custom channel at runtime performs the following steps to impersonate a given domain account declaratively defined in the WCF Send Port configuration:
For your convenience, I report below the new code for the InspectingRequestChannel and InspectingHelper classes (I purposely omitted parts for ease of reading):
InspectingHelper class
#region Copyright
//-------------------------------------------------
// Author: Paolo Salvatori
// Email: paolos@microsoft.com
// History: 2008-09-17 Created
//-------------------------------------------------
#endregion
#region Using Directives
using System;
using System.Diagnostics;
using System.Configuration;
using System.Runtime.InteropServices;
using System.Security.Principal;
using System.Security.Permissions;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.ServiceModel.Configuration;
using System.DirectoryServices.ActiveDirectory;
using System.Xml;
using System.IO;
using System.Text;
using Microsoft.BizTalk.XPath;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.ProtocolTransition.WCFExtensionLibrary
{
/// <summary>
/// This class exposes the logic to impersonate another user using the Protocol Transition mechanism.
/// </summary>
public class InspectingHelper
{
#region DllImport
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
public static extern bool LogonUser(String lpszUsername, String lpszDomain, String lpszPassword,
int dwLogonType, int dwLogonProvider, ref IntPtr phToken);
[DllImport("kernel32.dll", CharSet = CharSet.Auto)]
public extern static bool CloseHandle(IntPtr handle);
#endregion
#region Private Constants
...
// The following constants are used when calling the LogonUser external function
private const int LOGON32_PROVIDER_DEFAULT = 0;
//This parameter causes LogonUser to create a primary token.
private const int LOGON32_LOGON_INTERACTIVE = 2;
#endregion
#region Private Static Fields
private static string domainFQDN = string.Empty;
#endregion
#region Public Static Constructor
static InspectingHelper()
{
try
{
Domain domain = Domain.GetComputerDomain();
if (domain != null)
{
domainFQDN = domain.Name;
}
}
catch (Exception ex)
{
Debug.WriteLine(string.Format(MessageFormat, ex.Message));
}
}
#endregion
#region Static Public Methods
public static string GetUserPrincipalName(ref Message message,
WindowsUserPositionEnum windowsUserPosition,
string contextPropertyName,
string contextPropertyNamespace,
string windowsUserXPath,
string windowsUserName,
string windowsUserPassword,
int maxBufferSize,
bool traceEnabled,
out string userName,
out string domainName)
{
string windowsUser = null;
domainName = null;
userName = null;
try
{
switch (windowsUserPosition)
{
case WindowsUserPositionEnum.Message:
if (message != null &&
!string.IsNullOrEmpty(windowsUserXPath))
{
MessageBuffer messageBuffer = message.CreateBufferedCopy(maxBufferSize);
if (messageBuffer == null)
{
throw new ApplicationException(MessageBufferCannotBeNull);
}
Message clone = messageBuffer.CreateMessage();
if (message == null)
{
throw new ApplicationException(CloneCannotBeNull);
}
message = messageBuffer.CreateMessage();
if (message == null)
{
throw new ApplicationException(MessageCannotBeNull);
}
XmlDictionaryReader xmlDictionaryReader = clone.GetReaderAtBodyContents();
if (xmlDictionaryReader == null)
{
throw new ApplicationException(XmlDictionaryReaderCannotBeNull);
}
XPathCollection xPathCollection = new XPathCollection();
if (xPathCollection == null)
{
throw new ApplicationException(XPathCollectionCannotBeNull);
}
XPathReader xPathReader = new XPathReader(xmlDictionaryReader, xPathCollection);
if (xPathReader == null)
{
throw new ApplicationException(XPathReaderCannotBeNull);
}
xPathCollection.Add(windowsUserXPath);
bool ok = false;
while (xPathReader.ReadUntilMatch())
{
if (xPathReader.Match(0) && !ok)
{
windowsUser = xPathReader.ReadString();
ok = true;
}
}
}
break;
case WindowsUserPositionEnum.Context:
if (string.IsNullOrEmpty(contextPropertyName))
{
throw new ApplicationException(ContextPropertyNameCannotBeNull);
}
if (string.IsNullOrEmpty(contextPropertyNamespace))
{
throw new ApplicationException(ContextPropertyNamespaceCannotBeNull);
}
string contextPropertyKey = string.Format(ContextPropertyKeyFormat,
contextPropertyNamespace,
contextPropertyName);
if (message.Properties.ContainsKey(contextPropertyKey))
{
windowsUser = message.Properties[contextPropertyKey] as string;
}
else
{
throw new ApplicationException(string.Format(NoContextPropertyFormat, contextPropertyKey));
}
break;
case WindowsUserPositionEnum.Static:
windowsUser = windowsUserName;
break;
}
if (!string.IsNullOrEmpty(windowsUser))
{
string[] parts = windowsUser.Split(new char[] { Path.DirectorySeparatorChar });
if (parts != null &&
parts.Length > 1)
{
domainName = parts[0];
userName = parts[1];
Debug.WriteLineIf(traceEnabled, string.Format(CreatingUPNFormat, windowsUser));
string upn = string.Format(UserPrincipalNameFormat, parts[1], domainFQDN);
Debug.WriteLineIf(traceEnabled, string.Format(UsingUserPrincipalNameFormat, upn));
return upn;
}
}
}
catch (Exception ex)
{
Debug.WriteLineIf(traceEnabled, string.Format(MessageFormat, ex.Message));
throw ex;
}
return null;
}
public static bool LogonUser(string userName,
string domainName,
string windowsUserPassword,
bool traceEnabled,
ref IntPtr tokenHandle)
{
Debug.WriteLineIf(traceEnabled, string.Format(StartLogonUserFormat,
domainName ?? Unknown,
userName ?? Unknown));
// Call LogonUser to obtain a handle to an access token.
bool ok = LogonUser(userName,
domainName,
windowsUserPassword,
LOGON32_LOGON_INTERACTIVE,
LOGON32_PROVIDER_DEFAULT,
ref tokenHandle);
if (traceEnabled)
{
if (ok)
{
Debug.WriteLineIf(traceEnabled, string.Format(LogonUserSucceededFormat,
domainName ?? Unknown,
userName ?? Unknown));
}
else
{
Debug.WriteLineIf(traceEnabled, string.Format(LogonUserFailedFormat,
domainName ?? Unknown,
userName ?? Unknown));
}
}
return ok;
}
public static bool CloseHandle(IntPtr tokenHandle,
string userName,
string domainName,
bool traceEnabled)
{
Debug.WriteLineIf(traceEnabled, string.Format(StartCloseTokenFormat,
domainName ?? Unknown,
userName ?? Unknown));
bool ok = CloseHandle(tokenHandle);
if (traceEnabled)
{
if (ok)
{
Debug.WriteLineIf(traceEnabled, string.Format(CloseTokenSucceededFormat,
domainName ?? Unknown,
userName ?? Unknown));
}
else
{
Debug.WriteLineIf(traceEnabled, string.Format(CloseTokenFailedFormat,
domainName ?? Unknown,
userName ?? Unknown));
}
}
return ok;
}
#endregion
}
}
|
InspectingRequestChannel class
public class InspectingRequestChannel : InspectingChannelBase<IRequestChannel>,
IRequestChannel
{
...
public Message Request(Message message, TimeSpan timeout)
{
Message reply = null;
string upn = null;
WindowsImpersonationContext impersonationContext = null;
IntPtr tokenHandle = new IntPtr(0);
string userName = null;
string domainName = null;
try
{
if (componentEnabled)
{
WindowsIdentity identity = null;
upn = InspectingHelper.GetUserPrincipalName(ref message,
windowsUserPosition,
contextPropertyName,
contextPropertyNamespace,
windowsUserXPath,
windowsUserName,
windowsUserPassword,
maxBufferSize,
traceEnabled,
out userName,
out domainName);
if (windowsUserPosition == WindowsUserPositionEnum.Static)
{
// Call LogonUser to obtain a handle to an access token.
bool returnValue = InspectingHelper.LogonUser(userName,
domainName,
windowsUserPassword,
traceEnabled,
ref tokenHandle);
// Protocol Transition is not necessary in this case
identity = new WindowsIdentity(tokenHandle);
}
else
{
if (!string.IsNullOrEmpty(upn))
{
// Protocol Transition must be properly configured,
// otherwise the impersonation will fail
identity = new WindowsIdentity(upn);
}
}
Debug.WriteLineIf(traceEnabled, string.Format(ImpersonatingFormat, upn));
impersonationContext = identity.Impersonate();
Debug.WriteLineIf(traceEnabled, string.Format(ImpersonatedFormat, upn));
}
Debug.WriteLineIf(traceEnabled, CallingWebService);
reply = this.InnerChannel.Request(message);
Debug.WriteLineIf(traceEnabled, WebServiceCalled);
}
catch (Exception ex)
{
Debug.WriteLineIf(traceEnabled, string.Format(MessageFormat, ex.Message));
throw ex;
}
finally
{
if (impersonationContext != null)
{
impersonationContext.Undo();
Debug.WriteLineIf(traceEnabled, string.Format(ImpersonationUndoneFormat, upn ?? Unknown));
}
if (tokenHandle != IntPtr.Zero)
{
InspectingHelper.CloseHandle(tokenHandle,
userName,
domainName,
traceEnabled);
}
}
return reply;
}
}
|
Test Case
To test my component, I created the following test case:
WinForm driver application submits a new request to a WCF-NetTcp Request-Response Receive Location
- The Message Agent submits the incoming request message to the MessageBox (BizTalkMsgBoxDb).
- The inbound request is consumed by a Solicit Response WCF-Custom Send Port. This latter uses a Filter Expression to receive all the documents published by the Receive Port hosting the WCF Receive Location.
- The inbound message is mapped to the request format by the downstream HelloWorldService web service. This latter is hosted by IIS and exposes a single WsHttpBinding endpoint.
- The WCF-Custom Send Port impersonates the user statically defined in the Port configuration and invokes the underlying HelloWorldService WCF service that in the scenario is hosted by a separate Application Pool (w3wp.exe) on the same IIS instance.
- The HelloWorldService WCF service returns a response message.
- The incoming response message is mapped to the format expected by the client application.
- The transformed response message is published to the MessageBox.
- The response message is retrieved by the Request-Response WCF Custom Receive Location which originally received the request call.
- The response message is returned to the client WinForm application.
The following picture shows the binding configuration of the WCF Send Port used to communicate with the HelloWorldService.
Finally, the picture below reports the trace captured during a test run.
Conclusions
As I explained in one of my recent posts, using WCF extensibility points allows you customize in-depth the default behavior of BizTalk WCF Adapters. In particular, the WCF-Custom Adapter provides the possibility to specify the customize the composition of the binding and hence of the channel stack that will be created and used at runtime to communicate with external applications.
In this article we have seen how to exploit this characteristic to workaround and bypass a constraint of WCF Adapters. As usual, I had just a few hours to write the code and write the article, so should you find an error or a problem in my component, please send me an email or leave a comment on my blog, thanks!
You can find the new version of the code here.
by community-syndication | Jun 23, 2010 | BizTalk Community Blogs via Syndication
Can you use the updated development environment for BizTalk Server 2010 with Visual Studio 2010 while still deploying to BizTalk Server 2009?
Can you do it? On a file level, yes. On a project level, yes. On a solution level, doesn’t seem like it. On an assembly level, no.
I tried two scenarios.
- A simple messaging only scenario with a transformation on the receive port.
- An orchestration scenario picking up the file from the receive location, doing the mapping and delivering it to the send port.
I developed the solution on a BizTalk Server 2010 / Visual Studio 2010 combo and deployed it to BizTalk Server 2009.
So what worked and what didn’t?
Compile and deploy the .NET 4 assembly to BizTalk Server 2009
FAIL!
Why?
Even though I did Add a resource manually selecting the file to be a BizTalkAssembly BizTalk Server 2009 kept on reverting it back to File. Obviously it doesn’t recognize it, or recognizes that it’s an incorrect version of the framework.
Re-target the solution to .NET 3.5 or .NET 2.0
FAIL!
Why?
The following is taken from http://msdn.microsoft.com/en-us/library/ff629735(BTS.70).aspx
BizTalk Server 2010 supports building new applications only on .NET Framework version 4 . You can build both BizTalk applications as well as custom applications (such as custom functoids, custom pipeline components) using .NET Framework 4 . However, BizTalk Server 2010 continues to support the BizTalk Server 2006 R2 and BizTalk Server 2009 applications (BizTalk applications and custom applications) built on .NET Framework 3.5 SP2.
If you launch an existing BizTalk project in BizTalk Server 2010 , it automatically gets migrated, changing the .NET Framework version to 4.0. You cannot modify the .NET version number later in the Properties dialog box of the BizTalk project.
In essence it says that you cannot use the re-targeting functionality within Visual Studio 2010 for BizTalk Server 2010 projects. If you try, because the drop down is still there, you get the following dialog:
Using Files created by Visual Studio 2010, like .xsd, .btm or .odx
SUCCESS!
You can copy paste files from one project into another, without any issues I detected.
Open the project file created with Visual Studio 2010 and re-compile using Visual Studio 2008
SUCCESS! But with some issues
You can open a BizTalk project created in Visual Studio 2010 in Visual Studio 2008 and just re-compile it. Without any issues I detected. However, it does not seem like Visual Studio 2008 wants to open the solution file, which may be an issue in some situations, like for example if you wan’t to create automated builds. The structure of the solution file begins like this:
Microsoft Visual Studio Solution File, Format Version 11.00
# Visual Studio 2010
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "BizTalk Server Project1",
"BizTalk Server Project1\BizTalk Server Project1.btproj", "{DE7C19FB-A5BA-46A0-9382-ACFB3EB91409}"
This may very well be what causes Visual Studio to dismiss it.
Hope this helps someone thinking along these lines 🙂
Disclaimer: This is tested with BizTalk Server 2010 Beta. All artifacts used while performing these tests are trivial. It might very well be that in some more complex scenario this might not work.
by community-syndication | Jun 23, 2010 | BizTalk Community Blogs via Syndication
Problem Statement
Some of you posted the following feedback regarding my post How To Boost Message Transformations Using the XslCompiledTransform class:
“Very nice post. I just have a question. How would you handle multiple input messages in a map? I have a map that has 2 input source messages that i would like to unit test. The BTS expression shape has transform(outmsg) = map(msg1, msg2), but i have not yet found a way to do it in a C# class.”
Solution
Well, before answering to this question, I’ll briefly explain how BizTalk Server handles maps with multiple input messages. BizTalk Server uses a trick to manage this situation. In fact, when you create a transformation map with multiple source document, BizTalk uses an envelope to wrap the individual input messages. For those of you who love to disassemble BizTalk code with Reflector, the envelope is created at runtime by the private class called CompositeStreamReader that can be found within the assembly Microsoft.XLANGs.Engine. In particular, the ConstructCompositeOutline method uses the code reported in the table below
public void ConstructCompositeOutline()
{
...
XmlTextWriter writer = new XmlTextWriter(this.outlineStream, ...);
writer.WriteStartElement("Root", "http://schemas.microsoft.com/BizTalk/2003/aggschema");
for (int i = 0; i < this.readerCount; i++)
{
writer.WriteStartElement("InputMessagePart_" + i.ToString(), "");
writer.WriteComment("_");
writer.WriteEndElement();
}
writer.WriteEndElement();
writer.Flush();
...
}
|
to create an envelope which targetNamespace is equal to ‘http://schemas.microsoft.com/BizTalk/2003/aggschema’ and that contains as many InputMessagePart elements as the incoming documents to the map.
<ns0:Root xmlns:ns0='http://schemas.microsoft.com/BizTalk/2003/aggschema'>
<InputMessagePart_0>
-
</InputMessagePart_0>
<InputMessagePart_1>
-
</InputMessagePart_1>
</ns0:Root>
|
Therefore, I decided to extend the code of my classes XslCompiledTransformHelper and XslTransformHelper to handle the case of maps with multiple input messages. In particular, I developed a new class called CompositeStream to wrap an array of Stream objects, one for each input message to the map, an return the above envelope. The implementation of this custom class is quite smart, because instead of copying the bytes of the input streams within a new buffer or a new stream, the Read method just makes up the content of the envelope with the data of the inbound streams to return a composite message with the format expected by the map. When the inbound streams are significantly large, this approach allows saving time and memory for copying data from the inbound streams to a new object, regardless if this latter is a buffer or a stream. The code of the CompositeStream class is shown in the table below:
CompositeStream Class
#region Copyright
//-------------------------------------------------
// Author: Paolo Salvatori
// Email: paolos@microsoft.com
// History: 2010-04-07 Created
//-------------------------------------------------
#endregion
#region Using References
using System;
using System.IO;
using System.Text;
using System.Collections.Generic;
using System.Configuration;
using System.Xml;
using System.Xml.Xsl;
using System.Diagnostics;
using Microsoft.XLANGs.BaseTypes;
using Microsoft.XLANGs.Core;
using Microsoft.BizTalk.Streaming;
using Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Helpers.Properties;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Helpers
{
public class CompositeStream : Stream
{
#region Private Types
enum State
{
Start,
Overflow,
Stream,
End
}
#endregion
#region Private Constants
private const string DefaultPrefix = "babo";
private const string StartRoot = "<{0}:Root xmlns:{0}='http://schemas.microsoft.com/BizTalk/2003/aggschema'>";
private const string EndRoot = "</{0}:Root>";
private const string StartInputMessagePart = "<InputMessagePart_{0}>";
private const string EndInputMessagePart = "</InputMessagePart_{0}>";
#endregion
#region Private Fields
private int currentStream = 0;
private int currentIndex = 0;
private string prefix;
private State state;
private byte[] overflowBuffer;
private Stream[] streams;
private bool endOfDocument = false;
#endregion
#region Public Constructors
public CompositeStream(Stream[] streams)
{
this.prefix = DefaultPrefix;
this.streams = streams;
this.state = State.Start;
}
public CompositeStream(Stream[] streams, string prefix)
{
this.prefix = prefix;
this.streams = streams;
this.state = State.Start;
}
#endregion
#region Public Properties
public override bool CanRead
{
get
{
return true;
}
}
public override bool CanSeek
{
get
{
return true;
}
}
public override bool CanTimeout
{
get
{
return false;
}
}
public override bool CanWrite
{
get
{
return false;
}
}
public override long Length
{
get
{
int prefixLength = prefix.Length;
long length = 76 + 3 * prefixLength;
if (streams != null &&
streams.Length > 0)
{
string index;
for (int i = 0; i < streams.Length; i++)
{
if (streams[i].CanSeek)
{
index = i.ToString();
length += streams[i].Length + 39 + 2 * index.Length;
}
else
{
throw new NotImplementedException();
}
}
}
return length;
}
}
public override long Position
{
get
{
if (state == State.Start)
{
return 0L;
}
else
{
throw new NotImplementedException();
}
}
set
{
if (value == 0L)
{
ResetStream();
}
else
{
throw new NotImplementedException();
}
}
}
#endregion
#region Public Methods
public override int Read(byte[] buffer, int offset, int count)
{
int bytesWritten = 0;
int bytesRead = 0;
int length = 0;
byte[] localBuffer;
StringBuilder builder;
if (state == State.End)
{
return 0;
}
while (bytesWritten < count &&
state != State.End)
{
switch (state)
{
case State.Start:
builder = new StringBuilder(128);
builder.AppendFormat(StartRoot, prefix);
if (streams != null &&
streams.Length > 0)
{
builder.AppendFormat(StartInputMessagePart, currentStream);
}
localBuffer = Encoding.UTF8.GetBytes(builder.ToString());
if (localBuffer.Length <= count)
{
Array.Copy(localBuffer, 0, buffer, offset, localBuffer.Length);
bytesWritten += localBuffer.Length;
offset += bytesWritten;
state = State.Stream;
}
else
{
Array.Copy(localBuffer, 0, buffer, offset, count);
overflowBuffer = localBuffer;
currentIndex = count;
state = State.Overflow;
return count;
}
break;
case State.Overflow:
length = overflowBuffer.Length - currentIndex;
if (length <= count)
{
Array.Copy(overflowBuffer, currentIndex, buffer, offset, length);
bytesWritten += length;
offset += length;
overflowBuffer = null;
currentIndex = 0;
if (endOfDocument)
{
state = State.End;
}
else
{
state = State.Stream;
}
}
else
{
Array.Copy(overflowBuffer, currentIndex, buffer, offset, count);
currentIndex += count;
return count;
}
break;
case State.Stream:
length = count - bytesWritten;
bytesRead = streams[currentStream].Read(buffer, offset, length);
bytesWritten += bytesRead;
offset += bytesRead;
if (bytesWritten < count)
{
builder = new StringBuilder(128);
builder.AppendFormat(EndInputMessagePart, currentStream);
currentStream++;
if (currentStream < streams.Length)
{
builder.AppendFormat(StartInputMessagePart, currentStream);
localBuffer = Encoding.UTF8.GetBytes(builder.ToString());
}
else
{
builder.AppendFormat(EndRoot, prefix);
localBuffer = Encoding.UTF8.GetBytes(builder.ToString());
endOfDocument = true;
}
if (localBuffer.Length <= count - bytesWritten)
{
Array.Copy(localBuffer, 0, buffer, offset, localBuffer.Length);
bytesWritten += localBuffer.Length;
offset += localBuffer.Length;
if (endOfDocument)
{
if (bytesWritten <= count)
{
state = State.End;
}
return bytesWritten;
}
break;
}
else
{
length = count - bytesWritten;
Array.Copy(localBuffer, 0, buffer, offset, length);
overflowBuffer = localBuffer;
currentIndex = length;
state = State.Overflow;
return count;
}
}
else
{
return count;
}
break;
}
}
return bytesWritten;
}
public override IAsyncResult BeginRead(byte[] buffer, int offset, int count, AsyncCallback callback, object state)
{
return base.BeginRead(buffer, offset, count, callback, state);
}
public override IAsyncResult BeginWrite(byte[] buffer, int offset, int count, AsyncCallback callback, object state)
{
return base.BeginWrite(buffer, offset, count, callback, state);
}
public override void Write(byte[] buffer, int offset, int count)
{
throw new NotImplementedException();
}
public override void SetLength(long value)
{
throw new NotImplementedException();
}
public override long Seek(long offset, SeekOrigin origin)
{
if (offset == 0 &&
origin == SeekOrigin.Begin)
{
ResetStream();
}
else
{
throw new NotImplementedException();
}
return 0;
}
public override void Flush()
{
throw new NotImplementedException();
}
#endregion
#region Private Methods
private void ResetStream()
{
for (int i = 0; i < streams.Length; i++)
{
if (streams[i].CanSeek)
{
streams[i].Seek(0, SeekOrigin.Begin);
}
else
{
throw new NotImplementedException();
}
}
state = State.Start;
endOfDocument = false;
currentStream = 0;
currentIndex = 0;
}
#endregion
}
}
|
Then I extended the XslCompiledTransformHelper class with a set of new methods that accept as parameter an array of objects of type Stream or XLANGMessage and use an instance of the CompositeStream class to apply a transformation map to these latter.
XslCompiledTransformHelper Class
#region Copyright
//-------------------------------------------------
// Author: Paolo Salvatori
// Email: paolos@microsoft.com
// History: 2010-01-26 Created
//-------------------------------------------------
#endregion
#region Using References
using System;
using System.IO;
using System.Text;
using System.Collections.Generic;
using System.Configuration;
using System.Xml;
using System.Xml.Xsl;
using System.Xml.XPath;
using System.Diagnostics;
using Microsoft.XLANGs.BaseTypes;
using Microsoft.XLANGs.Core;
using Microsoft.BizTalk.Streaming;
using Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Helpers.Properties;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Helpers
{
public class XslCompiledTransformHelper
{
...
public static XLANGMessage Transform(XLANGMessage[] messageArray,
int[] partIndexArray,
string mapFullyQualifiedName,
string messageName,
string partName,
bool debug,
int bufferSize,
int thresholdSize)
{
try
{
if (messageArray != null &&
messageArray.Length > 0)
{
Stream[] streamArray = new Stream[messageArray.Length];
for (int i = 0; i < messageArray.Length; i++)
{
streamArray[i] = messageArray[i][partIndexArray[i]].RetrieveAs(typeof(Stream)) as Stream;
}
Stream response = Transform(streamArray, mapFullyQualifiedName, debug, bufferSize, thresholdSize);
CustomBTXMessage customBTXMessage = null;
customBTXMessage = new CustomBTXMessage(messageName, Service.RootService.XlangStore.OwningContext);
customBTXMessage.AddPart(string.Empty, partName);
customBTXMessage[0].LoadFrom(response);
return customBTXMessage.GetMessageWrapperForUserCode();
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException(Resources.XslCompiledTransformHelper, ex);
TraceHelper.WriteLineIf(debug,
null,
ex.Message,
EventLogEntryType.Error);
throw;
}
finally
{
if (messageArray != null &&
messageArray.Length > 0)
{
for (int i = 0; i < messageArray.Length; i++)
{
if (messageArray[i] != null)
{
messageArray[i].Dispose();
}
}
}
}
return null;
}
public static Stream Transform(Stream[] streamArray,
string mapFullyQualifiedName)
{
return Transform(streamArray,
mapFullyQualifiedName,
false,
DefaultBufferSize,
DefaultThresholdSize);
}
public static Stream Transform(Stream[] streamArray,
string mapFullyQualifiedName,
bool debug)
{
return Transform(streamArray,
mapFullyQualifiedName,
debug,
DefaultBufferSize,
DefaultThresholdSize);
}
public static Stream Transform(Stream[] streamArray,
string mapFullyQualifiedName,
bool debug,
int bufferSize,
int thresholdSize)
{
try
{
MapInfo mapInfo = GetMapInfo(mapFullyQualifiedName, debug);
if (mapInfo != null)
{
CompositeStream compositeStream = null;
try
{
VirtualStream virtualStream = new VirtualStream(bufferSize, thresholdSize);
compositeStream = new CompositeStream(streamArray);
XmlTextReader reader = new XmlTextReader(compositeStream);
mapInfo.Xsl.Transform(reader, mapInfo.Arguments, virtualStream);
virtualStream.Seek(0, SeekOrigin.Begin);
return virtualStream;
}
finally
{
if (compositeStream != null)
{
compositeStream.Close();
}
}
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException(Resources.XslCompiledTransformHelper, ex);
TraceHelper.WriteLineIf(debug,
null,
ex.Message,
EventLogEntryType.Error);
throw;
}
return null;
}
...
}
}
|
In a similar way, I extended the XslTransformHelper class to support maps with multiple input documents. For brevity, I omitted the code of this latter, but you can download the new version of both classes here.
Test
To test the new methods exposed by the classes XslCompiledTransformHelper and XslTransformHelper, I created three XML schemas:
- Credentials
- Address
- Customer
In particular, the Credentials and Address schemas define two complex types used to build the Customer schema. Then I created a transformation map (see the picture below) called AddressAndCredentialsToCustomer that accepts 2 input messages, the first of type Credentials and the second of type Address, and returns a document of type Customer.
Finally I created 2 Unit Test methods called:
- TestXslTransformHelperWithMultipleInputs
- TestXslCompiledTransformHelperWithMultipleInputs
to test the CompositeStream and the new methods exposed by the the classes XslCompiledTransformHelper and XslTransformHelper. You can change the entries contained in the appSettings section of the App.Config file within the UnitAndLoadTests project to test the 2 classes against your documents and multi-input-message maps.
Conclusions
I updated the original post to reflect the extensions I made. Here you can find a new version of my helper classes and the artifacts (schemas, maps, unit tests) I used to test them.
Note: I spent less than one day to write and test the new code. In particular, I conducted a basic code coverage of the CompositeStream class, but I didn’t test it with more than 2 streams or with large messages. If you find any error, please send me a repro and I’ll do my best, time permitting, to fix the code as soon as possible. Instead, should you find an error in my code and decide to sort it out by yourselves, please let me have the fixed version. 😉
by community-syndication | Jun 23, 2010 | BizTalk Community Blogs via Syndication
Text in Row Table Option in SQL Server
In SQL Server, text, ntext, or image fields are large character or binary strings (up to 2 gigabytes) that by default stored outside a data row. The data row contains only a 16-byte text pointer that points to the root node of a tree built of internal pointers. These pointers map the pages in which the string fragments are stored.
SQL Server provides a table option called text in row to declare that the contents of the fields of type text, ntext, or image whose dimensions are smaller than those of a data page (8Kb) must be stored in the data row. When a large value type or a large object data type column value is stored in the data row, the Database Engine does not have to access a separate page or set of pages to read or write the character or binary string. This makes reading and writing the in-row strings about as fast as reading or writing limited size varchar, nvarchar, or varbinary strings. Similarly, when the values are stored off-row, the Database Engine incurs an additional page read or write.
For more information on this topic, see the following articles:
SQL Server 2005
SQL Server 2008
How to exploit the Text in Row table option in BizTalk Server
The following section explains how and when applying the text in row table option to boost BizTalk performance.
Parts Table
When the message size is less than 8k the text in row table option can be applied to the Parts table that has 2 fields of type image:
This technique can lead to a decrease of CPU usage up to 20-30% on the SQL and BizTalk nodes and to a significant improvement in terms of latency and throughput, when messages are smaller than 8kb.
Spool Table
Similarly, when the average size of the message context is less than 8 kb, this technique can be extended to the Spool table. This latter contains a column of type image:
To successfully apply this technique to the Spool table, you must eliminate unnecessary context properties and distinguished fields to reduce the size of the message context below 8 Kb.
DynamicStateInfo Tables
Finally, the text in row table option can be applied to the DynamicStateInfo_Host tables that contain the serialized state of orchestrations when they encounter a persistence point during their execution. These tables, one for each host, contain a field of type image:
When orchestrations running within a host HostA are so small that their size once serialized is less than 8 kb, the text in row technique can successfully be applied to the DynamicStateInfo_HostA table. Therefore it’s advisable to keep the internal state of orchestrations as small as possible. This technique can significantly reduce the time spent by the XLANG Engine to serialize, persist and restore the internal state of an orchestration in case of persistence point. Moreover, when the size of this latter is smaller than 8 Kb, it allows using the text in row technique with DynamicStateInfo_Host tables. The following techniques can be used to reduce the memory footprint of orchestrations:
· Create variables and messages as late as possible and release them as early as possible for example introducing non transactional scopes inside your orchestrations and declaring variables and messages within these inner scopes instead of declaring them at the top-most level.
· Use static variables and components whenever possible. When a persistence point occurs, these latter will not be serialized along with the orchestration.
Text In Row in Action
During a recent Performance and Stability review with a customer, I applied the text in row technique to the Parts table (see the picture below) of the BizTalkMsgBoxDb as the average size of messages processed by the BizTalk application was less than 2 Kb.
EXEC sp_tableoption N'Parts', 'text in row', '4000'
|
This led to a decrease of 22% in % Processor Time on BizTalk nodes and to a significant reduction of I/O contention on the SQL Server instance hosting the BizTalkMsgBoxDb.
References
For more information on this topic, see the following articles:
by community-syndication | Jun 23, 2010 | BizTalk Community Blogs via Syndication
Introduction
The BizTalk Runtime still makes an extensive use of the System.Xml.Xsl.XslTransform. When you create and build a BizTalk project, a separate .NET class is generated. for each transformation map. Each of these classes inherits from the Microsoft.XLANGs.BaseTypes.TransformBase class. For convenience, I used Reflector to retrieve and report its code in the table below. As you can easily note, the get accessor of the Transform property returns a XslTransform object.
TransformBase class
[Serializable]
public abstract class TransformBase
{
// Methods
protected TransformBase()
{
}
// Properties
public virtual string[] SourceSchemas
{
get
{
return null;
}
}
public BTSXslTransform StreamingTransform
{
get
{
StringReader input = new StringReader(this.XmlContent);
XmlTextReader stylesheet = new XmlTextReader(input);
BTSXslTransform transform = new BTSXslTransform();
transform.Load(stylesheet, null, base.GetType().Assembly.Evidence);
return transform;
}
}
public virtual string[] TargetSchemas
{
get
{
return null;
}
}
public XslTransform Transform
{
get
{
StringReader input = new StringReader(this.XmlContent);
XmlTextReader stylesheet = new XmlTextReader(input);
XslTransform transform = new XslTransform();
transform.Load(stylesheet, null, base.GetType().Assembly.Evidence);
return transform;
}
}
public XsltArgumentList TransformArgs
{
get
{
XmlDocument document = new XmlDocument();
document.PreserveWhitespace = true;
document.LoadXml(this.XsltArgumentListContent);
XsltArgumentList list = new XsltArgumentList();
foreach (XmlNode node in document.SelectNodes("//ExtensionObjects/ExtensionObject"))
{
XmlAttributeCollection attributes = node.Attributes;
XmlNode namedItem = attributes.GetNamedItem("Namespace");
XmlNode node3 = attributes.GetNamedItem("AssemblyName");
XmlNode node4 = attributes.GetNamedItem("ClassName");
object extension = Assembly.Load(node3.Value).CreateInstance(node4.Value);
list.AddExtensionObject(namedItem.Value, extension);
}
return list;
}
}
public abstract string XmlContent { get; }
public abstract string XsltArgumentListContent { get; }
}
|
When BizTalk Server 2004 was built, the XslTransform was the only class provided by the Microsoft .NET Framework 1.1 to apply an XSLT to an inbound XML document. When the Microsoft .NET Framework version 2.0. was released, the XslTransform was declared obsolete and thus deprecated. As clearly stated on MSDN, the System.Xml.Xsl.XslCompiledTransform should be used instead. This class is used to compile and execute XSLT transformations. In most cases, the XslCompiledTransform class significantly outperforms the XslTransform class in terms of time need to execute the same XSLT against the same inbound XML document. The article Migrating From the XslTransform Class on MSDN reports as follows:
“The XslCompiledTransform class includes many performance improvements. The new XSLT processor compiles the XSLT style sheet down to a common intermediate format, similar to what the common language runtime (CLR) does for other programming languages. Once the style sheet is compiled, it can be cached and reused.”
The caveat is that because the XSLT is compiled to MSIL, the first time the transform is run there is a performance hit, but subsequent executions are much faster. To avoid paying the extra cost of initial compilation every time a map is executed, this latter could be cached in a static structure (e.g. Dictionary). I’ll show you how to implement this pattern in the second part of the article. For a detailed look at the performance differences between the XslTransform and XslCompiledTransform classes (plus comparisons with other XSLT processors) have a look at following posts.
Although the overall performance of the XslCompiledTransform class is better than the XslTransform class, the Load method of the XslCompiledTransform class might perform more slowly than the Load method of the XslTransform class the first time it is called on a transformation. This is because the XSLT file must be compiled before it is loaded. However, if you cache an XslCompiledTransform object for subsequent calls, its Transform method is incredibly faster than the equivalent Transform method of the XslTransform class. Therefore, from a performance perspective:
-
The XslTransform class is the best choice in a “Load once, Transform once” scenario as it doesn’t require the initial map-compilation.
-
The XslCompiledTransform class is the best choice in a “Load once, Cache and Transform many times” scenario as it implies the initial cost for the map-compilation, but then this overhead is highly compensated by the fact that subsequent calls are much faster.
As BizTalk is a server application (or, if you prefer an application server), the second scenario is more likely than the first. The only way to take advantage of this class (given that BizTalk does not currently make use of the XslCompiledTransform class) is to write custom components. If this seems a little strange to you, remember that all BizTalk versions since BizTalk Server 2004 have inherited that core engine, based on .NET Framework 1.1. Since the XslCompiledTransform class wasn’t added until .NET Framework 2.0, it wasn’t leveraged in that version of BizTalk. While I’m currently working with the product group to see how best to take advantage of this class in the next version of BizTalk, let’s go ahead and walk through creating a helper class to boost the performance of message transformations in your current BizTalk implementation using the XslCompiledTransform class and let’s compare its performance with another helper component that makes use the old XslTransform class.
BizTalk Application
In order to compare the performance of the XslTransform and XslCompiledTransform classes I created an easy BizTalk application composed of the following projects:
Helpers
This library contains 2 helpers classes called, respectively, XslTransformHelper and XslCompiledTransformHelper. These components share most of the code and expose the same static methods. I minimized the differences between the 2 classes as the final scope was to compare the performance of the XslTransform and XslCompiledTransform classes. As their name suggests, the first helper class uses the XslTransform class, while the second makes use of the XslCompiledTransform class. The Transform static method of both helper classes provides multiple overloads/variants/signatures. This allows the components to be invoked by any orchestration, pipeline component or .NET class in general. Either classes use a static Dictionary to cache maps in-process for later calls. The fully qualified name (FQDN) of a BizTalk map is used as key to retrieve the value of the corresponding instance within the Dictionary. The fully qualified name (FQDN) of a BizTalk map can be easily determined as follows:
-
Open the BizTalk Administration Console and navigate to the Maps folder within your BizTalk application.
-
Double click the map in question.
-
Copy the content of the Name label (see the picture below) and paste it in a text editor.
-
Append a comma followed by a space (“, “).
-
Copy the content of the Assembly label (see the picture below) and paste it in a text editor.
Pretty easy, don’t you think?
XslTransformHelper class
#region Copyright
//-------------------------------------------------
// Author: Paolo Salvatori
// Email: paolos@microsoft.com
// History: 2010-01-26 Created
//-------------------------------------------------
#endregion
#region Using References
using System;
using System.IO;
using System.Text;
using System.Collections.Generic;
using System.Configuration;
using System.Xml;
using System.Xml.XPath;
using System.Xml.Xsl;
using System.Diagnostics;
using Microsoft.XLANGs.BaseTypes;
using Microsoft.XLANGs.Core;
using Microsoft.BizTalk.Streaming;
using Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Helpers.Properties;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Helpers
{
public class XslTransformHelper
{
#region Private Constants
private const int DefaultBufferSize = 10240; //10 KB
private const int DefaultThresholdSize = 1048576; //1 MB
private const string DefaultPartName = "Body";
#endregion
#region Private Static Fields
private static Dictionary<string, TransformBase> mapDictionary;
#endregion
#region Static Constructor
static XslTransformHelper()
{
mapDictionary = new Dictionary<string, TransformBase>();
}
#endregion
#region Public Static Methods
public static XLANGMessage Transform(XLANGMessage message,
string mapFullyQualifiedName,
string messageName)
{
return Transform(message,
0,
mapFullyQualifiedName,
messageName,
DefaultPartName,
false,
DefaultBufferSize,
DefaultThresholdSize);
}
public static XLANGMessage Transform(XLANGMessage message,
string mapFullyQualifiedName,
string messageName,
bool debug)
{
return Transform(message,
0,
mapFullyQualifiedName,
messageName,
DefaultPartName,
debug,
DefaultBufferSize,
DefaultThresholdSize);
}
public static XLANGMessage Transform(XLANGMessage message,
int partIndex,
string mapFullyQualifiedName,
string messageName,
string partName,
bool debug,
int bufferSize,
int thresholdSize)
{
try
{
using (Stream stream = message[partIndex].RetrieveAs(typeof(Stream)) as Stream)
{
Stream response = Transform(stream, mapFullyQualifiedName, debug, bufferSize, thresholdSize);
CustomBTXMessage customBTXMessage = null;
customBTXMessage = new CustomBTXMessage(messageName, Service.RootService.XlangStore.OwningContext);
customBTXMessage.AddPart(string.Empty, partName);
customBTXMessage[0].LoadFrom(response);
return customBTXMessage.GetMessageWrapperForUserCode();
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException(Resources.XslCompiledTransformHelper, ex);
TraceHelper.WriteLineIf(debug,
null,
ex.Message,
EventLogEntryType.Error);
throw;
}
finally
{
if (message != null)
{
message.Dispose();
}
}
}
public static XLANGMessage Transform(XLANGMessage[] messageArray,
int[] partIndexArray,
string mapFullyQualifiedName,
string messageName,
string partName,
bool debug,
int bufferSize,
int thresholdSize)
{
try
{
if (messageArray != null &&
messageArray.Length > 0)
{
Stream[] streamArray = new Stream[messageArray.Length];
for (int i = 0; i < messageArray.Length; i++)
{
streamArray[i] = messageArray[i][partIndexArray[i]].RetrieveAs(typeof(Stream)) as Stream;
}
Stream response = Transform(streamArray, mapFullyQualifiedName, debug, bufferSize, thresholdSize);
CustomBTXMessage customBTXMessage = null;
customBTXMessage = new CustomBTXMessage(messageName, Service.RootService.XlangStore.OwningContext);
customBTXMessage.AddPart(string.Empty, partName);
customBTXMessage[0].LoadFrom(response);
return customBTXMessage.GetMessageWrapperForUserCode();
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException(Resources.XslCompiledTransformHelper, ex);
TraceHelper.WriteLineIf(debug,
null,
ex.Message,
EventLogEntryType.Error);
throw;
}
finally
{
if (messageArray != null &&
messageArray.Length > 0)
{
for (int i = 0; i < messageArray.Length; i++)
{
if (messageArray[i] != null)
{
messageArray[i].Dispose();
}
}
}
}
return null;
}
public static Stream Transform(Stream stream,
string mapFullyQualifiedName)
{
return Transform(stream,
mapFullyQualifiedName,
false,
DefaultBufferSize,
DefaultThresholdSize);
}
public static Stream Transform(Stream stream,
string mapFullyQualifiedName,
bool debug)
{
return Transform(stream,
mapFullyQualifiedName,
debug,
DefaultBufferSize,
DefaultThresholdSize);
}
public static Stream Transform(Stream stream,
string mapFullyQualifiedName,
bool debug,
int bufferSize,
int thresholdSize)
{
try
{
TransformBase transformBase = GetTransformBase(mapFullyQualifiedName);
if (transformBase != null)
{
VirtualStream virtualStream = new VirtualStream(bufferSize, thresholdSize);
XPathDocument xpathDocument = new XPathDocument(stream);
transformBase.Transform.Transform(xpathDocument, transformBase.TransformArgs, virtualStream);
virtualStream.Seek(0, SeekOrigin.Begin);
return virtualStream;
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException(Resources.DynamicTransformsHelper, ex);
TraceHelper.WriteLineIf(debug,
null,
ex.Message,
EventLogEntryType.Error);
throw;
}
return null;
}
public static Stream Transform(Stream[] streamArray,
string mapFullyQualifiedName)
{
return Transform(streamArray,
mapFullyQualifiedName,
false,
DefaultBufferSize,
DefaultThresholdSize);
}
public static Stream Transform(Stream[] streamArray,
string mapFullyQualifiedName,
bool debug)
{
return Transform(streamArray,
mapFullyQualifiedName,
debug,
DefaultBufferSize,
DefaultThresholdSize);
}
public static Stream Transform(Stream[] streamArray,
string mapFullyQualifiedName,
bool debug,
int bufferSize,
int thresholdSize)
{
try
{
TransformBase transformBase = GetTransformBase(mapFullyQualifiedName);
if (transformBase != null)
{
CompositeStream compositeStream = null;
try
{
VirtualStream virtualStream = new VirtualStream(bufferSize, thresholdSize);
compositeStream = new CompositeStream(streamArray);
XPathDocument xpathDocument = new XPathDocument(compositeStream);
transformBase.Transform.Transform(xpathDocument, transformBase.TransformArgs, virtualStream);
virtualStream.Seek(0, SeekOrigin.Begin);
return virtualStream;
}
finally
{
if (compositeStream != null)
{
compositeStream.Close();
}
}
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException(Resources.DynamicTransformsHelper, ex);
TraceHelper.WriteLineIf(debug,
null,
ex.Message,
EventLogEntryType.Error);
throw;
}
return null;
}
#endregion
#region Private Static Methods
private static TransformBase GetTransformBase(string mapFullyQualifiedName)
{
TransformBase transformBase = null;
lock (mapDictionary)
{
if (!mapDictionary.ContainsKey(mapFullyQualifiedName))
{
Type type = Type.GetType(mapFullyQualifiedName);
transformBase = Activator.CreateInstance(type) as TransformBase;
if (transformBase != null)
{
mapDictionary[mapFullyQualifiedName] = transformBase;
}
}
else
{
transformBase = mapDictionary[mapFullyQualifiedName];
}
}
return transformBase;
}
#endregion
}
}
|
XslCompiledTransformHelper class
#region Copyright
//-------------------------------------------------
// Author: Paolo Salvatori
// Email: paolos@microsoft.com
// History: 2010-01-26 Created
//-------------------------------------------------
#endregion
#region Using References
using System;
using System.IO;
using System.Text;
using System.Collections.Generic;
using System.Configuration;
using System.Xml;
using System.Xml.Xsl;
using System.Xml.XPath;
using System.Diagnostics;
using Microsoft.XLANGs.BaseTypes;
using Microsoft.XLANGs.Core;
using Microsoft.BizTalk.Streaming;
using Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Helpers.Properties;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Helpers
{
public class XslCompiledTransformHelper
{
#region Private Constants
private const int DefaultBufferSize = 10240; //10 KB
private const int DefaultThresholdSize = 1048576; //1 MB
private const string DefaultPartName = "Body";
#endregion
#region Private Static Fields
private static Dictionary<string, MapInfo> mapDictionary;
#endregion
#region Static Constructor
static XslCompiledTransformHelper()
{
mapDictionary = new Dictionary<string, MapInfo>();
}
#endregion
#region Public Static Methods
public static XLANGMessage Transform(XLANGMessage message,
string mapFullyQualifiedName,
string messageName)
{
return Transform(message,
0,
mapFullyQualifiedName,
messageName,
DefaultPartName,
false,
DefaultBufferSize,
DefaultThresholdSize);
}
public static XLANGMessage Transform(XLANGMessage message,
string mapFullyQualifiedName,
string messageName,
bool debug)
{
return Transform(message,
0,
mapFullyQualifiedName,
messageName,
DefaultPartName,
debug,
DefaultBufferSize,
DefaultThresholdSize);
}
public static XLANGMessage Transform(XLANGMessage message,
int partIndex,
string mapFullyQualifiedName,
string messageName,
string partName,
bool debug,
int bufferSize,
int thresholdSize)
{
try
{
using (Stream stream = message[partIndex].RetrieveAs(typeof(Stream)) as Stream)
{
Stream response = Transform(stream, mapFullyQualifiedName, debug, bufferSize, thresholdSize);
CustomBTXMessage customBTXMessage = null;
customBTXMessage = new CustomBTXMessage(messageName, Service.RootService.XlangStore.OwningContext);
customBTXMessage.AddPart(string.Empty, partName);
customBTXMessage[0].LoadFrom(response);
return customBTXMessage.GetMessageWrapperForUserCode();
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException(Resources.XslCompiledTransformHelper, ex);
TraceHelper.WriteLineIf(debug,
null,
ex.Message,
EventLogEntryType.Error);
throw;
}
finally
{
if (message != null)
{
message.Dispose();
}
}
}
public static XLANGMessage Transform(XLANGMessage[] messageArray,
int[] partIndexArray,
string mapFullyQualifiedName,
string messageName,
string partName,
bool debug,
int bufferSize,
int thresholdSize)
{
try
{
if (messageArray != null &&
messageArray.Length > 0)
{
Stream[] streamArray = new Stream[messageArray.Length];
for (int i = 0; i < messageArray.Length; i++)
{
streamArray[i] = messageArray[i][partIndexArray[i]].RetrieveAs(typeof(Stream)) as Stream;
}
Stream response = Transform(streamArray, mapFullyQualifiedName, debug, bufferSize, thresholdSize);
CustomBTXMessage customBTXMessage = null;
customBTXMessage = new CustomBTXMessage(messageName, Service.RootService.XlangStore.OwningContext);
customBTXMessage.AddPart(string.Empty, partName);
customBTXMessage[0].LoadFrom(response);
return customBTXMessage.GetMessageWrapperForUserCode();
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException(Resources.XslCompiledTransformHelper, ex);
TraceHelper.WriteLineIf(debug,
null,
ex.Message,
EventLogEntryType.Error);
throw;
}
finally
{
if (messageArray != null &&
messageArray.Length > 0)
{
for (int i = 0; i < messageArray.Length; i++)
{
if (messageArray[i] != null)
{
messageArray[i].Dispose();
}
}
}
}
return null;
}
public static Stream Transform(Stream stream,
string mapFullyQualifiedName)
{
return Transform(stream,
mapFullyQualifiedName,
false,
DefaultBufferSize,
DefaultThresholdSize);
}
public static Stream Transform(Stream stream,
string mapFullyQualifiedName,
bool debug)
{
return Transform(stream,
mapFullyQualifiedName,
debug,
DefaultBufferSize,
DefaultThresholdSize);
}
public static Stream Transform(Stream stream,
string mapFullyQualifiedName,
bool debug,
int bufferSize,
int thresholdSize)
{
try
{
MapInfo mapInfo = GetMapInfo(mapFullyQualifiedName, debug);
if (mapInfo != null)
{
XmlTextReader xmlTextReader = null;
try
{
VirtualStream virtualStream = new VirtualStream(bufferSize, thresholdSize);
xmlTextReader = new XmlTextReader(stream);
mapInfo.Xsl.Transform(xmlTextReader, mapInfo.Arguments, virtualStream);
virtualStream.Seek(0, SeekOrigin.Begin);
return virtualStream;
}
finally
{
if (xmlTextReader != null)
{
xmlTextReader.Close();
}
}
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException(Resources.XslCompiledTransformHelper, ex);
TraceHelper.WriteLineIf(debug,
null,
ex.Message,
EventLogEntryType.Error);
throw;
}
return null;
}
public static Stream Transform(Stream[] streamArray,
string mapFullyQualifiedName)
{
return Transform(streamArray,
mapFullyQualifiedName,
false,
DefaultBufferSize,
DefaultThresholdSize);
}
public static Stream Transform(Stream[] streamArray,
string mapFullyQualifiedName,
bool debug)
{
return Transform(streamArray,
mapFullyQualifiedName,
debug,
DefaultBufferSize,
DefaultThresholdSize);
}
public static Stream Transform(Stream[] streamArray,
string mapFullyQualifiedName,
bool debug,
int bufferSize,
int thresholdSize)
{
try
{
MapInfo mapInfo = GetMapInfo(mapFullyQualifiedName, debug);
if (mapInfo != null)
{
CompositeStream compositeStream = null;
try
{
VirtualStream virtualStream = new VirtualStream(bufferSize, thresholdSize);
compositeStream = new CompositeStream(streamArray);
XmlTextReader reader = new XmlTextReader(compositeStream);
mapInfo.Xsl.Transform(reader, mapInfo.Arguments, virtualStream);
virtualStream.Seek(0, SeekOrigin.Begin);
return virtualStream;
}
finally
{
if (compositeStream != null)
{
compositeStream.Close();
}
}
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException(Resources.XslCompiledTransformHelper, ex);
TraceHelper.WriteLineIf(debug,
null,
ex.Message,
EventLogEntryType.Error);
throw;
}
return null;
}
#endregion
#region Private Static Methods
private static MapInfo GetMapInfo(string mapFullyQualifiedName,
bool debug)
{
MapInfo mapInfo = null;
lock (mapDictionary)
{
if (!mapDictionary.ContainsKey(mapFullyQualifiedName))
{
Type type = Type.GetType(mapFullyQualifiedName);
TransformBase transformBase = Activator.CreateInstance(type) as TransformBase;
if (transformBase != null)
{
XslCompiledTransform map = new XslCompiledTransform(debug);
using (StringReader stringReader = new StringReader(transformBase.XmlContent))
{
XmlTextReader xmlTextReader = null;
try
{
xmlTextReader = new XmlTextReader(stringReader);
XsltSettings settings = new XsltSettings(true, true);
map.Load(xmlTextReader, settings, new XmlUrlResolver());
mapInfo = new MapInfo(map, transformBase.TransformArgs);
mapDictionary[mapFullyQualifiedName] = mapInfo;
}
finally
{
if (xmlTextReader != null)
{
xmlTextReader.Close();
}
}
}
}
}
else
{
mapInfo = mapDictionary[mapFullyQualifiedName];
}
}
return mapInfo;
}
#endregion
}
public class MapInfo
{
#region Private Fields
private XslCompiledTransform xsl;
private XsltArgumentList arguments;
#endregion
#region Public Constructors
public MapInfo()
{
this.xsl = null;
this.arguments = null;
}
public MapInfo(XslCompiledTransform xsl,
XsltArgumentList arguments)
{
this.xsl = xsl;
this.arguments = arguments;
}
#endregion
#region Public Properties
public XslCompiledTransform Xsl
{
get
{
return this.xsl;
}
set
{
this.xsl = value;
}
}
public XsltArgumentList Arguments
{
get
{
return this.arguments;
}
set
{
this.arguments = value;
}
}
#endregion
}
}
|
Note: Support for embedded scripts is an optional XSLT setting on the XslCompiledTransform class. Script support is disabled by default. Therefore, to enable script support, it’s necessary to create an XsltSettings object with the EnableScript property set to true and pass the object to the Load method. That’s what I did in my code above.
Schemas
This project contains 2 Xml Schemas, CalculatorRequest and CalculatorResponse, which define, respectively, the request and response message and a PropertySchema that defines the Method promoted property. A CalculatorRequest message can contain zero or multiple Operation elements, as shown in the following picture:
CalculatorRequest message
<CalculatorRequest xmlns="http://microsoft.biztalk.cat/10/dynamictransforms/calculatorrequest">
<Method>UnitTest</Method>
<Operations>
<Operation>
<Operator>+</Operator>
<Operand1>82</Operand1>
<Operand2>18</Operand2>
</Operation>
<Operation>
<Operator>-</Operator>
<Operand1>30</Operand1>
<Operand2>12</Operand2>
</Operation>
<Operation>
<Operator>*</Operator>
<Operand1>25</Operand1>
<Operand2>8</Operand2>
</Operation>
<Operation>
<Operator>\</Operator>
<Operand1>100</Operand1>
<Operand2>25</Operand2>
</Operation>
</Operations>
</CalculatorRequest>
|
A CalculatorResponse message contains a Result element for each Operation element within the corresponding CalculatorRequest message, as shown in the following picture:
CalculatorResponse message
<CalculatorResponse xmlns="http://microsoft.biztalk.cat/10/dynamictransforms/calculatorresponse">
<Status>Ok</Status>
<Results>
<Result>
<Value>100</Value>
<Error>None</Error>
</Result>
<Result>
<Value>18</Value>
<Error>None</Error>
</Result>
<Result>
<Value>200</Value>
<Error>None</Error>
</Result>
<Result>
<Value>4</Value>
<Error>None</Error>
</Result>
</Results>
</CalculatorResponse>
|
Maps
This project contains the CalculatorRequestToCalculatorResponse map (see the picture below) that transforms an inbound request message into the corresponding response message.
Orchestrations
This project contains the 4 orchestrations.
SingleDynamicTransform Test Case
This flow had been created just to test the XslCompiledTransformHelper class within an orchestration.
The following picture depicts the architecture of the SingleDynamicTransform test case.
Message Flow:
-
A One-Way FILE Receive Location receives a new CalculatorRequest xml document from the IN folder.
-
The XML disassembler component within the XMLTransmit pipeline promotes the Method element inside the CalculatorRequest xml document. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
-
The inbound request starts a new instance of the SingleDynamicTransform. This latter uses a Direct Bound Port and a Filter to receive only the CalculatorRequest messages with the Method promoted property = “SingleDynamicTransform”.
-
The SingleDynamicTransform invokes the Transform static method exposed by the XslCompiledTransformHelper class to apply the CalculatorRequestToCalculatorResponse map to the inbound CalculatorRequest message and generate the corresponding CalculatorResponse document.
-
The CalculatorRequestToCalculatorResponse publishes the CalculatorResponse message to the MessageBox (BizTalkMsgBoxDb).
-
The response message is retrieved by a One-Way FILE Send Port.
-
The response message is written to an OUT folder by the One-Way FILE Send Port.
DefaultStaticLoop Test Case
As shown in the picture below, this orchestration receives a CalculatorRequest xml document (80KB) and executes a loop (1000 iterations) in which it uses a Transform Shape to apply the CalculatorRequestToCalculatorResponse map to the inbound message. The orchestration does not produce any response message. The code within the StartStepTrace and EndStepTrace Expression Shapes keeps track of the time spent to execute the map at each iteration, while the code contained in the final Trace Expression Shape writes the total elapsed time on the standard output. The objective of this test case is to measure the time spent by the orchestration to apply the map to the inbound document 1000 times using the Transform Shape.
The following picture depicts the architecture of the DefaultStaticLoop test case.
Message Flow:
-
A One-Way FILE Receive Location receives a new CalculatorRequest xml document from the IN folder.
-
The XML disassembler component within the XMLTransmit pipeline promotes the Method element inside the CalculatorRequest xml document. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
-
The inbound request starts a new instance of the DefaultStaticLoop . This latter uses a Direct Bound Port and a Filter to receive only the CalculatorRequest messages with the Method promoted property = “DefaultStaticLoop”.
-
The DefaultStaticLoop executes a loop (1000 iterations) in which it uses a Transform Shape to apply the CalculatorRequestToCalculatorResponse map to the inbound CalculatorRequest message (80KB).
DefaultDynamicLoop Test Case
This component is a variation of the DefaultStaticLoop orchestration. As this latter, it receives a CalculatorRequest xml document (80KB) and executes a loop (1000 iterations), but it doesn’t use a Transform shape to execute the CalculatorRequestToCalculatorResponse map against the inbound message, it rather uses a Message Assignment Shape that contain the following code. See How to Use Expressions to Dynamic Transform Messages for more information on this topic. The objective of this test case is to measure the time spent by the orchestration to apply the map to the inbound document 1000 times using the transform statement provided by the XLANG Runtime.
startTime = System.DateTime.Now;
type = System.Type.GetType("<Map FQDN>");
transform(CalculatorResponse) = type(CalculatorRequest);
stopTime = System.DateTime.Now;
elapsedTime = stopTime.Subtract(startTime);
total = total + elapsedTime.TotalMilliseconds;
i = i + 1;
|
As the DefaultStaticLoop, the orchestration does not produce any response. The code within the CreateResponse Shape keeps track of the time spent to execute the map at each iteration, while the code contained in the final Trace Expression Shape writes the total elapsed time on the standard output.
The following picture depicts the architecture of the DefaultDynamicLoop test case.
Message Flow:
-
A One-Way FILE Receive Location receives a new CalculatorRequest xml document from the IN folder.
-
The XML disassembler component within the XMLTransmit pipeline promotes the Method element inside the CalculatorRequest xml document. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
-
The inbound request starts a new instance of the DefaultDynamicLoop . This latter uses a Direct Bound Port and a Filter to receive only the CalculatorRequest messages with the Method promoted property = “DefaultDynamicLoop”.
-
The DefaultDynamicLoop executes a loop (1000 iterations) in which it uses a Message Assignment Shape to execute the CalculatorRequestToCalculatorResponse map against the inbound CalculatorRequest message (80KB).
CustomDynamicLoop Test Case
As the previous orchestrations, the CustomDynamicLoop receives a CalculatorRequest xml document (80KB) and executes a loop (1000 iterations). However, instead of using a Transform shape or the Dynamic Transformation mechanism provided by BizTalk to apply the map to the inbound document, it uses an Expression Shape (see the code below) to invoke the Transform method exposed by my XslCompiledTransformHelper component. The objective of this test case is to measure the time spent by the orchestration to apply the map to the inbound document 1000 times using the XslCompiledTransformHelper class.
startTime = System.DateTime.Now;
CalculatorResponse = Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Helpers.XslCompiledTransformHelper.Transform(CalculatorRequest, "<Map FQDN>");
stopTime = System.DateTime.Now;
elapsedTime = stopTime.Subtract(startTime);
total = total + elapsedTime.TotalMilliseconds;
i = i + 1;
|
As the previous orchestrations, the CustomDynamicLoop does not produce any response. The code within the final Trace Expression Shape writes the total elapsed time on the standard output.
The following picture depicts the architecture of the CustomDynamicLoop test case.
Message Flow:
-
A One-Way FILE Receive Location receives a new CalculatorRequest xml document from the IN folder.
-
The XML disassembler component within the XMLTransmit pipeline promotes the Method element inside the CalculatorRequest xml document. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
-
The inbound request starts a new instance of the CustomDynamicLoop. This latter uses a Direct Bound Port and a Filter to receive only the CalculatorRequest messages with the Method promoted property = “CustomDynamicLoop”.
-
The CustomDynamicLoop executes a loop (1000 iterations) in which it uses a XslCompiledTransformHelper class to execute the CalculatorRequestToCalculatorResponse map against the inbound CalculatorRequest message (80KB).
Pipeline Components
This project contains 2 custom pipeline components called, respectively, TransformPipelineComponent and LoopbackPipelineComponent.
TransformPipelineComponent
This component can be used within a Receive or a Send custom pipeline to transform the inbound message using the XslCompiledTransformHelper class. For the sake of brevity, we just report the code of the Execute method of in the picture below. Note that the if the loopback property exposed by the component equals true, this latter promotes the RouteDirectToTp context property to true. This way, when the TransformPipelineComponent is used by a Receive Pipeline within a Request-Response Receive Location, when the Message Agent posts the transformed message to the MessageBox, this latter is immediately returned as a response to the Receive Location (Loopback pattern).
Execute method
public IBaseMessage Execute(IPipelineContext context, IBaseMessage message)
{
try
{
if (componentEnabled)
{
if (context == null)
{
throw new ArgumentException("The pipeline context parameter cannot be null.");
}
if (message != null)
{
IBaseMessagePart bodyPart = message.BodyPart;
Stream inboundStream = bodyPart.GetOriginalDataStream();
Stream outboundStream = XslCompiledTransformHelper.Transform(inboundStream, mapFQDN, traceEnabled, bufferSize, thresholdSize);
bodyPart.Data = outboundStream;
context.ResourceTracker.AddResource(inboundStream);
context.ResourceTracker.AddResource(outboundStream);
if (loopback)
{
message.Context.Promote("RouteDirectToTP", "http://schemas.microsoft.com/BizTalk/2003/system-properties", true);
}
}
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException("TransformPipelineComponent", ex);
TraceHelper.WriteLineIf(traceEnabled,
context,
ex.Message,
EventLogEntryType.Error);
}
return message;
}
|
LoopbackPipelineComponent
This component can be used to set the RouteDirectToTp context property to true to implement the Loopback pattern. When used within a Receive Pipeline, the component allows to promote the MessageType property without the need to use an Xml Disassembler. At runtime, the MessageType is mandatory to determine the map to apply to a given message on a Request or Send Port.
Execute method
public IBaseMessage Execute(IPipelineContext context, IBaseMessage message)
{
try
{
if (loopback)
{
message.Context.Promote("RouteDirectToTP", "http://schemas.microsoft.com/BizTalk/2003/system-properties", true);
if (messageType != null)
{
message.Context.Promote("MessageType", "http://schemas.microsoft.com/BizTalk/2003/system-properties", messageType);
}
}
}
catch (Exception ex)
{
ExceptionHelper.HandleException("LoopbackPipelineComponent", ex);
TraceHelper.WriteLineIf(traceEnabled,
context,
ex.Message,
EventLogEntryType.Error);
}
return message;
}
|
Pipelines
This project contains 2 custom pipelines:
Then, I created 2 use cases to compare the performance of the default message transformation provided by BizTalk Messaging Engine and the message transformation accomplished using my XslCompiledTransformHelper class.
TransformStaticallyDefined Test Case
The following picture depicts the architecture of the TransformStaticallyDefined test case.
Message Flow:
-
The DT.TransformStaticallyDefined.WCF-NetTcp.RL WCF-NetTcp Request-Response Receive Location receives a CalculatorRequest xml document submitted running the InvokeStaticMap Unit Test within Visual Studio.
-
The LoopbackReceivePipeline promotes the RouteDirectToTp property to true and the MessageType property. I could have used the Xml Disassembler component within the Receive Pipeline to find and promote the MessageType, but I preferred to specify the MessageType of the inbound message as part of the configuration of the Receive Location (see the picture below). This way I can avoid the overhead introduced by the Xml Disassembler component and measure just the time spent by the Messaging Engine to apply the CalculatorRequestToCalculatorResponse map statically defined on the Receive Port. Once transformed the CalculatorRequest message into a CalculatorResponse document, the Message Agent posts this latter to the MessageBox.
-
The transformed message is immediately returned to the Receive Location.
-
The response message is returned to the InvokeStaticMap Unit Test.
DT.TransformStaticallyDefined.RP Configuration
The screen below shows that the use of the CalculatorRequestToCalculatorResponse map has been statically configured on the DT.TransformStaticallyDefined.RP Receive Port.
DT.TransformStaticallyDefined.WCF-NetTcp.RL Configuration
The following picture shows the configuration of the LoopbackReceivePipeline on the DT.TransformStaticallyDefined.WCF-NetTcp.RL Receive Location.
TransformReceivePipeline Test Case
The following picture depicts the architecture of the TransformReceivePipeline test case.
Message Flow:
-
The DT.TransformReceivePipeline.WCF-NetTcp.RL WCF-NetTcp Request-Response Receive Location receives a CalculatorRequest xml document submitted running the InvokeDynamicMap Unit Test within Visual Studio.
-
The TransformPipelineComponent (the following picture shows its configuration) promotes the RouteDirectToTp property to true and transforms the inbound message using the XslCompiledTransformHelper class and the CalculatorRequestToCalculatorResponse map. Then the Message Agent posts the transformed message to the MessageBox.
-
The transformed message is immediately returned to the Receive Location.
-
The response message is returned to the InvokeDynamicMap Unit Test.
DT.TransformReceivePipeline.WCF-NetTcp.RL Configuration
The following picture shows the configuration of the TransformReceivePipeline on the DT.TransformReceivePipeline.WCF-NetTcp.RL Receive Location.
UnitAndLoadTests
Finally, I created a Test Project called UnitAndLoadTests that contains a small set of unit and load tests described below:
- TestXslTransformHelper: this unit test can be used to measure the time spent to execute loops transformations using the XslTransformHelper class, where loops is defined in the configuration file. The following picture reports the code of the TestXslTransformHelper unit test.
TestXslTransformHelper method
[TestMethod]
public void TestXslTransformHelper()
{
Assert.AreNotEqual<string>(null,
inputFile,
"The inpuFile key in the configuration file cannot be null.");
Assert.AreNotEqual<string>(String.Empty,
inputFile,
"The inpuFile key in the configuration file cannot be empty.");
Assert.AreEqual<bool>(true,
File.Exists(inputFile),
string.Format(CultureInfo.CurrentCulture, "The {0} file does not exist.", inputFile));
Assert.AreNotEqual<string>(null,
mapFullyQualifiedName,
"The mapFullyQualifiedName key in the configuration file cannot be null.");
Assert.AreNotEqual<string>(String.Empty,
mapFullyQualifiedName,
"The mapFullyQualifiedName key in the configuration file cannot be empty.");
if (traceResponses)
{
Assert.AreEqual<bool>(true,
Directory.Exists(outputFolder),
string.Format(CultureInfo.CurrentCulture, "The {0} folder does not exist.", outputFolder));
}
Type type = null;
try
{
type = Type.GetType(mapFullyQualifiedName);
}
catch (Exception ex)
{
Assert.Fail(ex.Message);
}
MemoryStream stream = null;
string message;
using (StreamReader reader = new StreamReader(File.Open(inputFile, FileMode.Open, FileAccess.Read, FileShare.Read)))
{
message = reader.ReadToEnd();
}
byte[] buffer = Encoding.UTF8.GetBytes(message);
Stopwatch stopwatch = new Stopwatch();
Stream output = null;
TestContext.BeginTimer("TestXslTransformHelper");
for (int i = 0; i < loops; i++)
{
stream = new MemoryStream(buffer);
stopwatch.Start();
output = XslTransformHelper.Transform(stream, mapFullyQualifiedName);
stopwatch.Stop();
if (output != null && traceResponses)
{
using (StreamReader reader = new StreamReader(output))
{
message = reader.ReadToEnd();
}
using (StreamWriter writer =
new StreamWriter(File.OpenWrite(
Path.Combine(outputFolder,
string.Format(CultureInfo.CurrentCulture, "{{{0}}}.xml", Guid.NewGuid().ToString())))))
{
writer.Write(message);
writer.Flush();
}
}
}
TestContext.EndTimer("TestXslTransformHelper");
Trace.WriteLine(String.Format(CultureInfo.CurrentCulture,
"[TestXslTransformHelper] Loops: {0} Elapsed Time (milliseconds): {1}", loops, stopwatch.ElapsedMilliseconds));
}
|
- TestXslCompiledTransformHelper: this unit test can be used to measure the time spent to execute loops transformations using the XslCompiledTransformHelper class, where loops is defined in the configuration file. For the sake of brevity, I omitted to include the code of the TestXslCompiledTransformHelper unit test as this latter is very similar to one of the previous unit test.
- InvokeStaticMap: this unit test can be used to send a single CalculatorRequest xml document to the DT.TransformStaticallyDefined.WCF-NetTcp.RL Receive Location used by the TransformStaticallyDefined Test case.
- InvokeDynamicMap: this unit test can be used to send a single CalculatorRequest xml document to the DT.TransformReceivePipeline.WCF-NetTcp.RL Receive Location used by the TransformReceivePipeline Test case.
- StaticMapLoadTest: this load test is based on the InvokeStaticMap unit test and can be used to generate traffic against the TransformStaticallyDefined Use Case.
- DynamicMapLoadTest: this load test is based on the InvokeDynamicMap unit test and can be used to generate traffic against the TransformReceivePipeline Use Case.
All these tests share the same configuration contained in the App.config configuration file. In particular this latter contains the following information:
-
The WCF Endpoint used to invoke the DT.TransformStaticallyDefined.WCF-NetTcp.RL and DT.TransformReceivePipeline.WCF-NetTcp.RL Receive Locations.
-
The appSettings section defines multiple keys that allows to control the runtime behavior of unit and load tests:
-
mapFullyQualifiedName: contains the name of the map used by TestXslTransformHelper and TestXslCompiledTransformHelper unit tests.
-
inputFile: defines the path of the inbound document used by all unit tests (TestXslTransformHelper , TestXslCompiledTransformHelper, InvokeStaticMap, InvokeDynamicMap).
-
outputFolder: indicates the path where to save response messages.
-
traceResponses: indicates whether to save response messages.
-
loops: allows to control the number of loop iterations performed by the TestXslTransformHelper and TestXslCompiledTransformHelper unit tests.
For the sake of completeness, I include below the App.config I used for my tests.
App.config file
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<system.serviceModel>
<!-- Bindings used by client endpoints -->
<bindings>
<netTcpBinding>
<binding name="netTcpBinding"
closeTimeout="01:10:00"
openTimeout="01:10:00"
receiveTimeout="01:10:00"
sendTimeout="01:10:00"
transactionFlow="false"
transferMode="Buffered"
transactionProtocol="OleTransactions"
hostNameComparisonMode="StrongWildcard"
listenBacklog="100"
maxBufferPoolSize="1048576"
maxBufferSize="10485760"
maxConnections="200"
maxReceivedMessageSize="10485760">
<readerQuotas maxDepth="32"
maxStringContentLength="8192"
maxArrayLength="16384"
maxBytesPerRead="4096"
maxNameTableCharCount="16384" />
<reliableSession ordered="true"
inactivityTimeout="00:10:00"
enabled="false" />
<security mode="None">
<transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
<message clientCredentialType="Windows" />
</security>
</binding>
</netTcpBinding>
</bindings>
<client>
<!-- Client endpoints used by client excahnge messages with the WCF Receive Locations -->
<endpoint address="net.tcp://localhost:3816/dynamictransforms"
binding="netTcpBinding"
bindingConfiguration="netTcpBinding"
contract="System.ServiceModel.Channels.IRequestChannel"
name="StaticMapEndpoint" />
<endpoint address="net.tcp://localhost:3817/dynamictransforms"
binding="netTcpBinding"
bindingConfiguration="netTcpBinding"
contract="System.ServiceModel.Channels.IRequestChannel"
name="DynamicMapEndpoint" />
</client>
</system.serviceModel>
<appSettings>
<add key="mapFullyQualifiedName" value="Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Maps.CalculatorRequestToCalculatorResponse,
Microsoft.BizTalk.CAT.Samples.DynamicTransforms.Maps, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8c83cae5bc47edb0"/>
<add key="inputFile" value="C:\Projects\DynamicTransforms\Test\UnitTest.xml"/>
<add key="outputFolder" value="C:\Projects\DynamicTransforms\Test\Out"/>
<add key="traceResponses" value="false"/>
<add key="loops" value="1000"/>
</appSettings>
</configuration>
|
Results
Let’s start running some of the test cases and unit tests I created. Take into account that the unit tests and that you can find in the code attached to the article are parametric and they can be executed using any xml message and map. Therefore, I strongly encourage you to repeat my tests using your own messages and maps.
TestXslTransformHelper vs TestXslCompiledTransformHelper
I configured both the unit tests to execute the CalculatorRequestToCalculatorResponse map against the the UnitTest.xml file (80KB) 1000 times. Each test method uses an instance of the Stopwatch class to measure the time spent to executing all calls and finally traces a message containing the total elapsed time. The screens below were taken within Visual Studio at the end of the 2 tests.
TestXslTransformHelper
TestXslCompiledTransformHelper
The difference in terms of performance between the 2 unit tests is simply astonishing:
- TestXslTransformHelper Unit Test: Total Elapsed Time = ~144 seconds, Average Elapsed Time/Transformation = ~144 milliseconds
- TestXslCompiledTransformHelper Unit Test: Total Elapsed Time = ~3.5 seconds, Average Elapsed Time/Transformation = ~3.5 milliseconds
Obviously, I conducted several test runs and they all confirmed that the XslCompiledTransformHelper is class incredibly faster than the XslTransformHelper class and this clearly demonstrates that the XslCompiledTransform class is absolutely much better than the XslTransform class in a Load once, Cache and Transform many times” scenario.
DefaultStaticLoop Test Case vs DefaultDynamicLoop Test Case vs CustomDynamicLoop Test Case
All the orchestrations used in the 3 test cases share the same structure and implement the same behavior using a different technique:
- DefaultStaticLoop orchestration: uses a Transform shape to execute the CalculatorRequestToCalculatorResponse against the inbound document.
- DefaultDynamicLoop orchestration: uses the transform method within a Message Assignment Shape to accomplish the same task.
- CustomDynamicLoop orchestration: uses the XslCompiledTransformHelper.Transform method to invoke the map against against the request message.
Each orchestration contains a loop that executes the message transformation exactly 1000 times and finally reports the total elapsed time. For the test I created 3 separate xml files (they can be found in the Test folder), one for each orchestration. As I explained in the first part of the article, each orchestration receives the request message through a Direct Bound Port. In particular, the following Filter Expression has been defined on the Activate Receive Shape of each orchestration:
Therefore, the following files are identical:
-
DefaultStaticLoop.xml
-
DefaultDynamicLoop.xml
-
CustomDynamicLoop.xml
with the exception of the Method element that contains the name of the related orchestration. To execute each test case is sufficient to copy the corresponding file to the Test\IN folder: the DT.FILE.RL FILE Receive Location will than receive the message and activate the intended test case. I used DebugView to keep track of the elapsed time reported by each of the test cases:
Results are quite eloquent and don’t give room to doubts:
- DefaultStaticLoop Test Case: Total Elapsed Time = ~57.3 seconds, Average Elapsed Time/Transformation = ~57 milliseconds
- DefaultDynamicLoop Test Case: Total Elapsed Time = ~56.2 seconds, Average Elapsed Time/Transformation = ~56 milliseconds
- CustomDynamicLoop Test Case: Total Elapsed Time = ~3.6 seconds, Average Elapsed Time/Transformation = ~3.6 milliseconds
Once again, I conducted several test runs to confirm the results obtained and reported above. These latter clearly demonstrated that the XslCompiledTransformHelper class is an order of magnitude faster than the default mechanisms provided by BizTalk for transforming messages.
TransformStaticallyDefined vs TransformReceivePipeline
The objective of this test is to compare the performance of the following test cases:
-
TransformStaticallyDefined Test Case: as explained in the first part of the article, the inbound CalculatorRequest message is transformed using the CalculatorRequestToCalculatorResponse map declaratively configured on the DT.TransformStaticallyDefined.RP Request Port. Once posted to the MessageBox, the CalculatorResponse transformed message is immediately returned to the DT.TransformStaticallyDefined.WCF-NetTcp.RL Receive Location (Loopback pattern).
-
TransformReceivePipeline Test Case: the inbound CalculatorRequest message is transformed by the TransformReceivePipeline hosted by the DT.TransformStaticallyDefined.WCF-NetTcp.RL Receive Location. In particular, the TransformPipelineComponent invokes the XslCompiledTransformHelper.Transform static method to apply the CalculatorRequestToCalculatorResponse map to the inbound xml document. The FQDN of the map is declaratively specified in the pipeline configuration. Once posted to the MessageBox, the CalculatorResponse transformed message is immediately returned to the DT.TransformStaticallyDefined.WCF-NetTcp.RL Receive Location (Loopback pattern).
To generate traffic against the 2 test cases and measure performance I used the following Load Tests defined in the UnitAndLoadTests Test Project:
-
StaticMapLoadTest: this load test is based on the InvokeStaticMap unit test and can be used to generate traffic against the TransformStaticallyDefined Use Case. The test is configured to send 1000 CalculatorRequest messages to the DT.TransformStaticallyDefined.WCF-NetTcp.RL Receive Location using 25 different worker threads.
-
DynamicMapLoadTest: this load test is based on the InvokeDynamicMap unit test and can be used to generate traffic against the TransformReceivePipeline Use Case. The test is configured to send 1000 CalculatorRequest messages to the DT.TransformStaticallyDefined.WCF-NetTcp.RL Receive Location using 25 different worker threads.
In particular, as shown in the picture below, I created a custom Counter Set called BizTalk composed of the following performance counters:
-
Inbound Latency (sec): measures the average latency in milliseconds from when the Messaging Engine receives a document from the adapter until the time it is published to Message Box.
-
Request-Response Latency (sec): measures the average latency in milliseconds from when the Messaging Engine receives a request document from the adapter until the time a response document is given back to the adapter.
Specifically, the average latency measured by the Inbound Latency (sec) counter includes the time spent for transforming the message in both use cases. Obviously it counts also the time spent running other activities like posting the message to the MessageBox, but still it represents a good mechanism to measure to compare the time spent by the 2 test cases for transforming the inbound message.
I conducted several test runs to confirm results obtained. The screens below were taken, respectively, at the end of StaticMapLoadTest and DynamicMapLoadTest:
StaticMapLoadTest Graphs & Summary
DynamicMapLoadTest Graphs & Summary
The following table reports for convenience the results highlighted in the screens above:
| Test Case |
Inbound Latency (sec) |
Request Response Latency (sec) |
Avg Test Time (sec) |
Tests/sec (Throughput) |
Duration (sec) |
% CPU Time |
|
| StaticMapLoadTest |
0.41 |
0.90 |
2.55 |
8.77 |
114 |
65.2 |
| DynamicMapLoadTest |
0.11 |
0.49 |
1.29 |
17.6 |
56 |
43 |
The difference in terms of latency and throughput between the 2 test cases is quite dramatic and this clearly confirms once again that the XslCompiledTransform class is much faster than the XslTransform class natively used by BizTalk. In our case, the adoption of the custom XslCompiledTransformHelper class allowed to double the throughput and halve the latency. Obviously, the performance gain can vary from case to case as it depends on many factors (inbound message size, map complexity, etc.), but it’s quite evident that the overall performance of a BizTalk application that makes an extensive use of message transformations can greatly be improved using a helper component like the XslCompiledTransformHelper class that exploits the XslCompiledTransform class to compile, invoke and cache maps for later calls.
Conclusions
As I said in the first part of the article, I started to work with the product group to see how best to take advantage of the XslCompiledTransform class in the next version of BizTalk. Nevertheless, you can immediately exploits this class in your custom components to boost the execution of you message transformations. Therefore, I encourage you to download my code here and repeat the tests described in this article using your own messages and maps.
Follow-Up
I wrote another article on this subject and extended my code to support multi-source-document-maps. You can find my post here.
Code
Here you can download the code. Any feedback is highly appreciated. 😉
by community-syndication | Jun 23, 2010 | BizTalk Community Blogs via Syndication
Introduction
In the following post I’ll explain in detail a demo that I presented at TechReady 9 and TechEd 2009 in Berlin.
The Problem
Consider the following scenario:
Your BizTalk application receives an XML request message through a two-way Receive Location (e.g. WCF, SOAP, HTTP). The inbound document contains multiple elements. For each item contained in the request message, the application needs to synchronously invoke a downstream WCF service that returns a response message. The application needs to process all the elements contained in the request document as a unit of work and aggregate the result of individual calls in a single response message that will be returned to the original caller.
Let’s make an example to better scope the problem. Let’s assume that our BizTalk application is exposed via a two-way WCF Receive Location and receives a request message, as the one shown below, containing multiple Operation elements.
CalculatorRequest Message
<CalculatorRequest xmlns="http://Microsoft.BizTalk.CAT.Samples.CalculatorRequest">
<Method>XmlDocumentOrchestration</Method>
<Operations>
<Operation>
<Operator>+</Operator>
<Operand1>82</Operand1>
<Operand2>18</Operand2>
</Operation>
<Operation>
<Operator>-</Operator>
<Operand1>30</Operand1>
<Operand2>12</Operand2>
</Operation>
<Operation>
<Operator>*</Operator>
<Operand1>25</Operand1>
<Operand2>8</Operand2>
</Operation>
<Operation>
<Operator>\</Operator>
<Operand1>100</Operand1>
<Operand2>25</Operand2>
</Operation>
<Operation>
<Operator>+</Operator>
<Operand1>100</Operand1>
<Operand2>32</Operand2>
</Operation>
</Operations>
</CalculatorRequest>
|
For each Operation element contained in the inbound XML document the application has to read the child Operator element and accordingly invoke one of the four methods (Add, Subtract, Multiply, Divide) exposed by the downstream WCF service. This latter is a variation of the CalculatorService that can be found on MSDN between the WCF samples. The return value of the individual calls to the CalculatorService must be aggregated in a single response message, as the one shown below. This latter will be finally returned to the original caller.
CalculatorResponse Message
<CalculatorResponse xmlns="http://Microsoft.BizTalk.CAT.Samples.CalculatorResponse">
<Status>Ok</Status>
<Results>
<Result>
<Value>100</Value>
<Error>None</Error>
</Result>
<Result>
<Value>18</Value>
<Error>None</Error>
</Result>
<Result>
<Value>200</Value>
<Error>None</Error>
</Result>
<Result>
<Value>4</Value>
<Error>None</Error>
</Result>
<Result>
<Value>132</Value>
<Error>None</Error>
</Result>
</Results>
</CalculatorResponse>
|
The easier way to solve the above problem is to create a BizTalk application that exploits an orchestration to implement the Scatter-Gather design pattern. The orchestration in question should perform the following steps:
- Receive the request message.
- Extract and loop through the individual Operation elements, for example using the XPath function.
- Use a Decide shape with 4 branches, one for each possible value of the Operator element.
- Within each branch, apply a transformation map and invoke one of the 4 Operations (Add, Subtract, Multiply, Divide) exposed by a two-way Solicit-Response Logical Port.
- Receive the response message from the underlying WCF service.
- At each iteration, save the result in a collection variable.
- Create and return a response message containing results.
The following graph depicts the architecture of the solution described above. In the remainder of this post, I’ll refer to this solution as LogicalPortsOrchestration.
Message Flow:
-
A WCF-BasicHttp or WCF-Custom Request-Response Receive Location receives a new CalculatorRequest xml document from a Test Agent.
-
The XML disassembler component within the XMLTransmit pipeline promotes the Method element inside the CalculatorRequest xml document. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
-
The inbound request starts a new instance of the LogicalPortsOrchestration. This latter uses a Direct Port receive the CalculatorRequest messages with the Method promoted property = “LogicalPortsOrchestration”.
-
The LogicalPortsOrchestration uses a loop to retrieve operations and for each item it invokes the downstream Calculator WCF service using a Logical Solicit-Response Port bound to a Physical Solicit-Response WCF-BasicHttp Send Port. The request message for the Calculator WCF web service is created using an helper component and published to the (BizTalkMsgBoxDb).
-
The request message is consumed by a WCF-BasicHttp Send Port.
-
The WCF-BasicHttp Send Port invokes one of the methods (Add, Subtract, Multiply, Divide) exposed by the Calculator WCF service.
-
The Calculator WCF service returns a response message.
-
The response message is published to the MessageBox (BizTalkMsgBoxDb).
-
The response message is returned to the caller LogicalPortsOrchestration. The orchestration repeats this pattern for each operation within the inbound CalculatorRequest xml document.
-
The LogicalPortsOrchestration creates and publishes the CalculatorResponse message to the MessageBox (BizTalkMsgBoxDb).
-
The response message is retrieved by the Request-Response WCF-BasicHttp or WCF-Custom Receive Location.
-
The response message is returned to the Test Agent.
The solution described above works fine but as you can easily note it requires 4 distinct roundtrips to the MessageBox for each individual call to the underlying WCF service and therefore this architectural approach is not adequate when performance goals in terms of latency and throughput are very aggressive. More in general, whenever possible you should minimize the use of orchestrations and privilege messaging only patterns to increase the overall throughput and reduce the latency of the business processes. However, the Gather-Scatter and Service Composition are 2 scenarios that can be easily implemented using an orchestration, so the question is: how can I solve the same problem using a different pattern?
An alternative solution could be using the Inline Send design pattern to solve the problem: in this case to invoke the underlying CalculatorService our orchestration would not use a Logical Port bound to Physical WCF Send Port, but it rather use a custom WCF proxy component within an Expression shape that would replace the Send and Receive shapes. Note that not using Adapters and Physical Send Ports, the application would not benefit from their functional capabilities such as batching, retries, correlation sets initialization, declarative configuration and secondary transports. The advantage of the Inline Send technique is that for each Solicit Response call made by the orchestration, it allows to eliminate 4 roundtrips to the MessageBox:
- Orchestration Logical Port –> Request –> MessageBox
- MessageBox –> Request –> Physical Send Port
- Physical Send Port –> Response –> MessageBox
- MessageBox –> Response –> Orchestration Logical Port
The following graph depicts the architecture of the solution based on the Inline Send pattern. In the remainder of this post, I’ll refer to this solution as InlineSendOrchestration.
Message Flow:
-
A WCF-BasicHttp or WCF-Custom Request-Response Receive Location receives a new CalculatorRequest xml document from a Test Agent.
-
The XML disassembler component within the XMLTransmit pipeline promotes the Method element inside the CalculatorRequest xml document. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
-
The inbound request starts a new instance of the InlineSendOrchestration. This latter uses a Direct Port receive the CalculatorRequest messages with the Method promoted property = “InlineSendOrchestration”.
-
The InlineSendOrchestration uses a loop to retrieve operations and for each item it invokes the downstream Calculator WCF service using a generic WCF proxy component called InlineSendProxy. The endpoint used by this object must be configured in the BtsNtSvc.exe.config and BtsNtSvc64.exe.config configurations files. See the BtsNtSvc.exe.config file in the Setup folder for a sample of how configuring this component to access the downstream Calculator WCF web service.
-
The InlineSendProxy component directly invokes one of the methods (Add, Subtract, Multiply, Divide) exposed by the Calculator WCF service.
-
The Calculator WCF service returns a response message.
-
The response message is written in the outbound CalculatorResponse document using an XmlWriter and a VirtualStream objects. The orchestration repeats this pattern for each operation within the inbound CalculatorRequest xml document.
-
The InlineSendOrchestration publishes the CalculatorResponse message to the MessageBox (BizTalkMsgBoxDb).
-
The response message is retrieved by the Request-Response WCF-BasicHttp or WCF-Custom Receive Location.
-
The response message is returned to the Test Agent.
In February 2009, the BizTalk Customer Advisory Team conducted a performance lab to determine and compare the performance in terms of latency and throughput of the LogicalPortsOrchestration and InlineSendOrchestration patterns. To this purpose, we used a lab rig composed of 2 BizTalk Server 2009 (Beta version) nodes (2 Dual-Core CPUs, 8 GB RAM), a SQL Server 2008 machine to host the MessageBox. The following table reports the results obtained using a request message that contained 5 distinct Operations:
| Test Case |
Concurrent Test Client Users |
Messages/Second |
Average Response Time (Sec) |
| LogicalPortsOrchestration |
100 |
60.64 |
1.61 |
| InlineSendOrchestration |
100 |
236.96 |
0.24 |
As you can easily note reviewing the numbers reported in the table above, the InlineSendOrchestration implementation provides better performance than the LogicalPortsOrchestration solution, both in terms of latency and throughput. Therefore, a question arises naturally: should I renounce to the capabilities provided by Adapters and apply the Inline Send pattern to debatch an incoming message and implement the Scatter-Gather pattern in a performant way? Fortunately, the answer is no as there’s another architectural approach to solve the problem using a messaging-only approach that does not require any orchestration.
The following picture depicts the architecture of the third solution that I’ll refer to as MessagingOnly. Unfortunately, this solution was developed after the performance lab, so no comparative tests were conducted to compare its latency and throughput with the 2 preceding implementations. Nevertheless, at the end of this article I provide a link to the code of the 3 implementations, so you can eventually run your own tests and eventually customize the solution to fit your needs.
Message Flow:
-
A WCF-BasicHttp or WCF-Custom Request-Response Receive Location receives a new CalculatorRequest xml document from a Test Agent.
-
The XML disassembler component within the XMLTransmit pipeline promotes the Method element inside the CalculatorRequest xml document. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
-
The inbound request is consumed by the POG.MessagingOnly.WCF-Custom.SendPort WCF-Custom Send Port. This latter uses a Filter Expression to receive CalculatorRequest messages with the Method promoted property = “MessagingOnly”.
-
The WCF-Custom Send Port uses a Custom Binding which contains the following binding elements:
CalculatorServiceBindingElement,
TextMessageEncodingBindingElement,
HttpTransportBindingElement. The custom
CalculatorServiceBindingElement at runtime creates a
ServiceFactoryChannelFactory object that in turn creates a
CalculatorServiceRequestChannel object. The
Request method of this latter debatches the
Operation elements contained in the inbound
CalculatorRequest message, and for each of them makes a separate call to the downstream
CalculatorService using the reference to the inner
HttpTransportChannel.
-
The
CalculatorService returns a response message. The custom channel executed by the WCF Adapter repeats this pattern for each
Operation element within the inbound
CalculatorRequest xml document. The custom channel stores the return value of individual calls in a collection variable and finally uses results to generate the
CalculatorResponse message (
Scatter-Gather pattern) that is finally returned by the WCF-Custom Adapter.
-
The POG.MessagingOnly.WCF-Custom.SendPort WCF-Custom Send Port publishes the CalculatorResponse message to the MessageBox (BizTalkMsgBoxDb).
-
The response message is retrieved by the Request-Response WCF-BasicHttp or WCF-Custom Receive Location.
-
The response message is returned to the Test Agent.
As I have noted in my previous article, WCF Bindings provide a mechanism for configuring channel stacks. In other words, a binding defines a precise recipe for building a channel stack using a transport channel, a message encoder, and a set of protocol channels. WCF ships with several built-in bindings that target common communication scenarios, such as the BasicHttpBinding, WsHttpBinding and NetTcpBinding and BizTalk Server provides a full range of WCF Adapters that correspond 1:1 to the most commonly used WCF bindings:
- WCF-BasicHttp
- WCF-WSHttp
- WCF-NetTcp
- etc.
However, if you need maximum flexibility and you need to use one or multiple custom protocol channels at runtime, you can use the CustomBinding that gives you the possibility to control which binding elements compose your binding. Likewise, in a BizTalk application, if you want to extend the default behavior of the WCF Adapters with a custom component (Service Behavior, Endpoint Behavior, Custom Binding, Custom Binding Element), you have to use the WCF-Custom and/or WCF-CustomIsolated Adapters. In fact, these latter offer you complete control over the channel stack and behaviors configuration, and as a consequence, they are the only WCF adapters you really need. Compared with the other WCF Adapters, they are the only ones to provide the possibility to:
- Implement and exploit extensibility points.
- Have full access to properties exposed by bindings/behaviors.
- Enable the use of the bamInterceptor endpoint behavior.
- Export/Import the binding configuration.
- Disable a receive location on failure.
- Run an http-based RL within an in-process host.
- Use bindings (e.g. wsDualHttpBinding) for which a WCF Adapter does not exist.
The Code
Before looking at the code of the custom channel, let’s complicate the scenario as follows: let’s assume that the underlying WCF Calculator service invoked by the BizTalk application is hosted by IIS 7.0 and exposes an endpoint that uses a CustomBinding composed of the following binding elements:
-
TransactionFlowBindingElement
-
TextMessageEncodingBindingElement
-
HttpTransportBindingElement
The following picture reports the web.config used by the Calculator service:
Web.config
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<connectionStrings>
<add name="LogDB"
providerName="System.Data.SqlClient"
connectionString="Data Source=.;Initial Catalog=LogDB;Integrated Security=SSPI;" />
</connectionStrings>
<system.diagnostics>
<sources>
<source name="System.ServiceModel.MessageLogging" switchValue="Warning, ActivityTracing">
<listeners>
<add type="System.Diagnostics.DefaultTraceListener" name="Default">
<filter type="" />
</add>
<add name="ServiceModelMessageLoggingListener">
<filter type="" />
</add>
</listeners>
</source>
</sources>
<sharedListeners>
<add initializeData="C:\calculatorservicetxn.svclog"
type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
name="ServiceModelMessageLoggingListener"
traceOutputOptions="Timestamp">
<filter type="" />
</add>
</sharedListeners>
</system.diagnostics>
<system.serviceModel>
<diagnostics performanceCounters="All">
<messageLogging logEntireMessage="false"
logMalformedMessages="false"
logMessagesAtServiceLevel="false"
logMessagesAtTransportLevel="false"
maxSizeOfMessageToLog="1000000" />
</diagnostics>
<!-- Service Endpoints -->
<services>
<service behaviorConfiguration="CalculatorServiceBehavior"
name="Microsoft.ServiceModel.Samples.CalculatorService">
<endpoint address=""
binding="customBinding"
bindingConfiguration="customBinding"
name="customBinding"
contract="Microsoft.ServiceModel.Samples.ICalculator" />
</service>
</services>
<!-- Binding Condfiguration -->
<bindings>
<customBinding>
<binding name="customBinding">
<transactionFlow transactionProtocol="WSAtomicTransactionOctober2004" />
<textMessageEncoding />
<httpTransport />
</binding>
</customBinding>
</bindings>
<!-- Service Behaviors -->
<behaviors>
<serviceBehaviors>
<behavior name="CalculatorServiceBehavior">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="true" />
<serviceThrottling maxConcurrentCalls="200"
maxConcurrentSessions="200"
maxConcurrentInstances="200" />
</behavior>
</serviceBehaviors>
</behaviors>
</system.serviceModel>
<system.web>
<customErrors mode="RemoteOnly"/>
<trust level="Full" />
</system.web>
</configuration>
|
The Calculator service requires the client application to create and flow a transaction using the WS-AtomicTransaction protocol. In particular, the WCF service will use this client-initiated transaction to log the data of the invoked operation to a SQL database. To implement this behavior, all the service operations exposed by the ICalculatorService contract interface have been decorated with the TransactionFlowAttribute. In particular, the TransactionFlowOption.Mandatory value has been used in the constructor of the TransactionFlow attribute to indicate that the client application must initiate and transmit a transaction, otherwise the service operation will immediately throw an exception.
In the CalculatorService class, all the service operations have been decorated with the OperationBehaviorAttribute. In particular, the value of the TransactionScopeRequired and TransactionAutoComplete properties has been set to true. This indicates that service operations require a transaction scope for their execution and this latter automatically completes if no unhandled exceptions occur and the method returns successfully.
All the service operations invoke the RecordToLog method that logs the current operation data and transactionId to the LogDB on SQL Server. This sample on MSDN demonstrates various aspects of creating a transactional service and the use of a client-initiated transaction to coordinate service operations.
Note in particular that the Transaction.Current.TransactionInformation.DistributedIdentifier static property is used by the RecordToLog method to retrieve the unique identifier of the client-initiated distributed transaction.
CalculatorService Class
#region Using Directives
using System;
using System.Diagnostics;
using System.ServiceModel;
using System.Transactions;
using System.Configuration;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
#endregion
namespace Microsoft.ServiceModel.Samples
{
// Define a service contract.
[ServiceContract(Namespace = "http://Microsoft.ServiceModel.Samples")]
public interface ICalculator
{
#region Contract Operations
[OperationContract]
[TransactionFlow(TransactionFlowOption.Mandatory)]
double Add(double n1, double n2);
[OperationContract]
[TransactionFlow(TransactionFlowOption.Mandatory)]
double Subtract(double n1, double n2);
[OperationContract]
[TransactionFlow(TransactionFlowOption.Mandatory)]
double Multiply(double n1, double n2);
[OperationContract]
[TransactionFlow(TransactionFlowOption.Mandatory)]
double Divide(double n1, double n2);
#endregion
}
// Service class which implements the service contract.
[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Single)]
public class CalculatorService : ICalculator
{
#region Private Constants
private const string AttemptedToDivideByZero = "Attempted to divide by zero.";
private const string TransactionIdFormat = "[CalculatorService] TransactionId: {0}";
private const string CallCompletedFormat = "[CalculatorService] Message [{0}] stored to LogDB.";
private const string MessageFormat = "[CalculatorService] {0}";
private const string InsertLogEntrySP = "usp_InsertLogEntry";
private const string IdParameter = "@Id";
private const string MessageParameter = "@Message";
private const string TypeParameter = "@Type";
private const string SourceParameter = "@Source";
private const string TransactionIdParameter = "@TransactionID";
private const string Information = "Information";
private const string Source = "CalculatorService";
private const string None = "None";
private const string AddFormat = "{0} + {1} = {2}";
private const string SubtractFormat = "{0} - {1} = {2}";
private const string MultiplyFormat = "{0} * {1} = {2}";
private const string DivideFormat = "{0} / {1} = {2}";
#endregion
#region Private Static Fields
private static string connectionString = null;
#endregion
#region Static Constructor
static CalculatorService()
{
try
{
if (ConfigurationManager.ConnectionStrings != null &&
ConfigurationManager.ConnectionStrings.Count > 0)
{
connectionString = ConfigurationManager.ConnectionStrings["LogDB"].ConnectionString;
}
}
catch (Exception ex)
{
Trace.WriteLine(string.Format(MessageFormat, ex.Message));
}
}
#endregion
#region Public Methods
[OperationBehavior(TransactionScopeRequired = true, TransactionAutoComplete = true)]
public double Add(double n1, double n2)
{
double n3 = n1 + n2;
RecordToLog(String.Format(AddFormat, n1, n2, n3));
return n3;
}
[OperationBehavior(TransactionScopeRequired = true, TransactionAutoComplete = true)]
public double Subtract(double n1, double n2)
{
double n3 = n1 - n2;
RecordToLog(String.Format(SubtractFormat, n1, n2, n3));
return n3;
}
[OperationBehavior(TransactionScopeRequired = true, TransactionAutoComplete = true)]
public double Multiply(double n1, double n2)
{
double n3 = n1 * n2;
RecordToLog(String.Format(MultiplyFormat, n1, n2, n3));
return n3;
}
[OperationBehavior(TransactionScopeRequired = true, TransactionAutoComplete = true)]
public double Divide(double n1, double n2)
{
if (n2 == 0)
{
Trace.WriteLine(AttemptedToDivideByZero);
throw new DivideByZeroException();
}
double n3 = n1 / n2;
RecordToLog(String.Format(DivideFormat, n1, n2, n3));
return n3;
}
#endregion
#region Private Static Methods
private static void RecordToLog(string message)
{
try
{
if (!string.IsNullOrEmpty(connectionString))
{
string transactionId = None;
if (Transaction.Current != null &&
Transaction.Current.TransactionInformation != null)
{
transactionId = Transaction.Current.TransactionInformation.DistributedIdentifier.ToString();
}
Trace.WriteLine(string.Format(TransactionIdFormat, transactionId));
using (SqlConnection sqlConnection = new SqlConnection(connectionString))
{
sqlConnection.Open();
SqlParameter sqlParameter = null;
using (SqlCommand sqlCommand = new SqlCommand(InsertLogEntrySP, sqlConnection))
{
sqlParameter = new SqlParameter(IdParameter, SqlDbType.UniqueIdentifier);
sqlParameter.Direction = ParameterDirection.Input;
sqlParameter.Value = Guid.NewGuid();
sqlCommand.Parameters.Add(sqlParameter);
sqlParameter = new SqlParameter(MessageParameter, SqlDbType.VarChar, 512);
sqlParameter.Direction = ParameterDirection.Input;
sqlParameter.Value = message.Length <= 512 ? message : message.Substring(0, 512);
sqlCommand.Parameters.Add(sqlParameter);
sqlParameter = new SqlParameter(TypeParameter, SqlDbType.VarChar, 64);
sqlParameter.Direction = ParameterDirection.Input;
sqlParameter.Value = Information;
sqlCommand.Parameters.Add(sqlParameter);
sqlParameter = new SqlParameter(SourceParameter, SqlDbType.VarChar, 64);
sqlParameter.Direction = ParameterDirection.Input;
sqlParameter.Value = Source;
sqlCommand.Parameters.Add(sqlParameter);
sqlParameter = new SqlParameter(TransactionIdParameter, SqlDbType.VarChar, 64);
sqlParameter.Direction = ParameterDirection.Input;
sqlParameter.Value = transactionId;
sqlCommand.Parameters.Add(sqlParameter);
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.ExecuteNonQuery();
Trace.WriteLine(string.Format(CallCompletedFormat, message));
}
}
}
}
catch (Exception ex)
{
Trace.WriteLine(string.Format(MessageFormat, ex.Message));
}
}
#endregion
}
}
|
Now let’s assume that all the operations contained in a CalculatorRequest document must be processed as a unit of work within the same distributed transaction. In this case, the BizTalk application will have to perform the following steps:
-
Initiate a MSDTC transaction.
-
Debatch the incoming CalculatorRequest document to extract individual Operation elements.
-
For each Operation, create a request message and invoke the corresponding operation exposed by the CalculatorService, making sure to flow the distributed transaction as part of each service call.
The following graph depicts the architecture of the described solution.
Message Flow:
-
A WCF-BasicHttp or WCF-Custom Request-Response Receive Location receives a new CalculatorRequest xml document from a Test Agent.
-
The XML disassembler component within the XMLTransmit pipeline promotes the Method element inside the CalculatorRequest xml document. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
-
The inbound request is consumed by the POG.MessagingOnly.WCF-Custom.SendPort WCF-Custom Send Port. This latter uses a Filter Expression to receive CalculatorRequest messages with the Method promoted property = “MessagingOnly”.
-
The WCF-Custom Send Port uses a Custom Binding which contains the following binding elements:
TransactionFlowBindingElement,
CalculatorServiceBindingElement,
TextMessageEncodingBindingElement,
HttpTransportBindingElement. The
TransactionFlowBindingElement at runtime creates a
TransactionFlowChannel that creates the
CoordinationContext header that is used to coordinate the distributed transaction across BizTalk and the WCF service. The custom
CalculatorServiceBindingElement at runtime creates a
ServiceFactoryChannelFactory object that in turn creates a
CalculatorServiceRequestChannel object. The
Request method of this latter debatches the
Operation elements contained in the inbound
CalculatorRequest message, and for each of them creates a new WCF message, copies to it the
CoordinationContext header from the
CalculatorRequest message, and makes a separate call to the downstream
CalculatorService using the reference to the inner
HttpTransportChannel.
-
The CalculatorService logs the operation data to the LogDB on SQL Server using the BizTalk-initiated transaction.
-
The
CalculatorService returns a response message. The custom channel executed by the WCF Adapter repeats this pattern for each
Operation element within the inbound
CalculatorRequest xml document. The custom channel stores the return value of individual calls in a collection variable and finally uses results to generate the
CalculatorResponse message (
Scatter-Gather pattern) that is finally returned by the WCF-Custom Adapter.
-
The POG.MessagingOnly.WCF-Custom.SendPort WCF-Custom Send Port publishes the CalculatorResponse message to the MessageBox (BizTalkMsgBoxDb).
-
The response message is retrieved by the Request-Response WCF-BasicHttp or WCF-Custom Receive Location.
-
The response message is returned to the Test Agent.
On the left hand side, the following picture shows the custom binding used by the WCF-Custom Send Port, while on the right hand side, it describes more in detail the individual steps performed by the WCF-Custom Send Port.
Message Flow:
- The BizTalk Messaging Runtime passes the CalculatorRequest xml document to the WCF-Custom Adapter.
- The WCF-Custom Adapter generates a WCF Message instance from the IBaseMessage object. The WCF Message contains a SOAP Envelope and in particular the Body element contains the CalculatorRequest as payload.
- The TransactionRequestChannel creates the CoordinationContext header that is used to coordinate the distributed transaction across BizTalk and the CalculatorService.
-
The
CalculatorServiceRequestChannel debatches the first
operation element contained in the inbound
CalculatorRequest message, copies the
CoordinationContext header from the original batch message to the new message and makes a separate call to the downstream
CalculatorService using the reference to the inner
HttpTransportChannel.
-
The CalculatorServiceRequestChannel stores the return value of the first call in a collection.
-
The
CalculatorServiceRequestChannel debatches the second
operation element contained in the inbound
CalculatorRequest message, copies the
CoordinationContext header from the original batch message to the new message and makes a separate call to the downstream
CalculatorService using the reference to the inner
HttpTransportChannel. The
CalculatorServiceRequestChannel repeats the same pattern for the remaining operation items contained in the inbound batch message.
-
The CalculatorServiceRequestChannel stores the return value of the second call in a collection.
-
The
CalculatorServiceRequestChannel finally uses the result value of individual calls to generate the
CalculatorResponse message (
Scatter-Gather pattern) that is finally returned by the WCF-Custom Adapter to the BizTalk Messaging Runtime.
Let’s look at the code. To create a custom protocol channel to debatch the request message at runtime within the WCF Send Port I built a Class Library called WCFExtensionLibrary composed of multiple components. As we noted in the first part of this article, on the sending side, a binding is used to build a IChannelFactory, which in turn builds a channel stack and returns a reference to the top channel in the stack. The application can then use this channel to send messages. A binding consists of an ordered set of binding elements that inherit from the BindingElement class. Each binding element at runtime creates a IChannelFactory, that in turn creates a channel that can be used to process the message in a given point of the channel stack. BizTalk developers can see the protocol channels that compose a channel stack as the pipeline components within a pipeline on a Receive Location or Send Port, and the transport channel as the transport Adapter used to receive or send a message. The only difference between the 2 approaches is that to create a custom pipeline component it’s necessary to create a single class, while to create a custom channel at runtime it’s necessary to create the multiple classes. Let’s see in detail the components that I created to register, configure, and execute my custom channel within the WCF-Custom Send Port.
CalculatorServiceBindingElement Class
This class inherits from the BindingElement class. This custom binding element exposes the following properties:
-
TraceEnabled: Gets or Sets a value indicating whether tracing is enabled.
-
MaxBufferSize: Gets or sets the maximum size for a buffer used the custom channel when creating a WCF message.
-
MessageVersion: Gets or sets the SOAP message and WS-Addressing versions that are expected by the target WCF service. This information is used at runtime by the custom channel to specify the version of request messages to send to the CalculatorService.
The BuildChannelFactory method initializes the CalculatorServiceChannelFactory that is responsible for creating an instance of the CalculatorServiceRequestChannel at runtime.
CalculatorServiceBindingElement Class
#region Using Directives
using System;
using System.ServiceModel.Channels;
using System.ServiceModel;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.WCFExtensionLibrary
{
public class CalculatorServiceBindingElement : BindingElement
{
#region Private Fields
private bool traceEnabled = true;
private int maxBufferSize = 2097152;
private MessageVersionEnum messageVersion = MessageVersionEnum.Default;
#endregion
#region Public Constructors
public CalculatorServiceBindingElement()
{
}
protected CalculatorServiceBindingElement(CalculatorServiceBindingElement other)
: base(other)
{
this.traceEnabled = other.traceEnabled;
this.maxBufferSize = other.maxBufferSize;
this.messageVersion = other.messageVersion;
}
#endregion
#region Public Methods
public override BindingElement Clone()
{
return new CalculatorServiceBindingElement(this);
}
public override bool CanBuildChannelFactory<TChannel>(BindingContext context)
{
return context.CanBuildInnerChannelFactory<TChannel>();
}
public override bool CanBuildChannelListener<TChannel>(BindingContext context)
{
return context.CanBuildInnerChannelListener<TChannel>();
}
public override IChannelFactory<TChannel> BuildChannelFactory<TChannel>(BindingContext context)
{
return new CalculatorServiceChannelFactory<TChannel>(context, traceEnabled, maxBufferSize, messageVersion);
}
public override IChannelListener<TChannel> BuildChannelListener<TChannel>(BindingContext context)
{
return new CalculatorServiceChannelListener<TChannel>(context, traceEnabled, maxBufferSize, messageVersion);
}
public override T GetProperty<T>(BindingContext context)
{
return context.GetInnerProperty<T>();
}
#endregion
#region Public Properties
public bool TraceEnabled
{
get
{
return this.traceEnabled;
}
set
{
this.traceEnabled = value;
}
}
public int MaxBufferSize
{
get
{
return this.maxBufferSize;
}
set
{
this.maxBufferSize = value;
}
}
public MessageVersionEnum MessageVersion
{
get
{
return this.messageVersion;
}
set
{
this.messageVersion = value;
}
}
#endregion
}
}
|
CalculatorServiceBindingExtensionElement Class
This class inherits from the BindingElementExtensionElement class. In general, a BindingElementExtensionElement enables the use of a custom BindingElement implementation from a machine or application configuration file. In our case, the CalculatorServiceBindingExtensionElement has been created to achieve two results:
-
Register the CalculatorServiceBindingElement in the machine.config to make it visible to BizTalk when defining a WCF-Custom/WCF-CustomIsolated Receive Location or a WCF-Custom Send Port.
-
Define the properties that can be set in a declarative way when configuring a WCF-Custom/WCF-CustomIsolated Receive Location or a WCF-Custom Send Port.
The CalculatorServiceBindingExtensionElement class exposes the same properties as the CalculatorServiceBindingElement. The CreateBindingElement method creates an instance of CalculatorServiceBindingElement class and assign a value to its properties. In practice, the following classes:
-
CalculatorServiceBindingExtensionElement
-
CalculatorServiceBindingElement
-
CalculatorServiceChannelFactory
-
CalculatorServiceRequestChannel
all expose the same properties. As we’ll see later on, these latter can be set when defining a WCF-Custom/WCF-CustomIsolated Receive Location or a WCF-Custom Send Port and at runtime they are propagated from the custom BindingElementExtensionElement to the custom channel as follows:
BindingElementExtensionElement –> CalculatorServiceBindingElement–>CalculatorServiceChannelFactory –> CalculatorServiceRequestChannel
CalculatorServiceBindingExtensionElement Class
namespace Microsoft.BizTalk.CAT.Samples.WCFExtensionLibrary
{
public enum MessageVersionEnum
{
Default,
None,
Soap11,
Soap11WSAddressing10,
Soap11WSAddressingAugust2004,
Soap12,
Soap12WSAddressing10,
Soap12WSAddressingAugust2004
}
public class CalculatorServiceBindingExtensionElement : BindingElementExtensionElement
{
#region Private Constants
private const string TraceEnabledProperty = "TraceEnabled";
private const string MaxBufferSizeProperty = "MaxBufferSize";
private const string MessageVersionProperty = "MessageVersion";
#endregion
#region Private Fields
private CalculatorServiceBindingElement calculatorServiceBindingElement;
#endregion
#region Public Constructors
public CalculatorServiceBindingExtensionElement()
{
}
#endregion
#region Public Properties
public override Type BindingElementType
{
get
{
return typeof(CalculatorServiceBindingElement);
}
}
#endregion
#region Public Methods
public override void ApplyConfiguration(BindingElement bindingElement)
{
base.ApplyConfiguration(bindingElement);
calculatorServiceBindingElement = (CalculatorServiceBindingElement)bindingElement;
}
#endregion
#region Protected Methods
protected override BindingElement CreateBindingElement()
{
calculatorServiceBindingElement = new CalculatorServiceBindingElement();
calculatorServiceBindingElement.TraceEnabled = this.TraceEnabled;
calculatorServiceBindingElement.MaxBufferSize = this.MaxBufferSize;
calculatorServiceBindingElement.MessageVersion = this.MessageVersion;
this.ApplyConfiguration(calculatorServiceBindingElement);
return calculatorServiceBindingElement;
}
#endregion
#region Configuration Properties
[ConfigurationProperty(TraceEnabledProperty, DefaultValue = true)]
public bool TraceEnabled
{
get
{
return (bool)base[TraceEnabledProperty];
}
set
{
base[TraceEnabledProperty] = value;
}
}
[ConfigurationProperty(MaxBufferSizeProperty, DefaultValue = 2097152)]
[IntegerValidator(MinValue = 0)]
public int MaxBufferSize
{
get
{
return (int)base[MaxBufferSizeProperty];
}
set
{
base[MaxBufferSizeProperty] = value;
}
}
[ConfigurationProperty(MessageVersionProperty, DefaultValue = MessageVersionEnum.Default)]
public MessageVersionEnum MessageVersion
{
get
{
return (MessageVersionEnum)base[MessageVersionProperty];
}
set
{
base[MessageVersionProperty] = value;
}
}
#endregion
}
}
|
CalculatorServiceChannelFactory and CalculatorServiceRequestChannel Classes
The CalculatorServiceChannelFactory inherits from ChannelFactoryBase that, as the name suggests, provides a common base implementation for channel factories on the client to create channels of a specified type connected to a specified address. In particular, at runtime the OnCreateChannel method creates and returns a new instance of the CalculatorServiceRequestChannel class. Before doing that, the method uses the reference to the inner channel factory to create an instance of the inner channel, that in our case is an instance of the HttpTransportChannel class, and then passes this reference in the constructor of the CalculatorServiceRequestChannel. The custom channel will exploit the reference to the http transport channel to send multiple messages to the CalculatorService, one for each Operation element contained in the inbound request message.
The CalculatorServiceRequestChannel is the class responsible for processing the CalculatorRequest message at runtime. The Request method uses a XmlDictionaryReader object to read through and disassembles the Operation elements contained within the inbound xml message and for each of them it creates and submits a separate message to the underlying CalculatorService. The The CreateOperationMessage method copies the headers of the CalculatorRequest message to each individual message sent to WCF service. One of these headers is the CoordinationContext used to coordinate the distributed transaction. To invoke the WCF service, the custom channel invokes the Request method exposed by the inner channel. Individual call responses are stored in a collection called responseList. Finally, the Request method invokes the CreateResponseMessage function to create and return the response message that aggregates the result of individual calls.
CalculatorServiceChannelFactory and CalculatorServiceRequestChannel Classes
#region Using Directives
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.Transactions;
using System.Xml;
using System.Text;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.WCFExtensionLibrary
{
/// <summary>
/// ChannelFactory that performs message inspection
/// </summary>
class CalculatorServiceChannelFactory<TChannel>
: ChannelFactoryBase<TChannel>
{
#region Private Constants
private const string ErrorMessageFormat =
"<CalculatorResponse xmlns=\"http://Microsoft.BizTalk.CAT.Samples.Schemas.CalculatorResponse\"><Status>{0}
</Status></CalculatorResponse>";
private const string CalculatorResponseNamespace = "http://Microsoft.BizTalk.CAT.Samples.Schemas.CalculatorResponse";
private const string GenericErrorMessage = "An error occured while processing the request.";
private const string OperationUnknownErrorMessageFormat = "The operation failed because the operator {0} is unknown.";
private const string OperationFailedErrorMessageFormat = "The operation number {0} returned an error. {1}";
private const string OperationFormat = "[CalculatorServiceRequestChannel] {0} {1} {2} = {3}";
private const string OperationFailed = "[CalculatorServiceRequestChannel] {0} {1} {2} Failed: {3}";
private const string MessageReceived = "[CalculatorServiceRequestChannel] Request message received.";
private const string MessageSuccessfullyProcessed =
"[CalculatorServiceRequestChannel] Response message successfully processed.";
private const string TransactionIdFormat = "[CalculatorServiceRequestChannel] TransactionId: {0}";
#endregion
#region Private Fields
IChannelFactory<TChannel> innerChannelFactory;
private bool traceEnabled = true;
private int maxBufferSize = 2097152;
private MessageVersionEnum messageVersion = MessageVersionEnum.Default;
#endregion
#region Public Constructors
public CalculatorServiceChannelFactory(BindingContext context,
bool traceEnabled,
int maxBufferSize,
MessageVersionEnum messageVersion)
{
this.innerChannelFactory = context.BuildInnerChannelFactory<TChannel>();
this.traceEnabled = traceEnabled;
this.maxBufferSize = maxBufferSize;
this.messageVersion = messageVersion;
if (this.innerChannelFactory == null)
{
throw new InvalidOperationException("CalculatorServiceChannelFactory requires an inner IChannelFactory.");
}
}
#endregion
#region Public Methods
public override T GetProperty<T>()
{
T baseProperty = base.GetProperty<T>();
if (baseProperty != null)
{
return baseProperty;
}
return this.innerChannelFactory.GetProperty<T>();
}
#endregion
#region Protected Methods
protected override void OnOpen(TimeSpan timeout)
{
this.innerChannelFactory.Open(timeout);
}
protected override IAsyncResult OnBeginOpen(TimeSpan timeout, AsyncCallback callback, object state)
{
return this.innerChannelFactory.BeginOpen(timeout, callback, state);
}
protected override void OnEndOpen(IAsyncResult result)
{
this.innerChannelFactory.EndOpen(result);
}
protected override void OnClose(TimeSpan timeout)
{
this.innerChannelFactory.Close(timeout);
}
protected override IAsyncResult OnBeginClose(TimeSpan timeout, AsyncCallback callback, object state)
{
return this.innerChannelFactory.BeginClose(timeout, callback, state);
}
protected override void OnEndClose(IAsyncResult result)
{
this.innerChannelFactory.EndClose(result);
}
protected override TChannel OnCreateChannel(EndpointAddress to, Uri via)
{
TChannel innerChannel = this.innerChannelFactory.CreateChannel(to, via);
if (typeof(TChannel) == typeof(IRequestChannel))
{
return (TChannel)(object)new CalculatorServiceRequestChannel(this,
(IRequestChannel)innerChannel,
traceEnabled,
maxBufferSize,
messageVersion);
}
throw new InvalidOperationException();
}
#endregion
class CalculatorServiceRequestChannel
: CalculatorServiceChannelBase<IRequestChannel>, IRequestChannel
{
#region Private Fields
private bool traceEnabled = true;
private int maxBufferSize = 2097152;
private MessageVersion messageVersion = MessageVersion.Default;
#endregion
#region Public Constructors
public CalculatorServiceRequestChannel(CalculatorServiceChannelFactory<TChannel> factory,
IRequestChannel innerChannel,
bool traceEnabled,
int maxBufferSize,
MessageVersionEnum messageVersion)
: base(factory, innerChannel)
{
this.traceEnabled = traceEnabled;
this.maxBufferSize = maxBufferSize;
switch (messageVersion)
{
case MessageVersionEnum.Default:
this.messageVersion = MessageVersion.Default;
break;
case MessageVersionEnum.None:
this.messageVersion = MessageVersion.None;
break;
case MessageVersionEnum.Soap11:
this.messageVersion = MessageVersion.Soap11;
break;
case MessageVersionEnum.Soap11WSAddressing10:
this.messageVersion = MessageVersion.Soap11WSAddressing10;
break;
case MessageVersionEnum.Soap11WSAddressingAugust2004:
this.messageVersion = MessageVersion.Soap11WSAddressingAugust2004;
break;
case MessageVersionEnum.Soap12:
this.messageVersion = MessageVersion.Soap12;
break;
case MessageVersionEnum.Soap12WSAddressing10:
this.messageVersion = MessageVersion.Soap12WSAddressing10;
break;
case MessageVersionEnum.Soap12WSAddressingAugust2004:
this.messageVersion = MessageVersion.Soap12WSAddressingAugust2004;
break;
default:
this.messageVersion = MessageVersion.Default;
break;
}
}
#endregion
#region Public Properties
public EndpointAddress RemoteAddress
{
get
{
return this.InnerChannel.RemoteAddress;
}
}
public Uri Via
{
get
{
return this.InnerChannel.Via;
}
}
#endregion
#region Public Methods
public IAsyncResult BeginRequest(Message message, AsyncCallback callback, object state)
{
return this.BeginRequest(message, this.DefaultSendTimeout, callback, state);
}
public IAsyncResult BeginRequest(Message message, TimeSpan timeout, AsyncCallback callback, object state)
{
try
{
Message reply = this.Request(message);
return new TypedCompletedAsyncResult<Message>(reply, callback, state);
}
catch (Exception e)
{
throw e;
}
}
public Message EndRequest(IAsyncResult result)
{
TypedCompletedAsyncResult<Message> reply = (TypedCompletedAsyncResult<Message>)result;
return reply.Data;
}
public Message Request(Message message)
{
return this.Request(message, this.DefaultSendTimeout);
}
public Message Request(Message message, TimeSpan timeout)
{
Message reply = null;
if (message != null)
{
Message operationMessage;
List<Response> responseList = new List<Response>();
string op = null;
StringBuilder statusBuilder = new StringBuilder();
string error = null;
double operand1 = 0;
double operand2 = 0;
bool ok = true;
int i = 0;
try
{
Trace.WriteLineIf(traceEnabled, MessageReceived);
if (traceEnabled)
{
string transactionId = "None";
if (Transaction.Current != null &&
Transaction.Current.TransactionInformation != null)
{
transactionId = Transaction.Current.TransactionInformation.DistributedIdentifier.ToString();
}
Trace.WriteLine(string.Format(TransactionIdFormat, transactionId));
}
using (XmlDictionaryReader reader = message.GetReaderAtBodyContents())
{
double result;
while (reader.Read() && ok)
{
if (reader.NodeType == XmlNodeType.Element &&
reader.LocalName == "Operator")
{
error = "None";
op = reader.ReadElementContentAsString();
operand1 = reader.ReadElementContentAsDouble();
operand2 = reader.ReadElementContentAsDouble();
reply = null;
i++;
try
{
switch (op)
{
case "+":
operationMessage = CreateOperationMessage("Add",
operand1, operand2, message);
reply = this.InnerChannel.Request(operationMessage);
break;
case "-":
operationMessage = CreateOperationMessage("Subtract",
operand1, operand2, message);
reply = this.InnerChannel.Request(operationMessage);
break;
case "*":
operationMessage = CreateOperationMessage("Multiply",
operand1, operand2, message);
reply = this.InnerChannel.Request(operationMessage);
break;
case "/":
operationMessage = CreateOperationMessage("Divide",
operand1, operand2, message);
reply = this.InnerChannel.Request(operationMessage);
break;
default:
error = string.Format(OperationUnknownErrorMessageFormat, op);
SetStatus(statusBuilder,
string.Format(OperationFailedErrorMessageFormat, i, error));
ok = false;
break;
}
}
catch (Exception innerException)
{
error = innerException.Message;
if (statusBuilder.Length > 0)
{
statusBuilder.Append(" - ");
}
ok = false;
}
if (reply != null && !reply.IsFault)
{
result = GetValue(reply);
Trace.WriteLineIf(traceEnabled, string.Format(OperationFormat,
operand1, op, operand2, result));
}
else
{
if (reply != null && reply.IsFault)
{
MessageFault fault = MessageFault.CreateFault(reply, maxBufferSize);
error = fault.Reason.ToString();
}
Trace.WriteLineIf(traceEnabled, string.Format(OperationFailed, operand1,
op, operand2, error));
SetStatus(statusBuilder, string.Format(OperationFailedErrorMessageFormat, i, error));
result = 0;
}
responseList.Add(new Response(error, result));
}
}
}
string status = statusBuilder.ToString();
if (string.IsNullOrEmpty(status))
{
status = "Ok";
}
return CreateResponseMessage(status, responseList, null);
}
catch (Exception ex)
{
Trace.WriteLineIf(traceEnabled, ex.Message);
if (statusBuilder.Length > 0)
{
statusBuilder.Append(" - ");
}
statusBuilder.Append(ex.Message);
return null;
}
finally
{
Trace.WriteLineIf(traceEnabled, MessageSuccessfullyProcessed);
}
}
else
{
Trace.WriteLineIf(traceEnabled, String.Format("Message action {0}", message.Headers.Action));
reply = this.InnerChannel.Request(message);
Trace.WriteLineIf(traceEnabled, String.Format("Reply is: {0}", reply));
return reply;
}
}
#endregion
#region Private Methods
private void SetStatus(StringBuilder statusBuilder, string message)
{
if (statusBuilder.Length > 0)
{
statusBuilder.Append(" - ");
}
statusBuilder.Append(message);
}
private Message CreateOperationMessage(string op, double operand1, double operand2, Message message)
{
string action = string.Format("http://Microsoft.ServiceModel.Samples/ICalculator/{0}", op);
Message operationMessage = Message.CreateMessage(messageVersion,
action,
new OperationBodyWriter(op, operand1, operand2));
if (message != null)
{
if (message.Headers != null &&
message.Headers.Count > 0)
{
operationMessage.Headers.Clear();
operationMessage.Headers.CopyHeadersFrom(message);
operationMessage.Headers.Action = action;
}
if (message.Properties != null &&
message.Properties.Count > 0)
{
operationMessage.Properties.CopyProperties(message.Properties);
}
}
if (messageVersion != MessageVersion.Soap11)
{
operationMessage.Headers.MessageId = new UniqueId();
}
return operationMessage;
}
private Message CreateResponseMessage(string status, List<Response> responseList, Message message)
{
Message reply = Message.CreateMessage(MessageVersion.Default,
"*",
new ResponseBodyWriter(status, responseList));
if (message != null)
{
if (message.Properties != null &&
message.Properties.Count > 0)
{
reply.Properties.CopyProperties(message.Properties);
}
}
reply.Headers.MessageId = new UniqueId();
return reply;
}
private double GetValue(Message message)
{
XmlDictionaryReader reader = message.GetReaderAtBodyContents();
while (reader.Read())
{
if (reader.Name == "AddResult" ||
reader.Name == "SubtractResult" ||
reader.Name == "MultiplyResult" ||
reader.Name == "DivideResult")
{
return reader.ReadElementContentAsDouble();
}
}
return 0;
}
#endregion
}
class OperationBodyWriter : BodyWriter
{
#region Private Fields
private string op;
private double operand1;
private double operand2;
#endregion
#region Public Constructors
public OperationBodyWriter(string op, double operand1, double operand2)
: base(false)
{
this.op = op;
this.operand1 = operand1;
this.operand2 = operand2;
}
#endregion
#region Protected Methods
protected override void OnWriteBodyContents(XmlDictionaryWriter writer)
{
writer.WriteStartElement(op, "http://Microsoft.ServiceModel.Samples");
writer.WriteStartElement("n1");
writer.WriteValue(operand1);
writer.WriteEndElement();
writer.WriteStartElement("n2");
writer.WriteValue(operand2);
writer.WriteEndElement();
writer.WriteEndElement();
}
#endregion
}
class ResponseBodyWriter : BodyWriter
{
#region Private Fields
private string status;
private List<Response> responseList;
private Dictionary<int, Response> responseDictionary;
#endregion
#region Public Constructors
public ResponseBodyWriter(string status, List<Response> responseList)
: base(false)
{
this.status = status;
this.responseList = responseList;
}
public ResponseBodyWriter(string status, Dictionary<int, Response> responseDictionary)
: base(false)
{
this.status = status;
this.responseDictionary = responseDictionary;
}
#endregion
#region Protected Methods
protected override void OnWriteBodyContents(XmlDictionaryWriter writer)
{
if (responseList != null)
{
writer.WriteStartElement("CalculatorResponse", CalculatorResponseNamespace);
writer.WriteStartElement("Status", CalculatorResponseNamespace);
writer.WriteString(status);
writer.WriteEndElement();
writer.WriteStartElement("Results", CalculatorResponseNamespace);
for (int i = 0; i < responseList.Count; i++)
{
writer.WriteStartElement("Result", CalculatorResponseNamespace);
writer.WriteStartElement("Value", CalculatorResponseNamespace);
writer.WriteString(responseList[i].Value.ToString());
writer.WriteEndElement();
writer.WriteStartElement("Error", CalculatorResponseNamespace);
writer.WriteString(responseList[i].Error);
writer.WriteEndElement();
writer.WriteEndElement();
}
writer.WriteEndElement();
writer.WriteEndElement();
}
if (responseDictionary != null)
{
writer.WriteStartElement("CalculatorResponse", CalculatorResponseNamespace);
writer.WriteStartElement("Status", CalculatorResponseNamespace);
writer.WriteString(status);
writer.WriteEndElement();
writer.WriteStartElement("Results", CalculatorResponseNamespace);
for (int i = 0; i < responseDictionary.Count; i++)
{
writer.WriteStartElement("Result", CalculatorResponseNamespace);
writer.WriteStartElement("Value", CalculatorResponseNamespace);
writer.WriteString(responseList[i].Value.ToString());
writer.WriteEndElement();
writer.WriteStartElement("Error", CalculatorResponseNamespace);
writer.WriteString(responseDictionary[i].Error);
writer.WriteEndElement();
writer.WriteEndElement();
}
writer.WriteEndElement();
writer.WriteEndElement();
}
}
#endregion
}
public class Response
{
#region Private Fields
private string error;
private double value;
#endregion
#region Public Constructors
public Response()
{
this.error = default(string);
this.value = default(double);
}
public Response(string error, double value)
{
this.error = error;
this.value = value;
}
#endregion
#region Public Properties
public string Error
{
get
{
return this.error;
}
set
{
this.error = value;
}
}
public double Value
{
get
{
return this.value;
}
set
{
this.value = value;
}
}
#endregion
}
public class Operation
{
#region Private Fields
private string op;
private double operand1;
private double operand2;
#endregion
#region Public Constructors
public Operation()
{
this.op = default(string);
this.operand1 = 0;
this.operand1 = 0;
}
public Operation(string op,
double operand1,
double operand2)
{
this.op = op;
this.operand1 = operand1;
this.operand2 = operand2;
}
#endregion
#region Public Properties
public string Operator
{
get
{
return this.op;
}
set
{
this.op = value;
}
}
public double Operand1
{
get
{
return this.operand1;
}
set
{
this.operand1 = value;
}
}
public double Operand2
{
get
{
return this.operand2;
}
set
{
this.operand2 = value;
}
}
#endregion
}
}
}
|
To register the WCFExtensionLibrary assembly containing my custom channel and use this latter within my BizTalk application I have to perform 3 operations:
- Install the assemblies implementing the WCF extensibility points in the global assembly cache (GAC).
- Modify the machine.config file on the BizTalk Server computer.
- Configure the WCF-Custom Send Port by using the BizTalk Server Administration console.
Installing the WCFExtensionLibrary in the GAC
This step is pretty straightforward and can be accomplished using the gacutil tool. You can automatically install your assembly to the GAC whenever you build the class library by including the execution of the gacutil tool in the post-build event command-line of the project, as shown in following picture:
An easy and handy way to verify that the assembly has been successfully installed in the GAC is to use the following command:
gacutil /lr Microsoft.BizTalk.CAT.Samples.WCFExtensionLibrary
|
How to configure the machine.config file for a WCF binding element extension
To accomplish this task you can proceed as follow:
- Use a text editor as the Notepad to open the machine.config that be found in the following folder:
– %windir%\Microsoft.NET\Framework\v2.0.50727\CONFIG on a 32 bit BizTalk Server node.
– %windir%\Microsoft.NET\Framework64\v2.0.50727\CONFIG on a 32 bit BizTalk Server node.
- Add the following element containing the Fully-Qualified Name (FQDN) of the CalculatorServiceBindingExtensionElement class to the <configuration><system.serviceModel>\<extensions><bindingElementExtensions> node:
<add name="calculatorService" type="Microsoft.BizTalk.CAT.Samples.WCFExtensionLibrary.CalculatorServiceBindingExtensionElement,
Microsoft.BizTalk.CAT.Samples.WCFExtensionLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d7f63d8d08d8f3a2" />
|
How to configure the WCF-Custom Send Port by using the BizTalk Admin Console
Using the BizTalk Server Administration console, you can proceed as follows:
-
Create a new Two-Way Solicit-Response Static Send Port and select the WCF-Custom Adapter. Then select the PassThruTransmit as Send Pipeline and the PassThruReceive as Receive Pipeline. They do not contain any pipeline component and do not perform any processing of the message. For this reason, they ensure maximum performance in receiving or sending messages.
- Click the Configure button. Enter the URL of the CalculatorService in the Address (URI) textbox:
- Select the Binding Tab then choose the customBinding from the Binding Type drop-down list. Then right click the Binding panel on the left hand side and select the Add Extension item from the context menu
- Select the transactionFlow and then repeat the same step to select the calculatorService binding element extension element.
-
Click the calculatorService element in the Binding panel. As you can see, you can set a value for each of the properties exposed by the custom binding element extension element in the same way as you can configure the properties of a pipeline component. This technique is very powerful and handy as it allows to control the runtime behavior of your custom channel defining in a declarative way.
-
Click the httpTransport element in the Binding panel. As you can see, you can access and change the value of all the properties (authenticationScheme, transferMode, maxReceivedMessageSize, etc.) exposed by binding element extension element and thus control the runtime behavior of the transport channel.
-
Select the
Message Tab and perform the following steps:
– Select
Path in the
Inbound BizTalk message body section and enter the following expression in the
Body path expression:
/*[local-name()=’Fault’]/*[local-name()=’Detail’]/* | /*[(local-name()=’AddResponse’)] | /*[(local-name()=’CalculatorResponse’)]
The body path expression is evaluated against the immediate child element of the SOAP Body element of an incoming message. This property is valid only for solicit-response ports. This pseudo-XPath expression (its syntax is indeed a subset of the XPath syntax) defines the xml element that will be extracted and returned by the Adapter. I used the Union (|) operator to select the CalculatorResponse element in case of success, while in case of error I specified to extract the Detail element from the SOAP Fault returned by the CalculatorService.
– Select Xml from the Node encoding drop-down list.
– Select the Propagate fault message checkbox to route a fault message to a subscribing application (such as another receive port or orchestration schedule).
– Select Use Transaction and set the Isolation Level to Read Committed. In this way you instruct the WCF-Custom Adapter to initiate a distributed transaction before passing the message to the channel stack. Note: the Isolation Level selected on the Send Port must match the Isolation Level used by the downstream WCF service, otherwise the call will immediately fail.
Running the Sample
For brevity, I’ll assume you have already deployed and configured all the remaining artifacts that compose the solution (e.g. the WCF Receive Location, the CalculatorService, etc.). To test application you can proceed as follows:
- Open SQL Server Management Studio and run the following command in a Query Window to clean up the Log table in the LogDB.
USE LogDb
GO
SET NOCOUNT ON
GO
DELETE Log
GO
|
- Open the DebugView to capture the trace generated by the CalculatorServiceRequestChannel and CalculatorService.
- Open the Calculator Client application, choose MessagingOnly from the Method drop-down list, press the Create button to create a new request document and then the Submit button to send the message to WCF-Custom or WCF-BasicHttp Receive Location exposed by BizTalk Server.
- The CalculatorServiceRequestChannel and CalculatorService produce the following output captured by DebugView:
- As you can see above, the TransactionId traced by the CalculatorServiceRequestChannel is the same traced by the CalculatorService during each call. To confirm that all the service calls participated to the same distributed transaction, I ran the following TSQL script within the Query Window:
USE LogDb
GO
SET NOCOUNT ON
GO
SELECT * FROM Log ORDER BY DateCreated DESC
GO
|
- The picture below reports the data written by the individual calls. Note that the TransactionId is the same in all rows and matches the one traced down by the DebugView.
To verify that all service calls execute in the context of the same transaction, you can run negative test. For example, if you try to submit a message that contains an invalid Operation as 72 / 0, this latter will cause the transaction to abort and as an effect nothing will be traced on the Log table.
Conclusions
Customizing the runtime behavior of WCF Adapters using WCF extensibility points (service behaviors, endpoint behaviors, custom channels, etc.) is a powerful technique as explained in my previous posts. Here you can download the PerformanceOptimizationGuide solution which contains other use cases in addition to the WCF custom channel project described in this post. Feel free to download and customize the code and please let me know your feedbacks. 🙂
by community-syndication | Jun 23, 2010 | BizTalk Community Blogs via Syndication
Introduction
The contents of the following post are taken from a presentation I created and delivered at TechReady 9 in July 2009 and replicated with Stephen Kaufman at Tech-Ed Europe 2009 .
The ABC of WCF
Windows Communication Foundation (WCF) is a runtime and a set of APIs for exchanging messages between components and applications. WCF was designed according to the tenets of service orientation and unified the existing communication technologies like .NET Remoting and ASMX web service into a single programming model and consistent architecture which provides high levels of functionality, interoperability and extensibility. As you know, a typical WCF service can expose one or multiple endpoints. In general, endpoints provide clients access to the functionality offered by a WCF service. Each endpoint consists of 3 properties as depicted in the picture below:
-
An address that indicates where the endpoint can be found.
-
A binding that specifies how a client app can communicate with the endpoint.
-
A contract that identifies the operations exposed by the endpoint.
WCF Architecture: Messaging Runtime
- The WCF runtime is divided into 2 primary layers as shown by the following picture:
The Service Layer aka Service Model defines the mechanisms and attributes used by developers to define and decorate service, message and data contracts.
- The Messaging Layer is responsible for preparing a WCF message for transmission on the send side and produce a WCF message for the dispatcher on the receive side. The messaging layer accomplishes this task using a Channel Stack. This latter is a pipeline of channel components that handle different processing tasks. Each channel stack is composed of exactly one transport channel, one message encoder, and zero or more protocol channels.
It’s the responsibility of the proxy component on the client side and dispatcher component on the service side to mediate and translate between the two layers. In particular, the proxy component transforms .NET method calls into Message objects, whereas the dispatcher component turns WCF Messages into .NET method calls. WCF uses the Message class to model all incoming/outgoing messages within the Messaging Layer. The message represents a a SOAP Envelope, and therefore it’s composed of a payload and a set of headers. A typical WCF communication can be described as follows:
-
The client application creates one or more input parameters. Each of these parameters is defined by a data contract.
-
The client application invokes one of the methods of the service contract exposed by the proxy.
-
The proxy delivers a WCF Message object to the channel stack.
-
At this point each protocol channel has a chance to operate on the message before the transport channel uses a message encoder to transmit the final Message as a sequence of bytes to the target service. Each protocol channel can modify the content or the headers of the message to implement specific functionalities or WS-* protocols like WS-AtomicTransaction, WS-Security.
-
The raw stream of data is transmitted over the wire.
-
On the service side, the transport channel receives the stream of data and uses a message encoder to interpret the bytes and to produce a WCF Message object that can continue up the channel stack. At this point each protocol channel has a chance to work on the message.
-
The final Message is passed to the Dispatcher.
-
The Dispatcher receives the WCF Message from the underlying channel stack, individuates the target service endpoint using the destination address and Action property contained in the Message, deserializes the content of the WCF Message into objects.
-
Finally the target service method is invoked.
WCF Binding Comparison
Bindings provide a mechanism for configuring channel stacks. A binding defines a precise recipe for building a channel stack using a transport channel, a message encoder, and a set of protocol channels.
WCF ships with several built-in bindings that target common communication scenarios.
| Binding Class Name |
Transport |
Message Encoding |
Message Version |
Security Mode |
Reliable Messaging |
Tx Flow* |
| BasicHttpBinding |
HTTP |
Text |
SOAP 1.1 |
None |
X |
X |
| WSHttpBinding |
HTTP |
Text |
SOAP 1.2 WS-A 1.0 |
Message |
Disabled |
WS-AT |
| NetTcpBinding |
TCP |
Binary |
SOAP 1.2 |
Transport |
Disabled |
OleTx |
| NetNamedPipesBinding |
Named Pipes |
Binary |
SOAP 1.2 |
Transport |
X |
OleTx |
| NetMsmqBinding |
MSMQ |
Binary |
SOAP 1.2 |
Message |
X |
X |
| CustomBinding |
You decide |
You decide |
You decide |
You decide |
You decide |
You decide
|
You should select the most appropriate binding based on your needs:
-
For example, BasicHttpBinding is designed for interoperability with simple web services that conform to the SOAP 1.1 message version.
-
Instead the WSHttpBinding is designed for interoperability with more advanced web services that might leverage different WS-* protocols. Both of these bindings use HTTP for the transport and the text message encoding because they are designed for maximum interoperability.
-
NetTcpBinding and NetNamedPipeBinding, on the other hand, are designed for efficient and performant communication with other WCF applications across machines or on the same machine respectively.
-
If you need maximum flexibility and you need to use one or multiple custom protocol channels at runtime, you can use the CustomBinding that gives you the possibility to decide which binding elements compose your binding.
Bindings have different characteristics in terms of response time and throughput, so the general advice to increase performance is using the NetTcpBindind and NetNamesPipeBinding whenever possible.
BizTalkServiceInstance Class
Each WCF Receive Location in BizTalk Server is indeed an instance of a WCF Service Class called BizTalkServiceInstance hosted by a separate instance of a ServiceHost-derived class
When you “enable” a WCF receive location, the adapter initializes and opens the ServiceHost, which dynamically builds the WCF runtime components within the BizTalk service process (BtsNtSvc.exe for in in-process host, w3wp.exe for an isolated host). This includes the channel stack, dispatcher, and the singleton service instance. WCF Receive and Send Ports are message-type agnostic or if you prefer untyped. This design comes in handy when you need to configure a single Receive Port to accept numerous message types or versions that you can normalize (via BizTalk maps or using a custom pipeline component) into a common message type before being posted the BizTalkMsgBoxDb. However, this design also implies that the WCF adapters will need to build on generic service contracts in order to remain message-type agnostic within the WCF implementation.
WCF Receive Locations have the responsibility to receive the incoming message bytes, perform any necessary SOAP and WS-* processing, and publish the message (or some part of it) to the message box. The WCF Adapters create a separate ServiceHost and singleton service object of this class for each receive location to handle client requests for the lifetime of the BizTalk Host instance running WCF receive locations. The service object uses multiple threads to process messages concurrently unless the WCF-NetMsmq receive locations are used with the Ordered processing property being selected.. As shown in the picture below, the class is decorated with the ServiceBehavior attribute:
BizTalkServiceInstance class
[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]
internal sealed class BizTalkServiceInstance : ITwoWayAsync,
ITwoWayAsyncVoid,
IOneWayAsync,
IOneWayAsyncTxn,
ITwoWayAsyncVoidTxn
{
...
}
|
In particular, you can note that:
Hence, all incoming messages to a WCF Receive Location are received and processed by a single well-known instance of the BizTalkServiceInstance class. This allows to avoid service activation/deactivation costs and improve performance/scalability.
The BizTalkServiceInstance class implements multiple untyped, generic service contracts.
Each contract was designed for a different scope (as suggested by their name):
[ServiceContract(Namespace = "http://www.microsoft.com/biztalk/2006/r2/wcf-adapter")]
public interface IOneWayAsync
{
// Methods
[OperationContract(AsyncPattern = true, IsOneWay = true, Action = "*")]
IAsyncResult BeginOneWayMethod(Message message, AsyncCallback callback, object state);
[OperationContract(IsOneWay = true, Action = "BizTalkSubmit")]
void BizTalkSubmit(Message message);
void EndOneWayMethod(IAsyncResult result);
}
[ServiceContract(Namespace = "http://www.microsoft.com/biztalk/2006/r2/wcf-adapter")]
public interface IOneWayAsyncTxn
{
// Methods
[OperationContract(AsyncPattern = true, IsOneWay = true, Action = "*")]
IAsyncResult BeginOneWayMethod(Message message, AsyncCallback callback, object state);
[OperationContract(IsOneWay = true, Action = "BizTalkSubmit")]
void BizTalkSubmit(Message message);
void EndOneWayMethod(IAsyncResult result);
}
[ServiceContract(Namespace = "http://www.microsoft.com/biztalk/2006/r2/wcf-adapter")]
public interface ITwoWayAsync
{
// Methods
[OperationContract(AsyncPattern = true, IsOneWay = false, Action = "*", ReplyAction = "*")]
IAsyncResult BeginTwoWayMethod(Message message, AsyncCallback callback, object state);
[OperationContract(IsOneWay = false, Action = "BizTalkSubmit")]
Message BizTalkSubmit(Message message);
Message EndTwoWayMethod(IAsyncResult result);
}
[ServiceContract(Namespace = "http://www.microsoft.com/biztalk/2006/r2/wcf-adapter")]
public interface ITwoWayAsyncVoid
{
// Methods
[OperationContract(AsyncPattern = true, IsOneWay = false, Action = "*", ReplyAction = "*")]
IAsyncResult BeginTwoWayMethod(Message message, AsyncCallback callback, object state);
[OperationContract(IsOneWay = false, Action = "BizTalkSubmit")]
void BizTalkSubmit(Message message);
void EndTwoWayMethod(IAsyncResult result);
}
[ServiceContract(Namespace = "http://www.microsoft.com/biztalk/2006/r2/wcf-adapter")]
public interface ITwoWayAsyncVoidTxn
{
// Methods
[TransactionFlow(TransactionFlowOption.Mandatory)]
[OperationContract(AsyncPattern = true, IsOneWay = false, Action = "*", ReplyAction = "*")]
IAsyncResult BeginTwoWayMethod(Message message, AsyncCallback callback, object state);
[TransactionFlow(TransactionFlowOption.Mandatory), OperationContract(IsOneWay = false, Action = "BizTalkSubmit")]
void BizTalkSubmit(Message message);
void EndTwoWayMethod(IAsyncResult result);
}
|
All the methods exposed by these service contracts are generic, asynchronous and untyped. In fact, as shown in the picture above, each of this method is decorated by the OperationContract attribute and in particular:
-
AsyncPattern = True indicates that an operation is implemented asynchronously using a Begin<methodName> and End<methodName> method pair in a service contractAction = “*” means that the method accepts a message with any Action.
-
ReplyAction = “*” means that the method can return a message with any Action.
-
Every method accepts as parameter or returns a generic WCF Message.
As a consequence, each WCF Receive Location can accept multiple message types and versions that can be normalized into a canonical format using a different map before being published to the MessageBox.
Also Send Ports are message-type agnostic.
One-Way WCF Receive Locations
When you define a One-Way WCF Receive Location that uses the NetMsmqBinding, the underlying WCF service will expose an endpoint which uses the IOneWayAsync service interface. If instead the One-Way WCF Receive Location is not configured to use NetMsmqBinding, the underlying WCF service will expose an endpoint using the ITwoWayAsyncVoid contract interface that returns an acknowledgment message. Otherwise, when you define a two-way WCF receive location, the underlying WCF service will expose an endpoint using the ITwoWayASync service interface.
As you can note in the picture above, the IOneWayAsync service contract has a single logical operation named OneWayMethod that matches all incoming messages (Action=”*”). It’s implemented with the WCF asynchronous programming model that ties two methods – BeginOneWayMethod and EndOneWayMethod – together via AsyncPattern=true.
ITwoWayAsyncVoid is similar in design except that it also returns a WCF Message object (notice the addition of ReplyAction=”*”) and the operation is not marked as one-way. The only difference with ITwoWayAsync is that it also returns a Message object in order to properly model two-way ports. The following picture depicts what happens when a message is received and processed through a One-way WCF Receive location.
- The Raw Message Data are sent over the wire.
- The Transport Channel receives and decodes the incoming stream of bytes and creates a WCF Message that is processed through the Channel Stack.
- The WCF message is passed on to the Dispatcher
- The WCF message is passed on to the BizTalkServiceInstance.
- Based on the RL configuration, the entire SOAP Envelope, the Body of the SOAP message or a specific Xml Element is used as content of the BizTalk message.
- The BizTalk message is processed through the pipeline.
- A Map is eventually applied.
- The BizTalk message is published to the MessageBox.
WCF Adapters
The WCF Adapters provided by BizTalk correspond 1:1 to the most commonly used WCF bindings.
| Adapter Name |
WCF Binding Name |
When to use? |
| WCF-BasicHttp |
BasicHttpBinding |
When you need interoperability with WS-I Basic Profile 1.1 services, such as those created with ASP.NET Web services (ASMX) or other first-generation service frameworks |
| WCF-WSHttp |
WSHttpBinding |
When you need interoperability with more advanced services that leverage WS-* protocols, such as those created with WCF or other modern service frameworks |
| WCF-NetTcp |
NetTcpBinding |
When you need efficient inter-machine communication with other WCF applications |
| WCF-NetNamedPipe |
NetNamedPipeBinding |
When you need efficient intra-machine communication with other WCF applications |
| WCF-NetMsmq |
NetMsmqBinding |
When you need durable, asynchronous communication with other WCF applications (using MSMQ as the underlying transport) |
| WCF-Custom |
Any |
When you need to define a custom binding configuration for an “in-process” host |
| WCF-CustomIsolated |
Any |
When you need to define a custom binding configuration for an “isolated” host – this is only a receive adapter, not used on send ports |
WCF-Custom and WCF-CustomIsolated Adapters
The WCF-Custom and WCF-CustomIsolated adapters offer you complete control over the channel stack and behaviors configuration, and as a consequence, they are the only WCF adapters you really need. Compared with the other WCF Adapters, they are the only ones to provide the possibility to:
-
Implement and exploit extensibility points.
-
Have full access to properties exposed by bindings/behaviors.
-
Enable the use of the bamInterceptor endpoint behavior.
-
Export/Import the binding configuration.
-
Disable a receive location on failure.
-
Run an http-based RL within an in-process host.
-
Use bindings (e.g. wsDualHttpBinding) for which a WCF Adapter does not exist.
WCF allows developers to modify and extend its standard runtime behavior with several extensibility points, some of which are not supported by BizTalk Server: for example you cannot customize the BizTalkServiceInstance class, create you own ServiceHost or create a custom ContractBehavior. BizTalk Server supports the following extensibility points:
-
Custom Behaviors
-
Service Behaviors: they enable the customization of the entire service runtime including the ServiceHost.
-
Endpoint Behaviors: they enable the customization of service endpoints and their associated EndpointDispatcher. The most common technique is creating an endpoint behavior to add a Message Inspector to the collection of Message Inspectors used at run time by a proxy/dispatcher. This allows to intercept an in-flight incoming/outgoing WCF message before this latter is passed/transmitted to the intended operation/target service, inspect and eventually change its content (headers or body).
-
Custom Binding Elements, Channels Binding Elements, Channels, Channel Factories, Binding Element Extension, etc. This technique is more complex, because in order to register and execute at runtime a custom channel within the channel stack used by a WCF Receive Location or Send Port, it’s necessary to create and properly register multiple components:
-
Custom Bindings: the
WCF LOB Adapter SDK allows developers to create new bindings to use with WCF-Custom and WCF-CustomIsolated Adapters. For example, the
WCF based SQL Adapter (WCF-SQL) is based on a new binding called
sqlBinding that is properly registered in the machine.config when installing the Adapter.
In my previous post called How to Throw Typed Fault Exceptions from Orchestrations Published as WCF Services I created 2 different Endpoint Behaviors to customize the default runtime behavior of the WCF-Custom/WCF-CustomIsolated Adapter. In my post entitled How to create a custom WCF Channel that debatches an inbound message I show how creating a custom channel to debatch (on a WCF Send Port) an inbound message containing a collection of operations and make multiple calls to the underlying WCF service, one for each operation.
Conclusions
I strongly suggest you to read the following whitepapers to have a better insight in WCF Adapters and their extensibility points.
- The Using the Windows Communication Foundation (WCF) Adapters in BizTalk Server whitepaper describes the use of the WCF adapters in BizTalk Server.
- The Consuming and Hosting WCF Services with Custom Bindings in BizTalk Server whitepaper provides an in-depth explanation on how to use the BizTalk Server WCF adapters for hosting and consuming Windows Communication Foundation (WCF) services with custom bindings. The paper compiles a series of lessons learned based on a real-world implementation of integrating a custom WCF binding with the WCF adapters. Furthermore, key industrial infrastructure concerns such as transaction management and security are discussed in the context of integrating custom WCF bindings and behaviors with the BizTalk Messaging Engine. Also emphasized are a few pragmatic paradigms such as the use of dynamic ports in consuming WCF services. Finally, some key integration challenges are discussed to streamline the correct use of the adapters for solving complex business problems.
Moreover, I recommend to download and review the following WCF Adapters samples on MSDN:
by community-syndication | Jun 23, 2010 | BizTalk Community Blogs via Syndication
Introduction
We all know that testing is very important for every kind of solution. However, testing and in particular performance testing is often neglected for time or budget reasons. One of the things that I have found when working with customers is that the amount of performance testing done on a project tends to fall into a couple of categories. The first and in my opinion the worst is when no performance testing is done at all. I would call this a write once, go live and cross your fingers approach. The next type is when load/stress testing is performed right at the end of the project lifecycle, typically a few weeks before the go live date. If the application is not able to meet the expected performance goals in terms of scalability and latency, there’s a serious risk that architects and developers will have to spend a considerable amount of time to redesign, rewrite and test large portions of the solution.
Sometimes performance testing is done using a rig that does not accurately reflect the production environment, while in other cases I have seen people using different adapters that do not match those used in the production environment. All these bad practices and factors expose the project to the risk of delays and failure.
BizTalk Server is used in mission and business critical scenarios where it is not just part of the business, it is actually running the entire business. Overlooking performing testing increases the risk of failure which can cause not just technical problems but political problems as well.
The way to minimize the risk of a BizTalk project (and of any software project in general) is to adopt a strong and consistent testing strategy throughout the development phase and before deploying the final solution into production. For this reason, the BizTalk Customer Advisory Team adopts a consistent and reusable methodology when running performance labs with customers. This process is fully documented in the BizTalk Server Performance Testing Methodology of the Microsoft BizTalk Server 2009 Performance Optimization Guide.
The best approach is to leverage a test framework like BizUnit to automate test runs and possibly schedule their execution over night. Another key factor is to individuate the right tool to generate load against the application. To this purpose you can use the BizTalk LoadGen 2007 as I explained in my previous article. The LoadGen tool offers the following important features and benefits to provide a simple, generic, reliable, and consistent tool for generating message transmission loads. By default, LoadGen supports several transport protocols:
| Transport component |
Description |
| File transport |
The File transport creates files including message data in the destination folder. |
| HTTP transport |
The HTTP transport sends messages with the POST method to the destination location that the BizTalk HTTP receive adapter hosts. |
| MQSeries transport |
The MQSeries transport sends messages to the target MQSeries queue. |
| MSMQLarge transport |
The MSMQLarge transport sends messages to the BizTalk MSMQT receive locations. Native message queuing cannot process a message with a body larger than 4 megabytes (MB). |
| MSMQ transport |
The MSMQ transport sends messages smaller than 4 megabytes (MB) to the target MSMQ queue. |
| SOAP transport |
The SOAP transport sends messages with the POST method to the target Web service. |
| WSE transport |
The WSE transport sends messages to the target Web service hosted by Web Services Enhancements (WSE) 2.0. |
| WSS transport |
The WSS transport sends messages with the PUT method to the target Windows SharePoint Services site. |
| WCF transport |
The WCF transport sends message data to the target Windows Communication Foundation (WCF) service. |
However, LoadGen is a great tool, but it suffers of the following problems:
-
Test definition is entirely done through XML configuration files. LoadGen does not provide any UI to model and define load tests.
-
It doesn’t provide the possibility to create rules to generate warnings when a give performance counter exceeds a configurable threshold.
-
Generating high loads against a BizTalk application is an intensive task and can easily lead to run out of system resources (CPU, Memory, Disk I/O, Network, etc.). For this reason, when running a performance lab against an enterprise level application is necessary to arrange multiple Test Agent machines in order to generate the expected load. Even though you can start several instances of LoadGen on separate machines, you can’t coordinate their work in a centralized manner. Multiple instances of LoadGen can be started using a script or the LoadGenExecuteStep class provided by BizUnit, but there’s no way to monitor their execution using a console.
-
When invoking a Two-Way Receive Location, LoadGen provides the possibility to save response times to a log file, but not to a centralized database. Of course, when the test is over, it’s possible to collect the log files produced by the different Test Agents, each running on a separate machine, and upload their contents to a SQL Server table with a Bulk Insert. This allows to conduct an in-depth analysis of the test results and eventually run a set of queries to generate statistics and compare key performance indicators of different test runs. Nevertheless, this process must be manually built as it is not provided out of the box by LoadGen.
Sometimes customers/partners create a custom multi-threaded application in order to generate load against their BizTalk Application, correlate request and response messages and persist response times to a relational database. This approach requires time and resources to design and develop the multi-threaded application capable to generate the load against the BizTalk platform and store test results to a custom repository. Another common technique adopted by customers is to use a dedicated product like HP LoadRunner to measure end-to-end performance, diagnose application and system bottlenecks and tune for better performance. So the question arises: is there any Microsoft product that I can use to perform stress and load tests in a easy way? The answer is yes.
Visual Studio Team System Test Edition
In my opinion, the best solution to design, configure, execute and monitor performance tests against BizTalk is to leverage Visual Studio Team System Test Edition that includes a suite of tools that allow to conduct several kinds of tests (unit tests, load tests, web tests, etc.). In particular, Visual Studio Team System Test Edition provides a tool for creating and running load tests. For more information on this topic see Understanding Load Tests. Visual Studio Team System Test Edition provides the following advantages:
-
It enables developers to set up a group of computers that generates simulated load for testing. The group consists of a single Controller and one or more Test Agents, each running on a separate machine. Collectively, this group is called a Rig. In particular, a Test Agent is the part of the Rig that is used to run tests and generate simulated load. For more information on this topic see
Controllers, Agents, and Rigs.
-
When you define a load test, you can choose between two different load patterns:
-
Constant: the constant load pattern is used to specify a user load that does not change during the load test. For example, when you run a smoke test on a Web application, you might want to set a light, constant load of 10 users.
-
Step: the step load pattern is used to specify a user load that increases with time up to a defined maximum user load. For stepping loads, you specify the Initial User Count, Maximum User Count, Step Duration (seconds), and Step User Count.For example a Step load with an Initial User count of one, Maximum User Count of 100, Step Duration (seconds) of 10, and a Step User Count of 1 creates a user load pattern that starts at 1, increases by 1 every 10 seconds until it reaches 100 Users.
-
It’s possible to create a Load Test Results Repository to store the information gathered during a test run. This repository contains performance counter data and information about any errors. See
Load Test Results Repository for more information on this topic.
-
When you create a load test, Visual Studio Team System Test Edition lets you specify multiple counter sets. A counter set is collection of performance counters that are useful to monitor during a load test run. Counter sets are part of the load test and apply to all its scenarios. They are organized by technology, for example, ASP.NET, IIS or SQL counter sets. Visual Studio allows developers to create their own Counter Sets to track any performance counters. Counter sets can be defined to gather performance counter data from all the machines composing the performance lab kit. (BizTalk Server, SQL Server, Test Agents, etc.).
-
This feature is extremely helpful because during a test run it allows to collect performance counter data (BizTalk Server, SQL, IIS, etc.) in a central storage and to create multiple diagrams to visualize the most relevant counters. When the test run is over, the summary page allows to browse and analyze test results.
How to use Visual Studio to generate load against a Two-Way WCF Receive Location
Recently I published an article in 2 parts (Part1, Part2) where I compare 4 different techniques to process an XLANGMessage within a method exposed by a business component invoked by an orchestration. In particular, I created a Two-Way Synchronous version and a One-Way Asynchronous version for each of these patterns. All of the 4 synchronous patterns share the same architecture depicted below:
Message Flow:
- A WCF-BasicHttp or WCF-Custom Request-Response Receive Location receives a new CalculatorRequest xml document from the Test Agent/Client Application.
- The XML disassembler component within the XMLTransmit pipeline promotes the Method element inside the CalculatorRequest xml document. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
- The inbound request starts a new instance of a given orchestration.
- The orchestration invokes a method exposed by a business component.
- The method in question returns a response message.
- The orchestration publishes the CalculatorResponse message to the MessageBox (BizTalkMsgBoxDb).
- The response message is retrieved by the Request-Response WCF-BasicHttp or WCF-Custom Receive Location.
- The response message is returned to the Test Agent/Client Application.
In order to generate the load against the WCF-Basic and WCF-Custom Receive Locations I proceeded as follows:
- I started creating a Test Project called WCFLoadTest.
- Then I created a Test Class called WCFLoadTest. This class is decorated with the Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute to indicate that the class contains test methods.
- I defined a Test Method for each combination of the following factors:
- Receive Location/Transport used to submit messages to BizTalk (WCF-NetTcp, WCF-BasicHttp)
- Message Size (small, medium, large)
- Orchestration Pattern (XmlDocumentOrchestration, StreamOrchestration, MessageClassesOrchestration, CustomBTXMessageOrchestration).
- Each combination of the factors above corresponds to a separate test case. I used the following naming convention for test methods:
- <Transport><Message Size><Orchestration Pattern>
- I created an helper method called InvokeTwoWayWCFReceiveLocation to allow test methods to exchange messages with a Request Response WCF Receive Location. This methods expects the following input parameters
- endpointConfigurationName: the name of one of the service endpoints defined in the app.config configuration.
- requestMessageFolder: the path of the folder containing the request documents.
- requestMessageName: the name of the request document to send.
- messageVersion: specifies the version of SOAP and WS-Addressing of the request and response messages.
- sessionMode: indicates the support for reliable sessions (Allowed, Required, NotAllowed).
- Each test method invokes the InvokeTwoWayWCFReceiveLocation passing a different combination of parameters.
The client endpoints used by the InvokeTwoWayWCFReceiveLocation method to exchange messages with the WCF Receive Locations are defined in the configuration file (App.config) of the project.
App.config
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<system.serviceModel>
<!-- Bindings used by client endpoints -->
<bindings>
<netTcpBinding>
<binding name="netTcpBinding"
closeTimeout="01:10:00"
openTimeout="01:10:00"
receiveTimeout="01:10:00"
sendTimeout="01:10:00"
transactionFlow="false"
transferMode="Buffered"
transactionProtocol="OleTransactions"
hostNameComparisonMode="StrongWildcard"
listenBacklog="100"
maxBufferPoolSize="1048576"
maxBufferSize="10485760"
maxConnections="200"
maxReceivedMessageSize="10485760">
<readerQuotas maxDepth="32"
maxStringContentLength="8192"
maxArrayLength="16384"
maxBytesPerRead="4096"
maxNameTableCharCount="16384" />
<reliableSession ordered="true"
inactivityTimeout="00:10:00"
enabled="false" />
<security mode="None">
<transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
<message clientCredentialType="Windows" />
</security>
</binding>
</netTcpBinding>
<basicHttpBinding>
<binding name="basicHttpBinding"
closeTimeout="00:10:00"
openTimeout="00:10:00"
receiveTimeout="00:10:00"
sendTimeout="00:10:00"
allowCookies="false"
bypassProxyOnLocal="false"
hostNameComparisonMode="StrongWildcard"
maxBufferSize="10485760"
maxBufferPoolSize="524288"
maxReceivedMessageSize="10485760"
messageEncoding="Text"
textEncoding="utf-8"
transferMode="Buffered"
useDefaultWebProxy="true">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
</binding>
</basicHttpBinding>
</bindings>
<client>
<!-- Client endpoints used by client excahnge messages with the WCF Receive Locations -->
<endpoint address="net.tcp://localhost:8123/calculator"
binding="netTcpBinding"
bindingConfiguration="netTcpBinding"
contract="System.ServiceModel.Channels.IRequestChannel"
name="netTcpCalculatorServiceReceiveLocationEndpoint" />
<endpoint address="http://localhost/CalculatorServiceReceiveLocation/CalculatorService.svc"
binding="basicHttpBinding"
bindingConfiguration="basicHttpBinding"
contract="System.ServiceModel.Channels.IRequestChannel"
name="basicHttpCalculatorServiceReceiveLocationEndpoint" />
</client>
</system.serviceModel>
<appSettings>
<!-- Folder containing test messages -->
<add key="testMessageFolder" value="C:\Projects\HandleXLANGMessages\WCFLoadTest\TestMessages" />
</appSettings>
</configuration>
|
The following table contains the code of the WCFLoadTest class:
WCFLoadTest Class
#region Copyright
//-------------------------------------------------
// Author: Paolo Salvatori
// Email: paolos@microsoft.com
// History: 2009-09-20 Created
//-------------------------------------------------
#endregion
#region Using Directives
using System;
using System.IO;
using System.Diagnostics;
using System.Text;
using System.Configuration;
using System.Collections.Generic;
using System.Linq;
using System.Xml;
using System.ServiceModel;
using System.ServiceModel.Channels;
using Microsoft.VisualStudio.TestTools.UnitTesting;
#endregion
namespace Microsoft.BizTalk.CAT.Samples.HandleXLANGMessages.WCFLoadTest
{
/// <summary>
/// Summary description for UnitTest1
/// </summary>
[TestClass]
public class WCFLoadTest
{
#region Constants
private const int MaxBufferSize = 2097152;
private const string Source = "WCF Load Test";
private const string Star = "*";
private const string TestMessageFolderParameter = "testMessageFolder";
private const string TestMessageFolderDefault = @"C:\Projects\HandleXLANGMessages\WCFLoadTest\TestMessages";
private const string TestMessageFolderFormat = @"Test Message Folder = {0}";
private const string NetTcpCalculatorServiceReceiveLocationEndpoint = "netTcpCalculatorServiceReceiveLocationEndpoint";
private const string BasicHttpCalculatorServiceReceiveLocationEndpoint = "basicHttpCalculatorServiceReceiveLocationEndpoint";
private const string SmallCustomBtxMessageOrchestration = "SmallCustomBtxMessageOrchestration.xml";
private const string SmallMessageClassesOrchestration = "SmallMessageClassesOrchestration.xml";
private const string SmallStreamOrchestration = "SmallStreamOrchestration.xml";
private const string SmallXmlDocumentOrchestration = "SmallXmlDocumentOrchestration.xml";
private const string MediumCustomBtxMessageOrchestration = "MediumCustomBtxMessageOrchestration.xml";
private const string MediumMessageClassesOrchestration = "MediumMessageClassesOrchestration.xml";
private const string MediumStreamOrchestration = "MediumStreamOrchestration.xml";
private const string MediumXmlDocumentOrchestration = "MediumXmlDocumentOrchestration.xml";
private const string LargeCustomBtxMessageOrchestration = "LargeCustomBtxMessageOrchestration.xml";
private const string LargeMessageClassesOrchestration = "LargeMessageClassesOrchestration.xml";
private const string LargeStreamOrchestration = "LargeStreamOrchestration.xml";
private const string LargeXmlDocumentOrchestration = "LargeXmlDocumentOrchestration.xml";
#endregion
#region Private Instance Fields
private TestContext testContextInstance;
#endregion
#region Private Static Fields
private static string testMessageFolder = null;
#endregion
#region Public Instance Constructor
public WCFLoadTest()
{
}
#endregion
#region Public Static Constructor
static WCFLoadTest()
{
try
{
testMessageFolder = ConfigurationManager.AppSettings[TestMessageFolderParameter];
if (string.IsNullOrEmpty(testMessageFolder))
{
testMessageFolder = TestMessageFolderDefault;
}
}
catch (Exception ex)
{
Trace.WriteLine(ex.Message);
EventLog.WriteEntry(Source, ex.Message, EventLogEntryType.Error);
}
}
#endregion
#region Public Properties
/// <summary>
///Gets or sets the test context which provides
///information about and functionality for the current test run.
///</summary>
public TestContext TestContext
{
get
{
return testContextInstance;
}
set
{
testContextInstance = value;
}
}
#endregion
#region Test Methods
[TestMethod]
public void BasicHttpSmallCustomBtxMessageOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
SmallCustomBtxMessageOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpSmallMessageClassesOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
SmallMessageClassesOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpSmallStreamOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
SmallStreamOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpSmallXmlDocumentOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
SmallXmlDocumentOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpMediumCustomBtxMessageOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
MediumCustomBtxMessageOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpMediumMessageClassesOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
MediumMessageClassesOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpMediumStreamOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
MediumStreamOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpMediumXmlDocumentOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
MediumXmlDocumentOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpLargeCustomBtxMessageOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
LargeCustomBtxMessageOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpLargeMessageClassesOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
LargeMessageClassesOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpLargeStreamOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
LargeStreamOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void BasicHttpLargeXmlDocumentOrchestration()
{
InvokeTwoWayWCFReceiveLocation(BasicHttpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
LargeXmlDocumentOrchestration,
MessageVersion.Soap11,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpSmallCustomBtxMessageOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
SmallCustomBtxMessageOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpSmallMessageClassesOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
SmallMessageClassesOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpSmallStreamOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
SmallStreamOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpSmallXmlDocumentOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
SmallXmlDocumentOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpMediumCustomBtxMessageOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
MediumCustomBtxMessageOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpMediumMessageClassesOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
MediumMessageClassesOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpMediumStreamOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
MediumStreamOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpMediumXmlDocumentOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
MediumXmlDocumentOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpLargeCustomBtxMessageOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
LargeCustomBtxMessageOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpLargeMessageClassesOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
LargeMessageClassesOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpLargeStreamOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
LargeStreamOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
[TestMethod]
public void NetTcpLargeXmlDocumentOrchestration()
{
InvokeTwoWayWCFReceiveLocation(NetTcpCalculatorServiceReceiveLocationEndpoint,
testMessageFolder,
LargeXmlDocumentOrchestration,
MessageVersion.Default,
SessionMode.Allowed);
}
#endregion
#region Helper Methods
public void InvokeTwoWayWCFReceiveLocation(string endpointConfigurationName,
string requestMessageFolder,
string requestMessageName,
MessageVersion messageVersion,
SessionMode sessionMode)
{
ChannelFactory<IRequestChannel> channelFactory = null;
IRequestChannel channel = null;
XmlTextReader xmlTextReader = null;
Message requestMessage = null;
Message responseMessage = null;
try
{
channelFactory = new ChannelFactory<IRequestChannel>(endpointConfigurationName);
channelFactory.Endpoint.Contract.SessionMode = sessionMode;
channel = channelFactory.CreateChannel();
string path = Path.Combine(requestMessageFolder, requestMessageName);
xmlTextReader = new XmlTextReader(path);
requestMessage = Message.CreateMessage(messageVersion, Star, xmlTextReader);
TestContext.BeginTimer(requestMessageName);
responseMessage = channel.Request(requestMessage);
channel.Close();
channelFactory.Close();
}
catch (FaultException ex)
{
HandleException(ref channelFactory,
ref channel,
ex);
throw;
}
catch (CommunicationException ex)
{
HandleException(ref channelFactory,
ref channel,
ex);
throw;
}
catch (TimeoutException ex)
{
HandleException(ref channelFactory,
ref channel,
ex);
throw;
}
catch (Exception ex)
{
HandleException(ref channelFactory,
ref channel,
ex);
throw;
}
finally
{
TestContext.EndTimer(requestMessageName);
CloseObjects(channelFactory,
channel,
xmlTextReader,
requestMessage,
responseMessage);
}
}
private void HandleException(ref ChannelFactory<IRequestChannel> channelFactory,
ref IRequestChannel channel,
Exception ex)
{
try
{
if (channelFactory != null)
{
channelFactory.Abort();
channelFactory = null;
}
if (channel != null)
{
channel.Abort();
channel = null;
}
Trace.WriteLine(ex.Message);
EventLog.WriteEntry(Source, ex.Message, EventLogEntryType.Error);
}
catch (Exception)
{
}
}
private void CloseObjects(ChannelFactory<IRequestChannel> channelFactory,
IRequestChannel channel,
XmlTextReader xmlTextReader,
Message requestMessage,
Message responseMessage)
{
try
{
if (channelFactory != null)
{
channelFactory.Close();
}
if (channel != null)
{
channel.Close();
}
if (xmlTextReader != null)
{
xmlTextReader.Close();
}
if (requestMessage != null)
{
requestMessage.Close();
}
if (responseMessage != null)
{
responseMessage.Close();
}
}
catch (Exception)
{
}
}
#endregion
}
}
|
Each test method corresponds to a separate Unit Test. Visual Studio permits to create a Load Test as a composition of one or multiple unit tests. Explaining in detail how creating a Load Test is out of the scope of the present post. This process is fully documented on MSDN at the following link: Walkthrough: Creating and Running a Load Test. However, for the sake of the completeness I’m going to describe the steps necessary to create a Load Test based on a single Unit Test:
-
This opens a a Wizard that walks you through the creation of a Load Test. Click the Next button. See
Creating Load Tests for more information on this topic.
-
Enter a name for the load test scenario and select one of the available think time profiles (I always choose ‘Do not use think times’ to increase the load against the tested application). Click the Next button. See
How to: Specify Scenarios for more information on this step.
- Choose a load pattern for your load test and configure parameters applicable to your choice. Click the Next button. See How to: Specify Load Patterns for more information on this step.
- A load test contains one or more scenarios, each of which contains one or more Unit Tests. Press the Add button to add one or multiple Unit Tests to your scenario. See How to: Specify Browser Mix for more information on this step.
- Select one or multiple Unit Tests and click the OK button. One again, each Test Method corresponds to a separate Unit Test.
- If you selected multiple Unit Tests, as in the picture below, you can select your preferred browser mix by adjusting the sliders in the Distribution column, or by typing the percentage values directly into the % column, then click the Next button to advance to the next step.
- The following dialog allows you specify which computers and counter sets to monitor during a load test run. After completing the wizard, you can define additional counter sets. This is very helpful because it allows to select any performance counters exposed by a product (BizTalk Server in our case) or by your application. When you have done, click the Next button to advance to the next step. See How to: Specify Counter Sets for more information on this step.
- In this dialog, you can specify Run settings that affect the entire load test. The run settings determine the length of the test, warm-up duration, maximum number of error details reported, sampling rate, connection model (Web tests only), results storage type, validation level and SQL tracing. The run settings should reflect the goals of your load test. For more information, see About Run Settings. When you have done, click the Finish button to complete the Load Test creation. See How to: Specify Run Settings for more information on this step.
- At this point you can open the newly created Load Test, right click the Counter Sets and select Add Custom Counter Set from the drop menu. Rename your new Counter Set ‘BizTalk Server’.
- Now, right click the BizTalk Server Counter Set and click the Add Counters menu item to add the most relevant BizTalk Server, System, .NET and WCF performance counters:
BizTalk Server performance counters
\\.\BizTalk:FILE Receive Adapter(*)\*
\\.\BizTalk:FILE Send Adapter(*)\*
\\.\BizTalk:FTP Receive Adapter(*)\*
\\.\BizTalk:FTP Send Adapter(*)\*
\\.\BizTalk:HTTP Receive Adapter(*)\*
\\.\BizTalk:HTTP Send Adapter(*)\*
\\.\BizTalk:Message Agent(*)\*
\\.\BizTalk:Message Box:General Counters(*)\*
\\.\BizTalk:Message Box:Host Counters(*)\*
\\.\BizTalk:Messaging Latency(*)\*
\\.\BizTalk:Messaging(*)\*
\\.\BizTalk:MSMQ Receive Adapter(*)\*
\\.\BizTalk:MSMQ Send Adapter(*)\*
\\.\BizTalk:POP3 Receive Adapter(*)\*
\\.\BizTalk:SMTP Send Adapter(*)\*
\\.\BizTalk:SOAP Receive Adapter(*)\*
\\.\BizTalk:SOAP Send Adapter(*)\*
\\.\BizTalk:SQL Receive Adapter(*)\*
\\.\BizTalk:SQL Send Adapter(*)\*
\\.\BizTalk:TDDS(*)\*
\\.\BizTalk:Windows SharePoint Services Adapter(*)\*
\\.\XLANG/s Orchestrations(*)\*
WCF performance counters
\\.\ServiceModelService 3.0.0.0
\\.\ServiceModelEndpoint 3.0.0.0
\\.\ServiceModelOperation 3.0.0.0
.NET performance counters
\\.\.NET CLR Exceptions(*)\*
\\.\.NET CLR Memory(*)\*
System performance counters
\\.\Cache\*
\\.\Distributed Transaction Coordinator\*
\\.\Enterprise SSO(*)\*
\\.\LogicalDisk(*)\*
\\.\Memory\*
\\.\Network Interface(*)\*
\\.\Paging File(*)\*
\\.\PhysicalDisk(*)\*
\\.\Process(*)\*
\\.\Processor(*)\*
\\.\System\*
\\.\TCPv4\*
\\.\IPv4\*
- Repeat the last step to include the most relevant performance counters on the SQL machine hosting the BizTalkMsgBoxDb. In the list below, the name of the SQL Server instance hosting the BizTalkMsgBoxDb is MSSQL$BTS:
SQL Server performance counters
\\.\MSSQL$BTS:Access Methods\*
\\.\MSSQL$BTS:Buffer Manager\*
\\.\MSSQL$BTS:Buffer Node(*)\*
\\.\MSSQL$BTS:Buffer Partition(*)\*
\\.\MSSQL$BTS:Databases(*)\*
\\.\MSSQL$BTS:General Statistics\*
\\.\MSSQL$BTS:Latches\*
\\.\MSSQL$BTS:Locks(*)\*
\\.\MSSQL$BTS:Memory Manager\*
\\.\MSSQL$BTS:SQL Statistics\*
\\.\MSSQL$BTS:Transactions\*
\\.\MSSQL$BTS:Wait Statistics(*)\*
\\.\MSSQL$BTS:Plan Cache\*
System performance counters
\\.\Objects\*
\\.\Memory\*
\\.\Paging File(*)\*
\\.\PhysicalDisk(*)\*
\\.\Process(*)\*
\\.\Processor(*)\*
\\.\Network Interface(*)\*
\\.\TCPV4(*)\*
\\.\LogicalDisk(*)\*
\\.\System\*
Latency:
-
- warning: response time < 2 sec for 90% of messages
- critical: response time < 5 sec for 99% of messages
-
In order to be notified during a test run that a response time exceeded one the threshold above, I can create a custom threshold rule. Nice, but how can I accomplish this task and which performance counter should I use? Let’s start asking to this latter question. The suitable performance counters for measuring the latency of a Request-Response WCF Receive Location are the following:
- \\.\BizTalk:Messaging Latency(<Host running your WCF Receive Location>)\Request-Response Latency (sec)
- \\.\ServiceModelService 3.0.0.0(<the service corresponding to your WCF Receive Location>)\Calls Duration
- \\.\ServiceModelEndpoint 3.0.0.0(<the endpoint used by your WCF Receive Location>)\Calls Duration
- \\.\ServiceModelEOperation 3.0.0.0(<the endpoint used by your WCF Receive Location>)\Calls Duration
-
In order to create a threshold on the Request-Response Latency (sec) performance counter, you can expand the BizTalk Server Counter Set and browse to the counter. Now right click the corresponding node and select the Add Threshold Rule from the drop menu:
Click the OK button to confirm. See How to: Add a Threshold Rule for more information on this topic
-
You can repeat the same process and set a Threshold Rule to be alerted when the throughput of your application goes below a certain threshold. A good candidate to measure the throughput of a BizTalk application composed by one or multiple orchestration is the \\.\XLANG/s Orchestrations(<Host>)\Orchestration completed/sec performance counter. However, in this case I will set the value of the Alert If Over property to False to indicate that a threshold violation occurs when the performance counter value is less then the threshold value (in the sample below 20 and 10 are respectively the Warning and Critical Threshold Values).
At runtime, threshold violations can be reviewed clicking the threshold violation link under the Load Test toolbar. If the counter in question is exposed by a Graph, each threshold violations is signaled with a yellow triangle.
At this point I’m ready to launch a load test, so I can just press the start button. In order to create a custom Graph to monitor most relevant counters I proceed as follows:
-
Now, it’s sufficient to select, drag & drop counters from the Counter panel to the newly created Graph. For the sake of I completeness, during my demo I created a graph called BizTalk Server and I dragged and dropped the following performance counters:
-
\\.\BizTalk:Messaging Latency(BizTalkServerIsolatedHost)\Request-Response Latency (sec)
-
\\.\BizTalk:Messaging (BizTalkServerIsolatedHost)\Documents received/sec
-
\.\XLANG/s Orchestrations(BizTalkServerApplication)\Orchestration completed/sec
-
\.\BizTalk:Message Box:General Counters(biztalkmsgboxdb:babo)\Spool Size
-
I strongly suggest to repeat the process and create a custom Graph to monitor the most relevant performance counters exposed by the SQL Server instance or instances that host the most solicited BizTalk Server databases:
-
BizTalkMsgBoxDb and any additional MessageBox.
-
BizTalkDTADb if global tracking is turned on.
-
BAMPrimaryImport if the application makes use of BAM.
and any custom databases used by the application.
When the load test is over, you can review the results on the Summary Page:
You can also click the links in the Test Results and Transaction Results tables to navigate to a page where you can review more in detail test results, errors and threshold violations.
Summary
Visual Studio Test Edition is a powerful suite to rapidly build Unit and Load Tests to conduct performance and stress tests against your BizTalk application. We assumed that the application in question was exposed via a WCF Two-Way Receive Location, but the approach exposed in the present post can be customized and extended to other scenarios where the application makes use of other Adapter to expose its services. Here you can download the Test Project I used to conduct tests against the HandleXLANGMessages application. See Part1 and Part2 for more information.
