I sometimes find myself in a situation where I need to make direct modifications to the XML code of an orchestration, map or pipeline. In Visual Studio you can do this easily by opening the file with the non default editor (right click orchestration file, ’Open With’ and ’XML (Text) Editor’). I encounter very strange […]
A challenge we have at Tallan is keeping our training up-to-date with the latest and greatest of technologies. We are constantly re-evaluating our lessons to keep up with ever changing technologies. In recent months, we have seen a shift in our client work from using traditional ASP.NET to using ASP.NET MVC 2. This has challenged […]
I was there when this feature was born. It’s a long story but basically the idea is that you should be able to have a WCF service (or Workflow Service for that matter) without a SVC file.
Simply put, an SVC file is configuration that lives in a file that gives IIS something to serve (after all it is a web server).
The .svc extension is handled by WCF so when a request comes into IIS with a URI that ends with .svc, WCF gets to handle it. You can find the details in your machine level web.config (just search for .svc).Here is an example of the markup you will find in an SVC file. To see this, just right click on a SVC file in Visual Studio and select View Markup
<@ ServiceHost Language="C#" Debug="true" Service="WebApplication1.Service1" CodeBehind="Service1.svc.cs" %>
As you can see, this configuration gives WCF the information it needs to build (if necessary) and host your service. Yes you can actually put code in an SVC file but I don’t recommend it. Usually all the code is in the code behind file.
Most people don’t mind .svc files. After all they are small 1 line text files. But if you have thousands of services and hundreds of web servers there is a certain amount of overhead involved in managing all these 1 line text files. The good news is that now with .NET 4 you can eliminate them if you want to.
Configuration Based Activation is the name of the feature that allows you to eliminate the .svc file. I’m going to walk you through step by step how you can eliminate an .svc file.
<serviceHostingEnvironment multipleSiteBindingsEnabled="true" >
<serviceActivations>
<add service="WebApplication1.Service1" relativeAddress ="Service1.svc"/>
</serviceActivations>
</serviceHostingEnvironment>
As the adoption of SO-Aware keep increasing, we have been running into more and more customers that are evaluating SO-Aware against traditional SOA governance products. Those comparisons are always very interesting for us because, to some extent, we built…(read more)
Windows Workflow Foundation 4 makes it real easy to create workflow services that do long running work on a server. However when we are doing long running work there could be an issue with lots of workflows being started and too many workflow instances competing for the same data and threads thereby causing problems like database timeouts or thread pool starvation.
To simulate a busy workflow service I have the following sample workflow:
The workflow service is started using a WCF request, returns a response, prints a start message, does some work and prints a message that it is done. The “long” running work is simulated by this activity:
public class MyActivity : CodeActivity
{
protected override void Execute(CodeActivityContext context)
{
Thread.Sleep(15000);
}
}
Of course you would never do this for real, just imagine some complex database queries being done instead of a Thread.Sleep() []:)
Adding a simple client application like this:
class Program
{
static void Main(string[] args)
{
for (int i = 0; i < 1000; i++)
{
Console.WriteLine("Sending {0}", i);
var proxy = new ServiceClient();
proxy.Operation1(i);
}
Console.ReadLine();
}
}
I originally expected the loop to quickly start 1000 workflows and the workflows to complete after 15 seconds. Turns out this isn’t the case [:(] Even though the SendReply has finished the WCF response is not actually send until the complete workflow is finished. The reason is the workflow will run as many activities as it can before relinquishing control. And that happens to include my Thread.Sleep().
The trick to making the workflow service return the response to the client sooner is to add a Delay activity with a short timeout, I am using 1 second here, just after the SendReply activity. Now the first 115 or so workflows are launched quite quickly and only after the WCF response is returned to the client does the actual work start. The problem is we have over 100 workflows active at the same time and that could be the cause of problems so we want to start throttling our workflows so a limited number runs at the same time.
Throttling the Workflow Service
When we are using WCF we can use the serviceThrottling behavior to limit the number of concurrent sessions, request and server instances. The incoming WCF requests for a workflow service respect these throttle limits just like any other WCF service. However after the SendReply we are no longer part of the WCF request handling. However the serviceThrottling maxConcurrentInstances setting is still respected for running workflow service instances. So adding the following to our workflow service behavior would mean no more than 10 workflows are active at any given moment.
<serviceThrottling maxConcurrentInstances="10"/>
This by itself isn’t quite the solution when a client wants to start 1000 workflows as the first 10 will be started and the next can only be started as the first are finished.
Creating a batch of Workflow Services
If we want to be able to start 1000 workflows quickly and have them execute with a maximum of 10 at the time we need to add the sqlWorkflowInstanceStore to the mix. By doing this, and increasing the Delay duration a bit we can quickly start 1000 workflows, persist them to disk and unload them. Only when that is done do we allow them to run, a few at the time.
To achieve this I have changed to workflow to the following structure with the Delay set to 10 seconds. You should use a duration that allows you to start all workflow instances before the actual execution starts.
With the following configuration file:
<system.serviceModel>
<behaviors>
<serviceBehaviors>
<behavior>
<serviceMetadata httpGetEnabled="true"/>
<serviceThrottling maxConcurrentInstances="10"/>
<sqlWorkflowInstanceStore connectionString="Data Source=.\sqlexpress;Initial Catalog=WorkflowInstanceStore;Integrated Security=True"/>
<workflowIdle timeToUnload="00:00:00"/>
</behavior>
</serviceBehaviors>
</behaviors>
</system.serviceModel>
The result is 1000 workflows being kicked of quickly, unloaded to SQL Server and then reloaded and executing on their own time, 10 at the time. Sweet for batch jobs kicking off a large number of workflows [:)]
Download the code here.
Enjoy!
www.TheProblemSolver.nl
Wiki.WindowsWorkflowFoundation.eu
Don’t miss out on the BizTalk Saturday – BizTalk 2010 Hands on Days, if you are in Adelaide or Canberra you need to get your registrations in soon to insure we get the numbers we need to run these days. The days a been very successful so far and we have had 89 people attend in the other cities, with Auckland leading the way with 25. So don’t miss out; go to http://www.biztalksaturday.com and register.
On December 2nd we are holding a free, all-day event focused on Silverlight. You can attend it either in person on the Microsoft campus (where you’ll get to network with a bunch of Silverlight experts, and meet the product team), or watch it streamed live online.
The all-day event includes a ton of great content and presentations focused on Silverlight development. It includes a bunch of self-paced labs and walkthrough videos that enable developers that are new to Silverlight to follow along. It also includes great presentations that will teach new tricks to advanced of Silverlight developers. Most of the talks during the day will be focused on business application scenarios.
I’ll be kicking off the event with a 90 minute keynote on the future of Silverlight. In it I’ll cover our future Silverlight roadmap, and provide a first look at some of the exciting things coming. If you are interested in Silverlight you won’t want to miss it.
Click here to register to attend the event in person, or watch it live on the web. Both options are available at no cost.
Hope this helps,
Scott
P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu
Introduction I have been playing with the Azure toolset lately in preparation for taking beta exam 71-583, which is the MCPD (Pro) exam on Windows Azure technologies. I had worked with these technologies on and off over the past 2 years during the early releases. One thing I have heard from many people is that […]
A common requirement in many development scenarios is caching. In BizTalk implementations, this can be the case, mainly for performance reasons.
As a test I wrote 2 pipelinecomponents that handle 2 common issues with caching inside BizTalk. These sample components are performing similar tasks to two components of the Codit implementation framework. This is a framework we use at a lot of our customers.
Because both pipelinecomponents use the AppFabric cache, I wrote a small class that takes care of this.
public class CacheHelper : IDisposable
{
private string _cacheName = "default";
private string _region;
public CacheHelper(string region)
{
_region = region;
CreateRegion(_cacheName,region);
}
///
/// Creates a Region in a specified cache.
///
/// Cache name
/// Region name
private void CreateRegion(string cacheName,string region)
{
DataCacheFactory dcf=ConnectToCache();
if (dcf != null)
{
DataCache dc=dcf.GetCache(cacheName);
dc.CreateRegion(region);
}
}
///
/// Connect to a Cache server
///
/// The Datacache
private DataCacheFactory ConnectToCache()
{
//This can also be kept in a config file
var config = new DataCacheFactoryConfiguration();
config.SecurityProperties = new DataCacheSecurity(DataCacheSecurityMode.None, DataCacheProtectionLevel.None);
config.Servers = new List
{
new DataCacheServerEndpoint(Environment.MachineName, 22233)
};
return new DataCacheFactory(config);
}
~CacheHelper()
{
Dispose(false);
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
}
///
/// Gets a value from the cache in the specified region (class level)
///
/// Key linked to the data
/// The found data. If null --> not found in the cache
public string GetLookUpCacheData(string keyValue)
{
DataCacheFactory dcf = ConnectToCache();
var cache = dcf.GetCache(_cacheName);
string data = cache.Get(keyValue,_region) as string;
return data;
}
///
/// Store a value in the cache
///
/// Key
/// Data
public void StoreLookUpCacheData(string keyValue, object value)
{
DataCacheFactory dcf = ConnectToCache();
var cache = dcf.GetCache(_cacheName);
cache.Add(keyValue, value, _region);
}
///
/// Stores a value in the cache for a specified amount of time
///
/// Key
/// Data
/// Time to keep the data in the cache
public void StoreLookUpCacheData(string keyValue, object value,TimeSpan expires)
{
DataCacheFactory dcf = ConnectToCache();
var cache = dcf.GetCache(_cacheName);
cache.Add(keyValue, value, expires, _region);
}
}
This is a very simple implementation that will store values in the default cache and in a specified region.
Codetable mapping is a very common scenario in BizTalk implementations. In my example we will be translating countrycodes to the country name.
The values are stored in a SQL table. But every time we get a value, we are going to save it to the AppFabric cache.
When we want to get the same value again, we are not going to the database but we will get the stored value from the AppFabric Cache.
public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
{
// Set variables
biztalkMessage = pInMsg;
XmlReader reader = XmlReader.Create(pInMsg.BodyPart.Data);
XPathCollection xpaths = new XPathCollection();
//For this example we are going to use 1 xpath expression
xpaths.Add(this.XPath);
ValueMutator vm = new ValueMutator(handleXpathFound);
pInMsg.BodyPart.Data = new XPathMutatorStream(reader, xpaths, vm);
return pInMsg;
}
private void handleXpathFound(int matchIdx, XPathExpression matchExpr, string origVal, ref string finalVal)
{
CacheHelper ch = new CacheHelper("Countries");
string data = ch.GetLookUpCacheData(origVal);
if (data == null)
{
finalVal = GetCountryFromDB(origVal);
ch.StoreLookUpCacheData(origVal, finalVal);
}
else
finalVal = ch.GetLookUpCacheData(origVal);
}
private string GetCountryFromDB(string countryCode)
{
string country = string.Empty;
SqlConnection conn = null;
SqlCommand comm = null;
try
{
//Connect to look up database and retrieve the names of the products.
conn = new SqlConnection("Data Source=(local);Initial Catalog=CacheTest;Integrated Security=SSPI;");
conn.Open();
comm = new SqlCommand();
comm.Connection = conn;
comm.CommandText = string.Format("SELECT Country FROM Countries WHERE CountryCode='{0}'", countryCode);
comm.CommandType = CommandType.Text;
country = (string)comm.ExecuteScalar();
if(string.IsNullOrEmpty(country))
throw new Exception(string.Format("No country found for code {0}",countryCode));
}
catch (Exception e)
{
throw new Exception(e.Message + e.StackTrace);
}
finally
{
comm.Dispose();
conn.Close();
conn.Dispose();
}
return country;
}
As a sample scenario I took one I read about a few months ago. BizTalk had to stop messages that come in multiple times within 2 minutes.
So if there are more then 2 minutes between the messages, they should continue.
Normally this would involve a SQL table to store some information and some job to do the cleanup of this table.
But for my example I use AppFabric cache. There you have the option to store something in the cache for a certain timespan.
It is automatically deleted after this period.
public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
{
//Create hash
VirtualStream input = new VirtualStream(pInMsg.BodyPart.GetOriginalDataStream());
MD5 md5 = MD5.Create();
byte[] hash = md5.ComputeHash(input);
string hashString = Convert.ToBase64String(hash);
//check Cache
CacheHelper ch = new CacheHelper("DuplicateMessages");
string date=ch.GetLookUpCacheData(hashString);
if (string.IsNullOrEmpty(date))
{
//If not in cache yet, store it --> lifetime is 2 minutes
ch.StoreLookUpCacheData(hashString, DateTime.Now.ToString(), new TimeSpan(0, 2, 0));
}
else
{
//Throw error
throw new ApplicationException(string.Format("Duplicate Message. Already received at {0}",date));
}
//Put stream back to beginning
input.Seek(0, SeekOrigin.Begin);
return pInMsg;
}
This makes the implementation very easy and you will not need a SQL table or anything to store the information.
You could say that you can do this with a custom caching solution as well. But what about HA environments with multiple BizTalk servers?
AppFabric is a distributed cache. So it doesn’t mather on which server the message is processed. It will end up in the same cache and will be accessible on all the servers.
As you see, AppFabric caching has some advantages in BizTalk as well. The API is very easy to use and I got this to work quite quickly.
Tim D’haeyer, CODit
After much digging while trying to have a 100% silent install of BizTalk, I have determined that a silent install of the ESB 2.1 Toolkit is not possible.
I have made it as hands free as I think is possible. The below scripts are designed for a single server installation.
Here is the batch script that needs to be run:
1: start /w pkgmgr /iu:IIS-ASPNET;IIS-BasicAuthentication;IIS-WindowsAuthentication;IIS-IIS6ManagementCompatibility;IIS-Metabase2: msiexec.exe /qn /i "\\BizTalk Server 2010 Enterprise\BizTalk ESB Toolkit 2.1-x64.msi"3: "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\VSIXInstaller.exe" "C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Tools\Itinerary Designer\Microsoft.Practices.Services.Itinerary.DslPackage.vsix" /q /s:Ultimate /v:10.04: move "C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Bin\ESBConfigurationTool.exe.config" "C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Bin\ESBConfigurationTool.exe.config.old"5: copy "\\BizTalk Server 2010 Enterprise\BizTalk Server\ESBConfigurationTool.exe.config" "C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Bin\ESBConfigurationTool.exe.config"6: echo "Run ESBConfigurationTool.exe and press <Enter> when complete"7: pause8: "C:\Program Files (x86)\Microsoft BizTalk Server 2010\BTSTask.exe" ImportApp -Package:"C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Microsoft.Practices.ESB.CORE_NOBINDING64.msi" -ApplicationName:Microsoft.Practices.ESB -Overwrite9: "C:\Program Files (x86)\Microsoft BizTalk Server 2010\BTSTask.exe" ImportBindings /Source:"\\BizTalk Server 2010 Enterprise\BizTalk Server\Microsoft.Practices.ESB.CORE_Bindings.xml" -Application:Microsoft.Practices.ESB10: "C:\Program Files (x86)\Microsoft BizTalk Server 2010\BTSTask.exe" ImportApp -Package:"C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Microsoft.Practices.ESB.ExceptionHandling_NOBINDING64.msi" -ApplicationName:Microsoft.Practices.ESB -Overwrite11: "C:\Program Files (x86)\Microsoft BizTalk Server 2010\BTSTask.exe" ImportBindings /Source:"\\BizTalk Server 2010 Enterprise\BizTalk Server\Microsoft.Practices.ESB.ExceptionHandling_Bindings.xml" -Application:Microsoft.Practices.ESB12: msiexec.exe /qn /i "C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Microsoft.Practices.ESB.CORE_NOBINDING64.msi"13: msiexec.exe /qn /i "C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Microsoft.Practices.ESB.ExceptionHandling64.msi"14: iisreset15: pauseLine 1 adds the necessary additions to IIS
Line 2 actually installs the ESB components, which in turn adds the items to the start menu and unpacks the BizTalk msi’s etc.Line 3 installs the Itinerary Designer into Visual Studio
Line 4 archives the default configuration file for the ESBConfigurationTool.exe application
Line 5 copies a new configuration file for the ESBConfigurationTool.exe to use
Line 6 echoes that it is time to run the configuration tool
Line 7 waits until the configuration has been run
Line 8 imports the ESB Core application into the BizTalk Administration Console
Line 9 applies the bindings (this needs to be customized from the samples provided during the install (Line 2))
Line 10 import the ESB Exception application into the BizTalk Administration Console
Line 11 applies the bindings like Line 9
Line 12 installs the ESB Core application into the GAC
Line 13 installs the ESB Exception application into the GAC
Line 14 resets IIS since modifications were made in step 1
Here is a sample of the configuration file, so when you open up the ESBConfiguration tool, you simply need to enable each of the features, press the Apply Configuration tool and walk through it.
<?xml version="1.0" encoding="utf-8" ?><configuration><appSettings><!-- Management Database --><add key="ServerInstance" value="."/><add key="DatabaseName" value="ESBManagementDB"/><add key="Username" value="CORP\svcBTSHost"/><add key="Password" value="P4ssw0rd!"/><add key="BizTalkAppGroup" value="CORP\AppUsers"/><add key="BizTalkAdminGroup" value="CORP\ServerAdmin"/><add key="UseSqlAuthentication" value="False"/><!-- Itinerary Database --><add key="ItineraryServerInstance" value="."/><add key="ItineraryDatabaseName" value="ExceptionDB"/><add key="ItineraryUsername" value="CORP\svcBTSHost"/><add key="ItineraryPassword" value="P4ssw0rd!"/><add key="ItineraryBizTalkAppGroup" value="CORP\AppUsers"/><add key="ItineraryBizTalkAdminGroup" value="CORP\ServerAdmin"/><add key="BizTalkIsolatedHostGroup" value="CORP\IsolatedUsers"/><add key="ItineraryUseSqlAuthentication" value="False"/><!-- Core Services--><add key="CoreWebSiteName" value="."/><add key="CoreUserAccount" value="CORP\svcBTSIsoHost"/><add key="CoreUserAccountPassword" value="P4ssw0rd!"/><add key="CoreBizTalkIsolatedHostGroup" value="CORP\IsolatedUsers"/><!-- Exception Services--><add key="ExceptionWebSiteName" value="."/><add key="ExceptionUserAccount" value="CORP\svcBTSIsoHost"/><add key="ExceptionUserAccountPassword" value="P4ssw0rd!"/><add key="ExceptionBizTalkIsolatedHostGroup" value="CORP\IsolatedUsers"/><!-- Configuration--><add key="ConfigurationSource" value="False" /><add key="ApplicationName" value="ESB Toolkit" /><add key="ContactInformation" value="[email protected]" /><add key="AdminGroupName" value="CORP\SSOAdmin" /><add key="UserGroupName" value="CORP\AppUsers" /><add key="ConfigurationFilePath" value="C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\esb.config" /></appSettings><system.diagnostics><switches><add name="ESBConfigTool" value="4" /></switches><trace autoflush="true" indentsize="4"><listeners><add name="FileListener"type="System.Diagnostics.TextWriterTraceListener"initializeData="EsbConfigurationTool.log"/></listeners></trace></system.diagnostics></configuration>