Windows Workflow Foundation 4 makes it real easy to create workflow services that do long running work on a server. However when we are doing long running work there could be an issue with lots of workflows being started and too many workflow instances competing for the same data and threads thereby causing problems like database timeouts or thread pool starvation.
To simulate a busy workflow service I have the following sample workflow:
The workflow service is started using a WCF request, returns a response, prints a start message, does some work and prints a message that it is done. The “long” running work is simulated by this activity:
public class MyActivity : CodeActivity
{
protected override void Execute(CodeActivityContext context)
{
Thread.Sleep(15000);
}
}
Of course you would never do this for real, just imagine some complex database queries being done instead of a Thread.Sleep() []:)
Adding a simple client application like this:
class Program
{
static void Main(string[] args)
{
for (int i = 0; i < 1000; i++)
{
Console.WriteLine("Sending {0}", i);
var proxy = new ServiceClient();
proxy.Operation1(i);
}
Console.ReadLine();
}
}
I originally expected the loop to quickly start 1000 workflows and the workflows to complete after 15 seconds. Turns out this isn’t the case [:(] Even though the SendReply has finished the WCF response is not actually send until the complete workflow is finished. The reason is the workflow will run as many activities as it can before relinquishing control. And that happens to include my Thread.Sleep().
The trick to making the workflow service return the response to the client sooner is to add a Delay activity with a short timeout, I am using 1 second here, just after the SendReply activity. Now the first 115 or so workflows are launched quite quickly and only after the WCF response is returned to the client does the actual work start. The problem is we have over 100 workflows active at the same time and that could be the cause of problems so we want to start throttling our workflows so a limited number runs at the same time.
Throttling the Workflow Service
When we are using WCF we can use the serviceThrottling behavior to limit the number of concurrent sessions, request and server instances. The incoming WCF requests for a workflow service respect these throttle limits just like any other WCF service. However after the SendReply we are no longer part of the WCF request handling. However the serviceThrottling maxConcurrentInstances setting is still respected for running workflow service instances. So adding the following to our workflow service behavior would mean no more than 10 workflows are active at any given moment.
<serviceThrottling maxConcurrentInstances="10"/>
This by itself isn’t quite the solution when a client wants to start 1000 workflows as the first 10 will be started and the next can only be started as the first are finished.
Creating a batch of Workflow Services
If we want to be able to start 1000 workflows quickly and have them execute with a maximum of 10 at the time we need to add the sqlWorkflowInstanceStore to the mix. By doing this, and increasing the Delay duration a bit we can quickly start 1000 workflows, persist them to disk and unload them. Only when that is done do we allow them to run, a few at the time.
To achieve this I have changed to workflow to the following structure with the Delay set to 10 seconds. You should use a duration that allows you to start all workflow instances before the actual execution starts.
With the following configuration file:
<system.serviceModel>
<behaviors>
<serviceBehaviors>
<behavior>
<serviceMetadata httpGetEnabled="true"/>
<serviceThrottling maxConcurrentInstances="10"/>
<sqlWorkflowInstanceStore connectionString="Data Source=.\sqlexpress;Initial Catalog=WorkflowInstanceStore;Integrated Security=True"/>
<workflowIdle timeToUnload="00:00:00"/>
</behavior>
</serviceBehaviors>
</behaviors>
</system.serviceModel>
The result is 1000 workflows being kicked of quickly, unloaded to SQL Server and then reloaded and executing on their own time, 10 at the time. Sweet for batch jobs kicking off a large number of workflows [:)]
Download the code here.
Enjoy!
www.TheProblemSolver.nl
Wiki.WindowsWorkflowFoundation.eu