Optimising VirtualBox

I blogged a couple of weeks ago about my issues running VirtualBox.At the time, I believed I had found an issue involving SuperFetch.In reality, turning SuperFetch off, while helpful, did not address the underlying issue.I eventually worked out that my machine is overheating whenever CPU usage of 40-50% is sustained over several minutes. Several things, or combinations of things, can cause this on my machine, including running full virus-checking scans in Forefront, SuperFetch and, of course, running VirtualBox. Every time the temperature goes up to about 85o, the power management throttles the CPU back to about 300MHz in order to let things cool down again. Once cooler (less than 80o), the CPU switches back to full speed. A cycle is set up that repeats for as long as the relatively high CPU load continues. I use the RightMark CPU Clock Utility to monitor core temperature.
The fan works OK, and my machine is not clogged with dust and detritus.It appears that my Dell Vostro 1720 simply cannot cope with sustained load. The Vostro range has documented overheating problems in relation to the NVidia GPU.Dell ended up modifying the BIOS to keep the fan running constantly to try to keep things cool.Sure enough, my fan never seems to switch off, but just can’t cope with sustained load.Short of convincing the company to buy me a new machine (some hope!), my only options are to adopt whatever strategy I can to reduce sustained CPU load to a minimum and to invest, at company expense, in a cooling pad.
How do I reduce sustained CPU load?Well, there are some challenges here.First, I avoid run full virus scans during working hours.Unfortunately, being a company machine, the policy is locked down and a full scan kicks off every Wednesday lunchtime.My only option has been to kill the scan and try to remember to run one manually before going to bed.
Turning SuperFetch off has helped.It doesn’t really use a huge number of cycles, but it does increase CPU load for several minutes at a time at certain times of the day.In combination with other things, this can be enough to cause my machine to overheat.I haven’t missed SuperFetch except when re-booting, which now takes a lot longer.
And now on to VirtualBox.I’m very dependent on running virtual images on my notebook for development, demonstrations, etc.Although Microsoft includes virtualisation with Windows 7, this is not an option for me.Microsoft only supports 32-bit images and, far worse, my experience was that most of my existing VPC 2007 images simply won’t run under the new WinVPC.Hence, I, like everyone else I know, have now switched to Sun/Oracle VirtualBox which happily runs all my old 32 bit images and allows me to create 64-bit images.Indeed, Microsoft is quite explicit on their web site that they only see WinVPC as a mechanism for handling legacy 32-bit apps that won’t run properly on Windows 7.I have yet to encounter any application that won’t run on Windows 7, so this strategy does little for me.
Unfortunately, VirtualBox can be quite intensive in terms of CPU usage, and causes my machine to overheat.Even if the image is completely idle, VirtualBox can constantly consume 30-50% CPU.I’ve been struggling with this for some time.Here is a summary of strategies to reduce CPU load:
On a single processor, dual core machine, configure each image to use a single virtual processor.This seems to help.I haven’t tested this on multi-processor boxes, so can’t say if the rule should be ‘always use a single virtual processor’ or ‘always use one less virtual processor than the physical processor/core count.
Avoid Oracle VirtualBox 3.2.8 and instead opt to use the old Sun VirtualBox 3.1.8 which is still available.The Sun version seems to be a bit less intensive than the current Oracle version.Oracle certainly introduced a number of changes and seems to burn through more CPU cycles.
And now, something I recently discovered. Try moving to SATA disk controllers. This has worked well for me.One of our system guys tells me that he suspects this better aligns VirtualBox with the controllers used on most modern notebooks, and cuts down the amount of work the machine has to do.I don’t know if this is correct, but I certainly see a significantly reduced load on the machine.
To move to SATA disk controllers, you need to stop you image and then go to ‘Settings/Storage’.Under the ‘Storage Tree’ panel, click the ‘Add Controller’ button and choose ‘Add SATA controller’.Now boot the image and make sure it has a valid network connection with Internet access.To load the required drivers, go to the Intel web site and download the latest version of their Matrix Storage Manager.You can currently find this on this page:
http://downloadcenter.intel.com/SearchResult.aspx?lang=eng&ProductFamily=Chipsets&ProductLine=Chipset+Software&ProductProduct=Intel%c2%ae+Rapid+Storage+Technology+(Intel%c2%ae+RST)
Once the manager is installed, shut down the image.Then go back to ‘Settings/Storage’, remove your hard disk from its current controller and add it to the SATA controller.Reboot you image, and hopefully things will improve.
Finally, if you have the same problems I have, invest in a cooling pad.Mine arrive at lunchtime today just in time for me to test it during a full Forefront scan.I placed the notebook on the pad, plugged it into my USB port (to run the fan) and watched the temperature reduce down by about 12/13o. For the next few hours it ran the full scan without going much above 60o, which is well inside the safety zone. At the same time, I was running, and using, VirtualBox. The scan is complete and I am down to about 54o. This still strikes me as a tad on the warm side, and I have to say I think there is probably some underlying fault on my machine, but our systems guys feel differently. So, for the time being, armed with VirtalBox optimisations and a cooling pad, I can once more enjoy a productive relationship with my notebook.

BizTalk Server 2010 released for manufacturing!!!

BizTalk Server 2010 has been released to manufacturing and will be available for purchase starting October 1st, 2010. The evaluation and free Developer editions are now available for download from the BizTalk Server website.

BizTalk Server 2010 is the seventh major release of Microsoft enterprise integration product for connecting systems within and across enterprises. BizTalk Server 2010 has 4 major themes – to help customers to:

·         Integrate Enterprise Applications Easily

·         Manage and Tune Solutions easily

·         Enhance Enterprise Interoperability

Let us look at each of themes in detail.

Theme: Integrate Enterprise Applications Easily

BizTalk Server 2010 helps developers put together applications that integrate various heterogeneous systems in an enterprise easily.  There are a number of new features and enhancements to the existing features that enable this easy integration.

Seamless access and mapping to Line-of-Business applications for new users and new scenarios

BizTalk Server 2010 includes out of box activities to connect and map data for AppFabric (.NET) developers. With these activities customers can now connect to LOB systems very easily and also use the award winning BizTalk Mapper and the mapping functionality within AppFabric.

 With this feature BizTalk Server 2010 now enables:

·         Out of box Line of Business Connectivity

·         Includes message data transformation

·         Delivered as WF 4.0 activities

·         Utilize existing BizTalk engine and tools

·         BizTalk LoB Adapters (SAP, Oracle, Siebel, Oracle DB, SQL Server)

·         Transformation Engine and Mapper

·         Provide LoB connectivity for scenarios that need better performance rather than reliability and control (e.g. Web)

·         Enables customers to develop solutions that need higher performance without the durability and reliability constraints.

More posts with detailed information on LOB access and mapper activities in AppFabric are here, here, here, and here.

 Better design experience of data transformation (maps)

The Mapper in BizTalk Server 2010 has been redesigned and enhanced to work easily with complex maps.

 A few highlights of the new Mapper are:

·         Reduce background “noise” using highlight propagation to show only the nodes that are relevant to the user

·         Auto-scrolling and coalescing of sibling nodes to help locate the nodes of interest easily

·         Search functionality to find the nodes of interest easily especially in large and complex maps

·         Improve productivity with Cut/Copy/Paste/Move/Undo functionality

·         Predictive match that help create maps faster 

More details on the new Mapper are available here.   

Built-in components for rapid development of RFID solutions

BizTalk Server 2010 helps developers build RFID solutions faster by providing out of box event handlers for duplicate elimination, TDT based filtering and dwell filtering. It also provides out of box mechanism for .NET apps to subscribe to RFID events, post events to BizTalk Server and also post EPCIS events to a Capture Service.

Theme: Manage and Tune Solutions easily

BizTalk Server 2010 helps IT Pros to easily tune their systems for performance and also manage their systems easily.  With the below features in BizTalk Server 2010, it is easier and faster to just get things done.

·         Easier to setup and update and migrate

·         Efficient performance and tuning and setting of run-time parameters with finer setting granularity and more intuitive and accessible settings

·         Better visibility and control of production environment with a new System Center Operations Management Pack

·         Enhanced security with Transparent Message Data Encryption

·         Compression option to speed up backup tasks

Let us look at the enhancements to settings and SCOM pack in details.

Settings Dashboard:

All the settings related to performance tuning are now consolidated in a single dashboard which is accessible from the Management Console. This settings dashboard provides:

·         Dashboard view for all performance related settings consolidated in a single place

·         Granular setting level options: group, host and host instance

·         For example – MaxReceiveInterval which is used to tune BizTalk solutions for high performance is now available at the host level. It is also split into separate settings – MessagePollingInterval and OrchestrationPollingInterval

·         Uniform access to settings through APIs

·         Ability to replicate settings between deployments using scriptable Export/Import operations

System Center Operations Manager Pack:

The SCOM pack has been redesigned and remodeled to help IT Professionals to easily monitor BizTalk installations and diagnose any problems. This is enabled with the enhancements below:

·         New model with separate application and deployment views

·         New alerts and diagnostics

·         Single representation of artifacts in a multi machine deployment

Upgrade Support:

The last two versions of BizTalk Server – BizTalk Server 2006 R2 and BizTalk Server 2009 can be upgraded to BizTalk Server 2010 seamlessly in-place.

 Theme: Enhance Enterprise Interoperability

BizTalk Server 2010 increases enterprise interoperability with the new, scalable TPM (Trading Partner Management) and new and enhanced Adapters and Accelerators. Now we will look at each of these in detail.

Scalable Trading Partner Management:

 

The new TPM enables easy on-boarding and lifecycle management of trading partners with the following features:

·         New Partner Management model that reflects typical B2B relationships.

·         Better and more accessible user interface with

·         Agreement templates and business profiles for rapid configuration

·         New TPM operator role to manage partner management

 

Another post with more details on TPM is here.

 

New and enhanced Adapters:

·         New FTPS adapter

·         Enhanced FTP Adapter

·        Supports files from read-only locations

·         Extends atomic write to text files

·         Fault tolerance and performance improvements

·         Support for new versions of LOB systems

·         Oracle EBS 12.1

·         SAP 7

·         JD Edwards 9.0

·         SharePoint 2010

·         SQL Server 2008 R2

·         Support for new Microsoft platforms

·         Windows Server 2008 R2

·         SQL Server 2008 R2

·         .NET Framework 4.0

·         Visual Studio 2010

 

As we discussed in this blog today, with the focus on making developing, tuning, and managing enterprise integration solutions and managing trading partners faster, simpler and easier, we hope that BizTalk Server 2010 will make it easy for enterprises to build composite solutions that span both on-premise LOB systems and also applications deployed onto the public cloud.

 

Here is another post that discusses the features of BizTalk Server 2010 along with a video.

Learn BizTalk Server 2010 at your own pace here.

 

To learn more regarding the BizTalk Server 2010 release and download our new free developer edition, please visit BizTalk Server website. You can learn more about Microsoft’s Application Infrastructure capabilities by exploring on-demand training at www.appinfrastructure.com.

 

BizTalk Sftp Adapter – New release 1.4 available on CodePlex

I’ve got quite a few request for Proxy support through the years, and eventually someone (no names) got tired of waiting, hired me as a consultant, and forced me to implement it. So I’m happy to say the adapter now comes with support for HTTP Proxy.

Support has also been added to:

  1. Enable "%UniversalDateTime%" macro on Receive Location (rename)
  2. "Leave File" on the Receive Location.
  3. Not throwing 10.000 exception to the eventlog (yes this is a feature)

Thanks to everyone giving feed-back through the codeplex site.

If you like it, please rate the project (If you don’t like it, you should not feel obligated to do so)

You will still need to set the appropriate SSH parameters such as SSH Host, User name and Remote path. These parameters will be sent to the proxy server, which in turn will act as “man-in-the-middle” and connect to the SSH server using the parameters provided by the adapter.

Using the WCF OperationContext from a Receive activity

The WF4 Receive activity shields you from a lot of the WCF pipeline. Normally that is a good thing but there are occasions where you want to know more about the incoming WCF request. Normally you can use the WCF OperationContext.Current to get at this information but with a workflow service this doesn’t work as it is null. The reason is that the workflow executes on a different thread.

 

So how do you get at the OperationContext?

The trick is to add a class implementing the IReceiveMessageCallback interface to the NativeActivityContext.Properties while the receive is executing and this will have it’s OnReceiveMessage() function executed with the OperationContext passed in. This means creating an activity to wrap the Receive activity like this.

public class GetMessageProperties : NativeActivity
{
    public Activity Body { get; set; }
 
    protected override void Execute(NativeActivityContext context)
    {
        var inspector = new GetMessagePropertiesInspector();
        context.Properties.Add(inspector.GetType().FullName, inspector);
        context.ScheduleActivity(Body, OnBodyCompleted);
    }
 
    private void OnBodyCompleted(NativeActivityContext context, ActivityInstance instance)
    {
        var inspector = context.Properties.Find(typeof(GetMessagePropertiesInspector).FullName) as GetMessagePropertiesInspector;
        if (inspector != null)
        {
            Console.WriteLine("ClientUri = {0}", inspector.ClientAddress);
        }
    }
}

 

The GetMessagePropertiesInspector, which implements the IReceiveMessageCallback, is quite simple. The main thing here is that it can be serialized along with the workflow state so make sure it contains the DataContract attribute.

[DataContract]
public class GetMessagePropertiesInspector : IReceiveMessageCallback
{
    [DataMember]
    public string ClientAddress { get; set; }
 
    public void OnReceiveMessage(OperationContext operationContext, ExecutionProperties activityExecutionProperties)
    {
        var properties = operationContext.IncomingMessageProperties;
        var endpoint = properties[RemoteEndpointMessageProperty.Name] as RemoteEndpointMessageProperty;
        ClientAddress = endpoint.Address;
    }
}

 

Creating and self hosting the workflow in a console app is easy, if you want to use the designer you will need to add an ActivityDesigner as well and you are good to go.

class Program
{
    static void Main(string[] args)
    {
        var workflow = CreateWorkflow();
        var host = new WorkflowServiceHost(workflow, new Uri("http://localhost:8080/MyService"));
 
        host.AddDefaultEndpoints();
 
        host.Open();
 
        Console.ReadLine();
        host.Close();
    }
 
    private static Activity CreateWorkflow()
    {
        var result = new Sequence();
 
 
        var receive = new Receive();
        receive.OperationName = "GetData";
        receive.CanCreateInstance = true;
 
        result.Activities.Add(new GetMessageProperties() { Body = receive });
        result.Activities.Add(new SendReply() { Request = receive });
 
        return result;
    }
}

 

If you want to work with the reply message there is also a ISendMessageCallback which does exactly the same for the result message.

 

Enjoy!

www.TheProblemSolver.nl

Wiki.WindowsWorkflowFoundation.eu

Frequently Asked Questions about the ASP.NET Security Vulnerability

Two days ago I published an important blog post about a security vulnerability in ASP.NET.  In it I discussed a workaround that we recommend customers use to help prevent attackers from using the vulnerability against your applications.

Below are answers to some common questions people have asked since then about the vulnerability.

Is Microsoft going to release an update to fix the vulnerability?

Yes.  We are working on an update to ASP.NET that we will release via Windows Update once it has been thoroughly tested and is ready for broad distribution.

Until the update is available, we will also publish details on workarounds (like the one described in this post) that can be applied immediately to help protect against the vulnerability.  Note that the workarounds are temporary – and will not be required once the update fixes the vulnerability in the underlying products.  They are intended to provide steps that you can take immediately until the update is available.

Is this an issue in ASP.NET or is it some cryptographic vulnerability?

This is a vulnerability in how ASP.NET uses cryptography in some circumstances that enables side-channel leaks through error responses.  The current ASP.NET use of encryption padding provides information in error responses that can be used by a malicious party.  We will be fixing this vulnerability in the security update.

Does this affect both ASP.NET Web Forms and ASP.NET MVC?

Yes – the publicly disclosed exploit can be used against all types of ASP.NET Applications (including both Web Forms and MVC).

Does this affect SharePoint?

Yes – the publicly disclosed exploit can also be used against SharePoint 2010.  The SharePoint team recently published a blog post that describes a workaround you can apply until we release the security update.

What would an attack look like on the network or in my logs?

The publicly disclosed exploit would cause the web server to generate thousands (or more likely tens of thousands) of HTTP 500 and 404 error responses to requests from a malicious client.

You can use stateful filters in your firewall or intrusion detection systems on your network to detect such patterns and block such clients.  The Dynamic IP Restrictions module supported by IIS 7 can also be used to block these types of attacks.

An attack attempt like this should also generate thousands of warnings in the application event log of your server similar to:

Event code: 3005

Event message: An unhandled exception has occurred.

Event time: 11/11/1111 11:11:11 AM

Application information:

    Application domain: c1db5830-1-129291000036654651

    Application Virtual Path: /

Exception information:

    Exception type: CryptographicException

    Exception message: Padding is invalid and cannot be removed.

Note that there are non-attack reasons to see this error as well (including cases where you have mismatched keys on a web-farm, or a search engine is following links incorrectly, etc), so its presence does not necessarily indicate an attack. 

The exception also does not mean that an attack was successful.  Implementing the <customErrors> workaround we have provided can protect your application from the public exploit, and ensure that these exceptions do not disclose information that an attacker can use against the application.

What does the <customErrors> workaround do?

A workaround you can use to protect your application from the public exploit is to enable the <customErrors> feature of ASP.NET, and explicitly configure your applications to always return the same error response – regardless of the error encountered on the server.  By mapping all error pages to a single error page, you can make it more difficult for an attacker using the public exploit to distinguish between the different types of errors that occur on a server. I covered how to implement the workaround in this blog post.

If you are using .NET Framework version 3.5 SP1 or 4.0, the workaround provides further protection by also helping to mitigate against potential timing analysis attacks.  The workaround uses the redirectMode="ResponseRewrite" option in the customErrors feature, and introduces a random delay in the error page.  These approaches work together to make it more difficult for an attacker to deduce the type of error that occurred on the server by measuring the time it took to receive the error.

Can I configure a custom 404 error page response and a default redirect for all other errors?

No. By doing this you are still letting an attacker draw distinction between a 404 and other errors. Homogenizing errors is a crucial component to mitigating this attack.  Note that this is a workaround until a security patch is available to fix the underlying product vulnerability. This workaround will not be required once we release a security update.

Am I vulnerable if I have my own custom error module?

If the responses that are sent out from your custom logging module do not let the client distinguish between error responses either through its content or time that it takes to serve out, then such a module is an adequate replacement for the customErrors workaround. These responses include both the entire HTTP response and the HTTP error code. If any of the above is not true at all times, then this is not sufficient.  Instead you should send out the same error response for all errors until the security update is available to fix the underlying vulnerability.

Should I be concerned about this vulnerability if I don’t store any sensitive information in my viewstate?

Yes you should.  There is a combination of attacks that was publicly demonstrated that can leak the contents of your web.config file, including any sensitive, unencrypted, information in the file.  You should apply the workaround to block the padding oracle attack in its initial stage of the attack.  The security update will fix this vulnerability.

What are best practices to secure my data within the web.config file?

It is always a best practice to encrypt sensitive configuration data within web.config files. That way if your web.config file is ever exposed, attackers can not use its contents maliciously.  This MSDN documentation describes how to encrypt web.config file configuration sections: http://msdn.microsoft.com/en-us/library/zhhddkxy(VS.80).aspx.  This tutorial also provides more samples of how to encrypt web.config file contents: http://www.4guysfromrolla.com/articles/021506-1.aspx

Why am I getting an error running the vulnerability detection .vbs script?

In my initial blog post I pointed at a .vbs script that you can run against a server to identify any applications within it that need to have their <customErrors> sections updated as a workaround against the publicly disclosed exploit. 

On IIS 7, the script requires you to have the IIS 6 management compatibility feature installed to be able to use this script.  To enable this, run Add/Remove Programs on workstations and Add Web Server Role Services on server operating systems and select the IIS 6.0 Management Compatibility feature under the “Internet Information Services” feature area. If this feature is already installed, please ensure that you are running the script with Administrator privileges.

Summary

We will post more information as our investigations continue, and we will post security advisory updates if the situation changes.  You can learn more about this vulnerability from:

Please post specific questions about the vulnerability in this forum on the www.asp.net web-site.

Thanks,

Scott

Financial Messaging Services Bus (FMSB) – Add-ons for ESB Toolkit

Summary: Financial Messaging Services Bus (FMSB) is a vertical industry implementation of Microsoft’s Enterprise Service Bus Toolkit 2.0 on top of BizTalk Server 2009 and BizTalk Accelerator for SWIFT. FMSB greatly improves time to market for many complex integration solutions especially in Banking and Capital Markets industries. This paper explains the rationale behind FMSB creation, provides a high%u2010level description of FMSB architecture, and discusses how FSMB is used to simplify application connectivity to SWIFT. FMSB helps software developers and solution architect by providing components and functionalities within engine which saves development time and give more value from the engine itself.

This document assumes the reader has a basic understanding of generic ESB concepts. For further reading on the Microsoft ESB Toolkit, refer to: http://www.microsoft.com/soa/solutions/esb.aspx Lets use this link: http://msdn.microsoft.com/en-us/biztalk/dd876606.aspx

Financial solutions (but this apply to any industry solution at the end), especially those implementing or leveraging a full-fledged messaging framework , can in fact constitute a foundation platform for the development of a specific application domain (payment or capital markets, government solution, manufacturing,..) where integration technology, data transformation, workflow management and other intermediary services are used to orchestrate transaction flows among different systems running on different, heterogeneous platforms. In addition to messaging, some commonly used services implementing specific behaviors are required for transaction processing, e.g., validation, routing, exception management, and repair. By developing these mechanisms as reusable services, the messaging infrastructure becomes more than an integration framework – it takes on the nature of a bus architecture where the lifecycle of a transaction can be mapped, calling the appropriate services as necessary. This is the essence of the FMSB (ESB): taking common processing services, abstracting and bundling them as reusable services that can be configured at implementation time and track execution KPI data as well as custom defined KPI. In addition, such services can be exposed to third party applications to leverage the preconfigured processing of the bus components, therefore enhancing client value. When defining FMSB, it is important to note that these services are business services, as defined by Microsoft Enterprise Service Bus (ESB) 2.0. The over%u2010arching concept is to use ESB to orchestrate all services and reuse them as needed.

The basic architectural elements of a financial services application can be categorized into the following segments or layers as shown in Figure 1.

Figure 1: Financial Application Architecture

When considering the architecture of a financial services application, FMSB sits in the layer known as “Business Process and Orchestration”, which is covered by the Microsoft technology stack, and provides integration, orchestration, transformation, and workflow services.

FMSB can be deployed directly to a financial institution client infrastructure project, but also embedded in a Microsoft partner application solution.

FMSB and Microsoft ESB

The FMSB is built principally on BizTalk Server because many of the services are implemented using BizTalk and Accelerator for SWIFT components. To conform to BizTalk ESB architectural best practices, the FMSB was developed upon the BizTalk ESB Toolkit. Most of the FMSB components are very generic and reusable for any solution built on ESB Toolkit.

The base ESB architecture is also the base architecture for FMSB as shown in Figure 2

Figure 2: FMSB and ESB

Financial Messaging Service Bus extends ESB by providing:

  • Resolvers which simplify solution creation by implementing support for:
  • Multipart messaging (Read Message Part, Replace Message Part)
  • Retrieve configuration from Dashboard (FMSB Value)
  • Retrieve complex configuration data for SWIFT service (SWIFT Service) (not covered as part of this post)
  • Storing itinerary designer value into itinerary runtime
  • Loopback adapter (message doesn’t leave message box)
  • Configuration model for defining BAM tracking data for service/itinerary execution
  • Service Broker Orchestration implementation
  • Silverlight Dashboard (built on Composite framework (previously known as Prism)) with 5 modules
  • Set of Financial services and itineraries together with configuration model (not covered as part of this post)
FMSB Architecture

FMSB Architecture is presented in the following figure:

Figure 3: FMSB architecture as add-on for ESB (red circles represent FMSB add-ons)

FMSB provides:

  • Core extensions of ESB (enhanced runtime, tracking KPI during itinerary execution)
  • Extended Exception handling (support for invoking pre-defined exception itinerary)
  • Loopback adapter (helps to mix messaging and orchestration services inside same itinerary)
  • Configuration database (new fmsb resolver extends BRE and UDDI resolvers and provides generic configuration on the Silverlight dashboard)
  • Silverlight self-service dashboard (for monitoring Live Data, view KPI Reports, Configuring KPI (BAM), SWIFT service administration)
  • Interact and FileAct (specific SWIFT SAG adapters) support (not covered as part of this post)
FMSB Components

FMSB has several modules which could run and be installed separately:

  1. CORE modules – those modules could be reused without SWIFT modules. Artifacts include resolvers, adapter, orchestration service broker, database, entity framework models..
  2. SWIFT modules – those modules are connected with BizTalk Accelerator for SWIFT (A4SWIFT) and pre-built for reusing in SWIFT scenario. SWIFT modules use Core modules. Requires A4SWIFT and BizTalk SWIFT adapters installed. (not covered as part of this post)
  3. Tracking modules – these module provide enhanced tracking capabilities over ESB and can be installed independently of other modules. They require the BAM infrastructure.
  4. Dashboard – modules for presenting Silverlight experience for working with Dashboard capabilities and configuration model. Independent of other modules.

Following Figure present the relation of all FMSB modules.

Figure4: Core + Tracking + Dashboard modules with configuration stores

Need for rich BI within ESB

Like an ocean surrounding an iceberg, business performance management (BPM) provides the business context for performance dashboards, which are layered applications built on a business intelligence and data integration infrastructure (i.e., the base of the iceberg). The most visible elements of a performance dashboard are the scorecard and dashboard screens, which display performance data using leading, lagging, and diagnostic metrics.

In custom implementation extracting relevant business data for Dashboard isn’t an easy task. By using ESB architecture ESB runtime (Dispatcher in messaging scenario and Advance method in Orchestration scenario) has a full control of every message flow inside ESB runtime. But, even with all this knowledge ESB2.0 doesn’t provide full tracking feature. With pure ESB runtime you can’t extract reports which give you answers on common questions:

  • How many itineraries/services worked in past ?
  • How many itineraries/services currently running ?

In any financial services application, it is very common to provide an answer to questions like the following:

  • How many payments have been processed today?
    • How many exceptions did we have?
    • How many were urgent requests?
  • How were today’s payments cleared?
    • How many were bulk payments?
    • How many were wire payments?
  • Domestic vs cross-border?
    • Who were our top 5 customers today?
    • What percentage of the total came from these 5

These are the main reasons why we enhanced the tracking capability of ESB toolkit.

Tracking architecture

Enriched ESB runtime (with FMSB assemblies „..V1.dll“) now has a capability to extract data by using new BAM interceptor. These data include:

  1. Itinerary data (start time, end time, name, version)
  2. Service data (start time, end time, business name, status,…)
  3. KPI inside Message body (user configure)

Interceptor will extract Itinerary and Service data from Itinerary header. KPI inside Message body would be extracted according to the configuration model and stored into BAM.

Administration of the system would define which itineraries and services should be tracked (ESB Itinerary DSL model), how to extract KPIs from message body, which services/itineraries and define tracking entity (Activity with Checkpoints analogy from BAM). See the screenshot below

With FMSB, configuration of KPIs isn’t done inside Excel (nor custom XML). Administrator of the system (or business person) could use Dashboard with Drag/Drop functionality to define all necessary data (Activities, Checkpoints, Cubes, Measures, Dimensions) together with service position where this data should be tracked.

This model is published in:

  1. Configuration model for tracking
  2. BAM star schema to persist tracked data.

During the runtime tracking Interceptor would read Configuration model and extract data from Message body as defined.

Dashboard

Dashboard presents visualization of the cubes inside SQL Analysis services. This is generic tool and could be re-used for any cube inside Microsoft SQL Analysis services.

  • Sources – Cubes from SQL Analysis services (OrderDocumentSource)
  • Measures – Defined measures for selected Cube (CountOf)
  • Dimensions – Defined Dimensions for selected Cube (CustomerName, RequestType)
  • Filter – Dimension for filtering (same as Dimension).

Dashboard provides several pre-defined type of reports (Column, Line, Pie, Bar, Area, Doughnut, Point, StackedArea) for any source. Following is the sample of Column report:

By selecting different type of report view would be redrawn with the same data.

FMSB installation creates BAM cubes for:

  • Storing Itinerary/Services

Those cubes provides source data for report like bellow (percentage of service execution):

  • Storing Itinerary/Services as Real Time Aggregation for live service view on the system
LiveData

LiveData view present IT real view on the system. See the screenshot below for details

LiveData presents:

  • Current Itineraries working status on the system – „How many itineraries currently working?“
  • Current Itineraries status – „Status of itineraries on the system?“
  • Current working services status – „How many services currently working ?“

Note: BAM system store data into BAM RTA Cubes with delay.

The above data provides great insight to IT administration to know exactly what the current system/service/itinerary load is.

Important Note: The Tracking architecture extension and the dashboard feature in FMSB are designed to be generic and can be used for any BizTalk ESB toolkit implementation. It is not restricted to financial services vertical. If the BizTalk implementation does not warrant the need for ESB toolkit the dashboard can still be used to view BAM data more visually.

Conclusions

FSMB provides great set of ESB add-ons. With core functionalities benefits are available either for developers or for business persons or for solution architect. By re-using ESB and BizTalk runtime FMSB provides solid foundation for any specific domain development.

My BizTalk infrastructure design baseline

How should I design my environment? What OS version? SQL Edition? BizTalk license? Etc. Etc. I get these questions frequently.

The only one true answer to this question is the architects favorite – “it depends”. And once you know the requirements – some of the things on which it depends – it will still be closely followed by its companion: “We need to test to know”.

Still, I think everyone has their favorite configuration – that they then add or deduct from based on the requirements. I do. This is how it goes.

Servers

Four. Two SQL, Two BizTalk.
Why? High availability for SQL and BizTalk. Load balancing on BizTalk machines.

OS

Windows Server 2008 (R2) Enterprise, on all machines.
Why? Clustering. Plain and simple.

SQL Edition

SQL Server 2008 (R2) Standard, on the SQL boxes.
Why? Off the BizTalk environments I have worked with they have only very seldom gone beyond two machines which is the only real limitation with Standard that I care about. I know of limitations to RTA in BAM and of performance gains with Enterprise, and sometimes that may be required – but not as a baseline.

SQL Instances

Four SQL Server Instances: BizTalkMgmt, DTA, BAM, MsgBox.
Why? Prepare for and maximize scale-out possibility. Simplify IO division. Help with memory reservation.

SQL IO

Three disks per instance: Data, Log and TempDb. Baseline is 40, 20, 15 GB. Varies with requirements.
SAN if available.
Why? Disks are un-expensive. IO is core to SQL. SQL is core to BizTalk.

BizTalk Edition

BizTalk Server 2009/2010 Enterprise
Why? Number one reason – you want to be able to go beyond a single server: Because you want load balancing and high availability.

Virtualization

Not in my baseline.
Why? Could be a requirement with certain customers and it certainly works, but I would recommend physical machines because I think it gives more bang for the buck. BizTalk can many times be a processor, memory and IO intensive application.

Processor

Quad cores for sure, Hexa or Octo if availability permits. One proc per server is enough with this amount of cores as a baseline. Requirements like high throughput messaging and processing may cause it to rise.
Why? Same as above. Higher ROI with multi-core.

Memory

4 GB minimum for a 64-bit OS, preferably 8GB or more total memory on BizTalk Servers.
At least 16GB on the SQL machines.
Why? Memory is a cheap commodity right now. Not the right place to be cheap.

Summary

The above is in no way thorough. It wasn’t meant to be. There is no “One Truth”. I stress that I call it a Baseline. It was meant to be a brief overview. It’s based on the most common questions and the most common requirements for my customers. Consider your own requirements. Mine might not match yours. “It depends”

Input

What’s your baseline? Where does it differ? Where have you drawn the same conclusions?

AppFabric Cache – Peeking into client & server WCF communication

The goal of this blog is to dig deeper into cache client & server communication, provide more insights into a set of configuration knobs and conclude with some best practices recommendation.

Introduction

If you are reading this blog, chances are that you are reasonably deep into AppFabric Cache, previously known as Velocity. If not, here is a quick 200 level summary with some useful links:

AppFabric Cache is Microsoft ‘s distributed caching solution. It is part of Windows Server AppFabric, and is available as an out of band release on Windows Server. You can get more information from http://msdn.microsoft.com/en-us/windowsserver/ee695849.aspx

Here are a set of scenarios for using a distributed cache:

  • Improving performance of your application
  • Reducing load on and scaling out the database tier
  • ASP.NET session store repository
  • Centralized in-memory repository for data shared by a set of services

Data suitable for caching can be broadly classified as:

  • Reference data: readonly / infrequently changing data shared by all users. Eg: catalog data
  • Activity data: read and write data, single user. Eg: shopping cart
  • Resource data: read and write data, shared by all users. Eg: airline tickets

Background

AppFabric Cache has 3 modes of installation – cache client, caching services and admin. When you install the cache service on a machine, there is a configuration phase for setting up a configuration store (XML file based or SQL Server or custom provider). When a set of cache servers is configured to point to the same configuration store, they constitute a cache cluster. AppFabric transparently manages the entire cluster – load balancing of data, data partitioning map, replication of data, availability of cache servers etc. Any .NET based cache client can reference the installed client DLLs and with the choice of config and/or code start using the cluster. The cache client would need access to at least one cache server, from which it would get the list of all the servers in the cluster. Objects can be stored and retrieved from the cache cluster using simple APIs. The term ‘client’ in this context may be misleading, it is from a cache usage perspective. It can actually be a .NET service that hosts business logic or an ASP.NET web application. The term usage is analogous to the same service being referred to as ‘SQL client’.

Cache Client – Server communication

The communication between the cache servers and cache clients uses the WCF channel model and net.Tcp binding. Cache Servers use the 22233 (default) TCP port for communicating with the cache client. There are some additional cache ports for more cache cluster functionality. In this context, there are several configuration knobs that will be useful to understand:

  • ChannelOpenTimeOut: wait time for a cache client to establish a network connection with the cache server
  • RequestTimeOut: wait time for a cache client to complete the cache operation and process the response.
  • MaxConnectionsToServer: specifies the number of channels that are opened from the cache client to the cache server(s)

Test scenario

We will have a simple .NET application that will instantiate the DataCacheFactory object, get a reference to the default cache, store a <key, value> and then retrieve the value. The application is running on CACHECLIENT-MACHINE and the cache server is running on CACHESERVER-MACHINE. In this case, we have a 1 node cache cluster.

Tools setup

We will use tracelog to capture the ETW events on CACHECLIENT-MACHINE as part of this communication. You can download the tracelog utility which is part of the Windows Software Development Kit.

Note: The Tracelog.exe file is not included with Windows SDK for Windows Server 2008 and the Microsoft .NET Framework 3.5. Therefore, install the Windows Vista SDK on Windows Server 2008.

Test Execution and Capturing Results

Now, let us run tracelog to start a session named ‘debugtrace’. The output will be stored in test.etl

C:\test>tracelog -start debugtrace -f test.etl -guid “c:\Windows\system32\AppFabric\Manifests\ProviderGUID.txt” -level 5 -cir 512

Note: If you have issues in accessing the ProviderGUID.txt, copy it to another folder and retry the command.

Then let us execute the following piece of code:

public
static
DataCacheFactory myCacheFactory;

DataCache cache;

List<DataCacheServerEndpoint> servers = new
List<DataCacheServerEndpoint>(1);

string srvName = “CACHESERVER-MACHINE”;

servers.Add(new
DataCacheServerEndpoint(srvName, 22233));

DataCacheSecurity security = new
DataCacheSecurity(DataCacheSecurityMode.None, DataCacheProtectionLevel.None);

DataCacheClientLogManager.ChangeLogLevel(System.Diagnostics.TraceLevel.Off);

//Create cache configuration

DataCacheFactoryConfiguration configuration = new
DataCacheFactoryConfiguration()

{

Servers = servers,

LocalCacheProperties = new
DataCacheLocalCacheProperties(),

SecurityProperties = security,

MaxConnectionsToServer = 1

};

myCacheFactory = new
DataCacheFactory(configuration);

Console.WriteLine(“Connected to {0}”, srvName);

Console.ReadLine();

cache = myCacheFactory.GetDefaultCache();

Console.WriteLine(“Simple PUT GET starts”);

cache.Put(“test”, new
object());

object foo = cache.Get(“test”);

Console.WriteLine(“Simple PUT GET ends”);

Console.ReadLine();

Here is the output from the application:

Now let us stop the trace collection.

C:\test>tracelog -stop debugtrace

NOTE: ETW tracing data is not localized in the V1 release. This analysis is based on the format and event data relevant to the V1 release.

Analyzing the output

Now, let us open test.etl from Event Viewer and analyze some of the entries. Let us split the entries into the following areas:

DataCacheFactory creation

  • DRM instance created.
  • Creating channel for [net.tcp://CACHESERVER-MACHINE:22233]
  • channel for [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 36610825.
  • Channel open succeeded – [Endpoint=[net.tcp://CACHESERVER-MACHINE:22233]; Lockstatus=12; ChannelState=Opened; Status=ChannelOpening]

Accessing the cache

  • GetCache: Creating the Named Cache ‘default’
  • GenericCache: Constructing Cache: default
  • Initialize: Completed
  • 1:-1′ GET_NAMED_CACHE_CONFIGURATION;Routed;default;;;;Version = 0:0 – Request saved for correlation.
  • ‘1:-1’ GET_NAMED_CACHE_CONFIGURATION;Routed;default;;;;Version = 0:0 – Sending request to [net.tcp://CACHESERVER-MACHINE:22233].
  • Message sent to [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 36610825.
  • ‘1:-1’ GET_NAMED_CACHE_CONFIGURATION;Routed;default;;;;Version = 0:0 – Send result – Status=Success.
  • ‘1:-1’;Ack;UNINITIALIZED_ERROR – Response got from remote DOM.

PUT Operation

  • SendReceive: Begin: ‘2:-1’ PUT;Routable;default;;;test;Version = 0:0
  • SendMsgAndWait: Serializing Object: msgId = 2
  • ‘2:-1’ PUT;Routed;default;Default_Region_0760;1043303394;test;Version = 0:0 – Starting to process.
  • ‘2:-1’ PUT;Routed;default;Default_Region_0760;1043303394;test;Version = 0:0 – Destination – [net.tcp://CACHESERVER-MACHINE:22233].
  • Message sent to [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 36610825.
  • ‘2:-1’;Ack;UNINITIALIZED_ERROR – Response got from remote DOM.
  • SendReceive: Request’s 2 response’s status is Ack : ErrorCode=UNINITIALIZED_ERROR

GET Operation

  • SendReceive: Begin: ‘3:-1’ GET;Routable;default;;;test;Version = 0:0
  • SendMsgAndWait: Begin: msgId = 3
  • ‘3:-1’ GET;Routed;default;Default_Region_0760;1043303394;test;Version = 0:0 – Starting to process.
  • ‘3:-1’ GET;Routed;default;Default_Region_0760;1043303394;test;Version = 0:0 – Destination – [net.tcp://CACHESERVER-MACHINE:22233].
  • ‘3:-1’ GET;Routed;default;Default_Region_0760;1043303394;test;Version = 0:0 – Sending request to [net.tcp://CACHESERVER-MACHINE:22233].
  • SendMsgAndWait: Request is Pending, msgId = 3
  • ‘3:-1’;Ack;UNINITIALIZED_ERROR – Response got from remote DOM.
  • ‘3:-1’ GET;Routed;default;Default_Region_0760;1043303394;test;Version = 0:0 – Request completed on response – ‘3:-1’;Ack;UNINITIALIZED_ERROR.
  • MyCallback: Begin Processing Response
  • ‘3:-1’;Ack;UNINITIALIZED_ERROR – MyCallback: End Processing Response
  • SendMsgAndWait: DeSerializing Object, msgId = 3

Summary

  • Serialization of the object happens on the cache client in a PUT operation (Step 2) and de-serialization happens on the cache client in a GET Operation (Step 11)
  • Responses are processed asynchronously using I/O threads.
  • Instantiation of a DataCacheFactory object includes a set of policy settings to control Local cache, ChannelOpenTimeout, RequestTimeout, Security etc settings as seen from the code snippet above. The instantiation creates a set of internal data structures (DRM, ThickClient).
  • There was 1 DataCacheFactory used in this sample. This can be found in the event trace which shows only 1 ‘DistributedCache.ClientChannel.Client1’ instance. Here is an extract from our trace collection.

Source

DistributedCache.ClientChannel.Client1

Param

Client channel opened.

  • maxConnectionsToServer was set to 1. This can be found out from the event trace which shows only 1 channelID being used. (ChannelID = 36610825). This can also be validated by looking at the perfmon counter ‘TCPv6:Connections Established’ which shows the connections established from this machine.

Source

DistributedCache.ClientChannel.Client1

Param

Message sent to [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 36610825

  • Changing maxConnectionsToServer = 3 in the code snippet above, collecting and analyzing the trace data now shows the additional ChannelIDs part of the same DataCacheFactory instance

Source

DistributedCache.ClientChannel.Client1

Param

channel for [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 36610825

Source

DistributedCache.ClientChannel.Client1

Param

channel for [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 62130720

Source

DistributedCache.ClientChannel.Client1

Param

channel for [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 4593845

  • Increasing the number of DataCacheFactories to 2 and maintaining maxConnectionstoServer=3 in our sample, let us collect the trace once again. Now we can see 2 clients for the DataCacheFactory instances (with the internal DRM , ThickClient objects) and each of them having 3 channels created, so 6 in total.

Source

DistributedCache.ClientChannel.Client2

Param

channel for [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 61325552

Source

DistributedCache.ClientChannel.Client2

Param

channel for [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 29028787.

Source

DistributedCache.ClientChannel.Client2

Param

channel for [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 36610825.

Source

DistributedCache.ClientChannel.Client1

Param

channel for [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 35885827.

Source

DistributedCache.ClientChannel.Client1

Param

channel for [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 37710717.

Source

DistributedCache.ClientChannel.Client1

Param

channel for [net.tcp://CACHESERVER-MACHINE:22233] ChannelID = 7995840.

This can also be validated from perfmon by looking at the set of connections established which shows the increase in 6. Here are the before and after snapshots.

Before


After

Best Practice Recommendation

  • Even though having just 1 cache server in the client config would suffice, it is recommended to maintain as many cache server hostnames in the config file. *When a cache client connects to one of the cache servers, it gets the routing table which has the partitioning logic to access other cache servers. Including more cache servers helps with more resiliency for the initial connection. If the logic is implemented in code, building a host lookup service will work better.
  • Instantiating a DataCacheFactory object creates several internal data structures (DRM, ThickClient), minimizing the number of DataCacheFactories and creating them in advance (on a separate thread) is recommended. A singleton pattern should work for most scenarios; do not create one DataCacheFactory object per cache operation. When you need to have different policy settings, for example, if local cache is required only for a set of named caches, then having different DataCacheFactory objects will be appropriate.
  • Default setting for ChannelOpenTimeout is 15 seconds, you can set it to much lower values if you want the application to fail fast while opening the channel.
  • Default setting for RequestTimeout is 10 seconds, do not set it to 0. If you do, your application will see a timeout on every cache call. Changes to the default value need to take into account the workload and physical resources (client machine configuration, cache server configuration, network bandwidth, object size, ratio of GETs Vs PUTs, number of concurrent operations, usage of Regions, etc).
  • Setting maxConnectionsToServer=1 (default) will work in most situations. In scenarios, when there is a single shared DataCacheFactory and a lot of threads are posting on that connection, there may be a need to increase it. So if you are looking at a high throughput scenario, then increasing this value beyond 1 is recommended. Also, be aware that if you had 5 cache servers in the cluster, if the application uses 3 DataCacheFactories and if maxConnectionsToServer=3, from each client machine there would be 9 outbound TCP connections to each cacheserver, 45 in total across all cache servers.

    Note: There will be a set of subsequent blogs for explaining capacity planning and debugging timeouts which will provide more details on the usage of these configuration knobs.