ACSUG Meeting – Building SOA with Microsoft Technology, October 9th – Ulrich Roxburgh

by | Sep 21, 2008 | BizTalk Community Blogs via Syndication

 
Auckland Connected Systems User Group
 
Building SOA With Microsoft Technology
Ulrich Roxburgh
 
 
 
Ulrich Roxburgh has worked for Microsoft Consulting Services for 9 %u00bd years, in various capacities ranging from Senior Consultant to Managing Consultant, in both New Zealand and Australia. He now works as the main consultant for Services2 Ltd. providing premium consultancy services in the areas […]

NCache and BizTalk Cross Referencing Example

by | Sep 21, 2008 | BizTalk Community Blogs via Syndication

Article Source: http://geekswithblogs.net/michaelstephenson

Following a recent post about the different approaches to caching you might consider when implementing reference data mapping in BizTalk one of the things that stands out most was that the solutions where a team had used a caching approach often resulted in them not using the BizTalk Cross Referencing features. As I’ve mentioned many times I prefer to use this unless there is good cause not to (there are reasons where you might not want to) but I feel development teams often ignore or don’t consider the impact of adding custom databases to a solution without consideration for the extra work this requires in development, testing , deployment and management.

In most cases why do you want to do this when you already have a data store designed for this purpose? One criticism I would make of BizTalk is that the product does not do a very good job of making it easy for people to use the cross referencing featured from a developer experience but these can all be worked around with few problems.

Anyway I have decided that I will produce this sample showing how I have combined the use of NCache and BizTalk Cross Referencing to get a solution which does not need custom databases yet can still have a high performance caching solution which will not increase the BizTalk hosts process memory unnecessarily. The sample can be downloaded from the bottom of the article.

Prerequisites for the sample

You can obviously review the code in this sample, but if you want to run it you will need to do the following things:

  • Install NCache Express Edition

NCache Express Edition is available free from the following link http://www.alachisoft.com/ncache/. I assume you will be installing this to the default location, but if not you might need to modify the msbuild script where I configure NCache.

  • Modify Cross Referencing Setup

In the SetupFiles.xml file in the solution it contains the xml used to setup the cross referencing data in BizTalk. This file requires absolute paths to work so you will need to tweak these to suit your location. The below picture shows what the xml looks like.

Setting up the sample

In the sample you will notice there is a file which is called Setup.cmd. If you run this file it will perform the appropriate actions to configure things for this sample. The actions it will take are as follows:

  • Stop the cache in NCache if it is already running
  • Clear the BizTalk cross referencing tables
  • Stop the NCache windows service
  • Copy the pre-configured NCache config files to their appropriate places to configure NCache with the cache we will use in this sample
  • Start the NCache windows service
  • Start the custom cache
  • Load the BizTalk cross referencing tables using BTSXRefImport

These actions are all done in an msbuild script (picture below) which should make it easy for you to see how this is done.

You should now be able to run the sample.

My Cross Referencing Component

To keep the sample simple I have developed a component which will provide an interface which is the same as that provided by BizTalk cross referencing. I provide a class called CrossReferencingFacade which implements the fa%u00e7ade pattern to give you an easy way to obtain the common and application specific id’s. The below picture shows this:

There is also a test in the test project which shows how to consume this component. It is as easy to consume as the BizTalk cross referencing dll. If you look in the CrossReferencingManager class you will see there are two key methods which are discussed below:

  • LoadXRefIDData

This will use some data access code to retrieve all of the cross reference data for one specific type of cross reference (xrefId) for example all of the mappings for Product Type. It will then return them to the calling method.

  • GetCrossReferenceIDData

This method will check the cache to see if the data is already there for the requested cross reference data type. If present it will be returned from the cache, and if not the data will be loaded using the LoadXRefIDData, and then placed in the cache.

The result of this means the data is cached once for both the GetAppID and Get CommonID methods.

One interesting bit on this (and there may be better ways to do this) is that to allow you to search for the appropriate mapping data from the same source by both CommonID and AppId I have held the data in a container object which houses a dictionary of the reference data with a unique key for each one, and then I also have 2 dictionaries of the app specific keys and common id keys. This just makes it possible to hold the data just once but search for it in different ways. As mentioned I’m sure if I have a think about this there are better ways to do this but it will do for this sample. (note although this last bit may have sounded overly complicated this is encapsulated so the consumer does not need to care about this)

The NCache bit

So from the above hopefully you can see I have provided a handy way to use BizTalk cross referencing within this sample. The next thing to discuss is NCache. I believe there are a number of additional features which come with the Enterprise version such as security features and tools to manage caches so for any production usage I would definitely recommend that version. For the purposes of this sample the Express edition is more than sufficient.

You can see from the below picture that the code to interact with NCache is very simple.

(Note: In the above picture you can only partly see it, but the cache allows you to insert objects with expiration parameters and also a cache dependency)

With NCache one of the things that I like is I don’t need to worry about configuration within my application, so you will notice there isn’t any app.config files in the sample which are used by the consumers of the cache. That said there is some configuration for the caching service. You will see in the NCache folder there is some config which controls how your caches are setup. The below picture from the config.ncconfig file shows how I have configured my cache for this sample.

You will notice here that I’m able to control if my cache runs in process or out of process which is how I’m able to move the cached data outside of my Biztalk host process and there are a bunch of other possible settings. This configuration is held along side the caching service.

Plugging it into BizTalk

This component is now very easy to add to a BizTalk implementation by using the call external assembly feature of the scripting functoid to call the component. You will now be able to use the cross referenced features of BizTalk but with out of process caching of the data.

Summary

Hopefully you will see that it is not that difficult to implement a good caching solution which has addresses a combination of the considerations I discussed in my previous article. I quite like this approach and based on my limited experience of the different caching systems available I would probably at present choose NCache over Memcached because it is an established 3rd party system which comes with additional tools and features to support it. That said I will be keeping an eye on the “velocity” project as I think this will definitely be one to watch for the future.

If you have any experiences with this I would be interested to hear your thoughts. The sample is available below:

Caching Strategies for Reference Data Mapping with BizTalk

by | Sep 21, 2008 | BizTalk Community Blogs via Syndication

Article Source: http://geekswithblogs.net/michaelstephenson

I’ve been asked the same question a few times recently by a couple of BizTalk projects about how to map their reference data. When this question comes up we often get involved in a discussion about the pros and cons of caching the reference data and increasing memory usage versus hitting the database every time.

As a rule I tend to use the BizTalk Cross Referencing features for this data mapping unless there is a specific requirement which requires some custom approach. I’ve blogged about this kind of thing a few times before but I thought its worth a post with some thoughts on the different approaches I’ve seen used when people have wanted to use caching.

I mentioned in a previous post that the Value cross referencing features already implement a simple caching mechanism. In my opinion though the value cross referencing is aimed more at mapping data type values between types of systems rather than business reference data which would be held in instances of systems which is what I feel the ID cross referencing is aimed more at.

Anyway when it comes to this design decision the things people are usually trying to balance are as follows:

  • Performance – If I have a lot of things to map I don’t want to be hitting the database thousands of times
  • Performance – If a I cache the reference data is there a risk it will consume a fair bit of memory and potentially cause throttling based on host process memory threshold being hit
  • Managability – If I cache the data it will have an instance of the cache in each host instance that uses it. How will I ensure these stay in sync
  • Managability – Caching will mean I have to restart all the hosts when the data changes

There are a number of possible ways to solve this problem and each have their own considerations which are discussed in the rest of this article.

Simple Singleton Approach

This is probably the most common approach I’ve seen. In this approach I’ve normally seen a custom database implemented to manage the reference data. The developer would then implement a custom data access method and a singleton which would be used to control access to the reference data. This is a pretty standard use of the singleton pattern. In this approach I think some of the considerations which need to be made are:

Pro’s

  • Fast access to the data
  • Easy to implement in terms of the C# coding

Con’s

  • In most cases there is additional development of a database to manage the data. This then involved additional development/testing/deployment and management work
  • The data is cached in the host process so you need to watch for the impact on the process memory of the BizTalk host
  • If you access this reference data from BizTalk maps running in different hosts then you may end up with multiple instances of the cached data on each server
  • By default your cache usually will not detect changes to the underlying data, however with additional coding you can monitor the custom data and update any changes
  • In most cases the hosts need to be restarted to pick up changes
  • The cache will not be cleared when the data is no longer used

Caching the Response from a Web Service

Sometimes I’ve seen an approach where a custom database has been implemented then a web service fa%u00e7ade has been implemented on top of it. The web service will access the data and return it. In consuming this from BizTalk a C# assembly has been developed which uses the web service to get the reference data which is then consumed by a map.

Pro’s

  • The caching is outside of the BizTalk process
  • The caching can be relatively easily configured
  • If the web service is located on the BizTalk box then a local machine hop would be quicker than going remotely, and also with WCF you could optimise this further using appropriate channels

Con’s

  • There is a lot of additional development in this approach, custom coding of the web service, development of the database
  • There is a lot of additional management and deployment effort for the database and virtual directories etc for the web service

Using the HTTPCache

In this approach I’ve normally seen it implemented in the same way as the singleton approach above. The key difference is that the reference data is usually held locally in a static hash table in the singleton approach where as in this approach the HttpCache object from the System.Web namespace is used. This gives a couple of options around a sliding and absolute expiration which will remove unused data from the cache helping to control the memory usage. You can also add one of the .net cache dependency objects which would allow you a way to detect changed and refresh the cache.

Pro’s

  • Would be fast access to data
  • Relatively simple to implement ways to detect changes
  • Ability to clear the cache for unused data

Con’s

  • Again usually has a custom database for the reference data
  • This is in process caching so you need to be aware of the memory usage

Using Enterprise Library/Caching Application Block

Enterprise Library has a caching block which provides a number of features which could help you solve this problem. One of the key benefits of enterprise library is that it supports different types of stores for the cached data including:

  • Null – means just stored in memory
  • Database
  • Isolated Storage

If I remember right the cache supports the same features as the HTTPCache approach which allows you to have a dependency and also expirations. There is an article at the following location which discusses using Enterprise Library Caching in BizTalk http://www.malgreve.net/2007/07/using-enterprise-library-in-biztalk.html.

Enterprise Library can also integrate with external backing stores to support out of process caching.

Pro’s

  • Ability to abstract the caching store from the consuming code
  • Standard caching feature set

Con’s

  • Again usually some requirement for custom data store for reference data
  • Enterprise Library usually required lots of configuration to setup and manage
  • Most commonly cached in process so near to be aware of memory usage

Out of Process Caching

One approach I quite like involves caching the data outside of the BizTalk process. This provides the benefit that you can cache without having to worry about the impact on the BizTalk process memory usage. There are a number of caching tools which you can use to help here such as:

Alachisoft offer an express version of their caching product which is free and a version for a relatively small cost which comes with some management tools for their distributed caching system.

Memcached is an open source distributed caching system. I know of some guys who have used this very successfully on a .net project with a major UK company.

  • Velocity – http://www.microsoft.com/downloads/details.aspx?FamilyId=B24C3708-EEFF-4055-A867-19B5851E7CD2&displaylang=en

Velocity is an initiative at Microsoft at present to create a distributed in memory caching platform. I feel that as this evolves it is important to keep an eye on this as it will in the future be likely to become the best approach to this.

These distributed caching systems offer the benefit of taking the memory usage out of your process, but offer fast access to the data via their API. Most of these products also offer high availability and synchronisation across a group of caches when you distribute them across your server group. I have in particular looked at NCache for this example and it is setup as a windows service which you would deploy on each BizTalk box. These services would then be configured to work as a cluster meaning they would synchronise themselves when changes were made.

Pro’s

  • Out of process caching offers still fast access to the cached data, but removes the likely hood that the cache might affect BizTalk performance
  • These caches are designed for high performance such as NCache which is intended for high performance customer facing ASP.net applications
  • They can be integrated with caching frameworks such as Enterprise Library (NCache comes with this out of the box)
  • NCache supports cache dependencies and expiration

Con’s

  • Again requires work and management of additional components. I think NCache (the buy version) offers a better management and operations
  • Potentially brings up the 3rd party or open source debate around which cache system to use

Summary

Hopefully this article has highlighted the many options available when you are considering a caching solution to support your BizTalk implementation. There are many considerations which can be made and there isn’t always a one size fits all rule like in most design decisions. I think some of the things that stand out from this discussion are that most of the approaches above always end up using a custom database to manage the reference data. I think in a future post I will look at how to combine some of the approaches discussed here with the BizTalk Cross Referencing features to produce a fairly simple yet effective combination of all of the approaches.

WCF Technology Specilist

by | Sep 21, 2008 | BizTalk Community Blogs via Syndication

On Friday I passed the “70-503 – TS: Microsoft .NET Framework 3.5 – Windows Communication Foundation Application Development”, so that makes me a “Technology Specialist” in WCF. This is the third technology specialist (TS) exam I have taken, the others being BizTalk Server 2004, and BizTalk Server 2006, so it felt good to broaden the spectrum a bit. I prefer the TS exams to the regular .net exams; I have the MCSD certification, which required passing four .net exams and a server product exam (I chose SQL Server 2000 for that). The TS exams are much more focussed instead of covering a broad area, and the questions tend to be tougher, but more focussed towards hands-on experience.
If you are thinking of going for the exam, make sure you really know the topics that are listed in the official preparation guide here. The best book that I found for covering the basics was “Learning WCF: A Hands-on Guide” by Michele Leroux Bustamante. This will cover most of the stuff you need for the exam, and go into sufficient depth on all the topics covered, there’s also about 20 or so labs that you should run through to get the hands-on experience. If you are fully knowledgeable with everything covered here you are well on your way to pass. There are some points on the official preparation guide that are not covered in Learning WCF, so you should make sure you have the 3.5 features covered and look at consuming non WCF services.

Basic Server Diff Tool

by | Sep 20, 2008 | BizTalk Community Blogs via Syndication

We had a few problems on a project recently where the server builds did not set machines up exactly the same. This is a pretty major problem as if the software installed across your BizTalk Group you can not expect to receive consistent or expected results. Based on this I reviewed a few servers using PsInfo recently but being a manual thing its a bit of a pain trying to identify any differences.

Im sure there must be tools to do this but a quick google didnt throw up too much joy so I knocked up a quick tool to help.

The tool will let you list a bunch of servers which you expect to be the same and will then run PsInfo agains them and report any differences which are identified.

There are a couple of things to note here though that it will not do things like compare registry settings etc, and ive not used the switch to get disc information from psinfo as if the servers have been used then their disc free space is likely to differ.

Anyway its a bit basic but saved me some hassle so ive shared it on codeplex; http://www.codeplex.com/ServerDiff

How To Create a Portable Office (Network Connectivity)

by | Sep 19, 2008 | BizTalk Community Blogs via Syndication

Phones

AT&T Tilt

I’ve carried an AT&T Tilt for nearly a year now as my primary personal phone.  I truly love this phone, Windows Mobile 6, GPS, WiFi, full keyboard, 3 mega-pixel camera, and high speed networking is fabulous.  I’ve been with AT&T for several years now.  I was an AT&T Wireless customer, then I was a Cingular customer, now I’m an AT&T customer again so I’ve got a really good deal on a tethering plan for this unit as well which makes it Option 3 in my always connected plan.

T-Mobile Dash

I’ve just begun to carry a T-Mobile Dash as my work phone.  It is replacing the 8700g (below) which I’ve carried for the last 2 years.  While it doesn’t have all the bells and whistles of the Tilt, it is running Windows Mobile, and so I can keep my mobile helpers on it as well.

Blackberry 8700g

My Blackberry that I’ve carried for many years is great for all the reasons that BBerries are great, and horrible for several other reasons.  First and foremost, the thing refuses to power of nearly any scheme other than a true Blackberry charger.  In my opinion, if I can’t charge you off of a USB connection without special software installed, I’m not interested and that was the deal breaker for this otherwise great phone.

Sprint Wireless Card

I carry a Sprint Wireless Card as my primary mobile connectivity option (Option 2) in my backpack.  I have nothing but gushing things to say about Sprint’s network.  As you can see from above, I commonly have had a chance to work with every network except Verizon for data, and Sprint is far and away the king.  This wonderful ExpressCard slot item just works, and that is what I want from my technology.

Wireless Router

So about now someone has to be asking … seriously?  You carry a router?  Heck yes!  My Linksys Compact Wireless Router has solved a ton of problems for me in the past, and takes up so little space I often forget I’ve got it until I get desperate.  When I land at a hotel room for a conference, I’ll pay for internet for my room and then share it out via this little baby.  It has also saved the day when visiting clients without wireless capability (NOTE : Ask about this folks, often they don’t have it for good reason) or when I’m on personal trips to family.  It has also saved the day at the first We Are Microsoft event when we overloaded the wireless capabilities of the sites router.  I plugged in and took Team Sogeti and several other teams onto this unit to clear up some spots.  Get one, you won’t regret it.

Network Cable

Eons ago, when .NET was young and I was trying to learn it, I attended a ton of MSDN events.  Inevitably you got a little bag of swag, and in one of those bags I got this little gem, a self-winding network cable.  It lives in my backpack, comes out when the client doesn’t have wireless and has saved the day many times.