BizTalk Adapters: Buy, Build, or CodePlex

In the recent edition of The BizTalker Newsletter I featured the below article about BizTalk Adapters.  Read the article below or join the newsletter to have it delivered directly to your email.  I welcome any comments. feedback, or thoughts on this subject.

BizTalk contains many built-in adapters for communication with external systems.  When those built-in adapters do not meet the needs of your project, some hard decisions need to be made such as whether to buy an adapter, build an adapter using the framework, or use an adapter from CodePlex (i.e. open source).  This article will take a look at the pros and cons of each approach.

Buy an Adapter: While this option would require an outlay of cash up front, it can be advantageous in terms of support and speed of implementation. 

Pros:

  • Support is provided by the adapter vendor which can be rapid and 24×7.

  • Adapter vendor provides a simple, legal licensing model.

  • Purchased adapters are usually buy now, install tomorrow so there is no lag time to get up and running.

Cons:

  • Purchased adapters have an upfront cost to the project.

  • You may not be able to customize the adapter if it does not meet your needs.

Build an Adapter: While this can seem like the obvious answer for a software or consulting company, usually this is a large undertaking.  I talked with the writers of the famous SFTP Adapter on CodePlex.  They said it took 300 to 400 hours of effort to build that adapter.  Out of this time, it took about 200 hours just to learn the adapter framework and the rest to implement the protocol-specific logic.  Even at a conservative rate of $100 per hour, this is between $30,000 and $40,000.

Pros:

  • There could potentially be additional services revenue for the company and they may maintain Intellectual Property (IP) rights.

  • You have the complete flexibility to build what you want and how you want it.

Cons:

  • The company that builds the adapter will have to do the maintenance and support.

  • The building company may not be a subject matter expert in either the BizTalk Adapters Framework or the implementation protocol.

  • It could take a significant amount of time to complete, test, and deploy.

Use a CodePlex Adapter: CodePlex is a large open source software community.  It hosts many different BizTalk Adapters.  Some of the most popular are the SFTP Adapter and the Scheduled Task Adapter.  Sometimes 3rd party support is available, but in general support is through community forums.

Pros:

  • Adapter is provided at no cost.

  • You have access to the source code to customize as needed.

Cons:

  • There is no support for things that do not work as expected.

  • There are gray areas in terms of licensing and the impact of customizations.

  • When a new release is available, customizations need to be reapplied.

  • Company legal policies might restrict or prohibit the use of open source code.

In my experience we have both bought an adapter and used CodePlex – both times with the SFTP Adapter.  I have never built a custom adapter.
Lets take a deeper look at the decisions that impacted my situations.

Case 1 – Bought /n software SFTP Adapter.  This was for a large Fortune 500 client.  It was simple; the service contract with the client did not allow the use of any open source software due to license and support concerns.  I am unsure of the cost of the SFTP Adapter, but using it was simple and we had no issues.

Case 2 – Used CodePlex SFTP Adapter.  This was also for a large Fortune 500 client.  While the adapter was free, it took about 40 hours over 3 weeks for legal to review the license and approve the use of the adapter.  Also, we ran into a few situations that required customization of the adapter.  This resulted in an additional 80 hours of development and testing effort.  Total costs to us for the free adapter was at least 120 hours and this consulting firm charges much more than $100 per hour. 

In conclusion, it is important to remember the trade-offs with each adapter choice.  Remember that free does not always mean free and that even the best custom adapter could end up resulting in months (if not years) of support calls down the road.

BizTalk and AppFabric Caching: Problem or Problem Solver?

In the recent edition of The BizTalker Newsletter I featured the below article.  Read the article below or join the newsletter to have it delivered to your email.  I welcome and comments. feedback, or thoughts on this subject.


Server AppFabric contains two core pieces – Hosting and Caching. Today, I’m going to focus on the Caching piece of AppFabric.
AppFabric Caching is a new offering from Microsoft as part of the Server AppFabric collection of technologies for the Application Server role inside Windows. AppFabric Caching is Microsoft’s Distributed Cache solution for on-premise, fast, in-memory, and durable object caching.

A lot of community bloggers have been blogging about the use of AppFabric Caching with BizTalk Server. A few of the best posts are:

  • Using Windows Server AppFabric Caching for Storing SSO & Configuration data by Abdul Rafay – This shows how to Cache SSO data inside BizTalk.
  • Use AppFabric Cache to cache your WCF Service response by Mikael Hakansson – This shows how to Cache WCF response messages inside BizTalk.
  • Using AppFabric Cache in BizTalk by Todd Dhaeyer – This shows how to use AppFabric to Cache reference data and even prevent double messages.

Is Server AppFabric Caching really solving problems or creating new ones?

Rather than diving too deep into AppFabric Caching, let me summarize with a list of Do’s and Don’ts.

AppFabric Rules of Thumb – Don’ts:

  • Never put anything in the Cache that MUST not be lost – it is a non-transactional cache, not a database.

  • Never install AppFabric Caching along with other server products.  It will take up all of the available resources of the server including the entire available RAM so it should only be installed on a dedicated server.

  • Avoid making a connection to the Cache Cluster with each request.  The initial connection to the Cache Cluster is more expensive than reading and writing to the Cache. 

AppFabric Rules of Thumb – Do’s:

  • Use the Cache for fast, in-memory access to serializable .Net Objects that may or may not actually be in the Cache.

  • To achieve fault tolerances in the Cache, simply add a second node to the Cache Cluster (in most cases).

  • Use the Cache to reduce the load, performance, and potentially the license costs of more expensive products like SQL Server.

  • As demand grows, add more servers to the Cache Cluster to increase the available RAM for Caching.

The guidelines seem simple enough, so what is my hold out on BizTalk and AppFabric Cache?

Using BizTalk and AppFabric together makes awesome demos and can simplify some problems like working with reference data.  The current release of AppFabric Caching is designed to run alone on its own server since it could take up all available RAM.  This would not be a good situation on a production BizTalk Server.

My main concern with AppFabric Caching is if you have a separate caching layer along with your BizTalk Server, why not just use SQL rather than the Cache? 

  • Lower Cost – AppFabric Cache is included with the Windows License but you need extra hardware to run it on its own.

  • Better Performance – AppFabric Caching would reduce the load on SQL but I would like to see better data on SQL connection vs. Cache connection costs.  Although, if the existing SQL Cluster was already reaching capacity, moving more items away from that would be a plus. 

  • Lower Latency – In my mind, reducing latency is the biggest factor for using a Cache.  Cache would reduce the latency vs. a call to the database.  I would say in most BizTalk Solutions an external call to a SQL Server would probably not be the bottle neck given other BizTalk tuning parameters.  

  • Mission Critical Data – SQL wins out here (at least for now) because it has a true transactional backend data store. 

I love all three of the AppFabric community demos shown above.  At this stage, I would be comfortable using AppFabric Cache for reference and SSO data – not so much for removing double messages unless I was OK with maybe missing one now and then (but then what’s the point?).  I would still look to leverage SQL for double message removal given the need for true transactions.  Like everything – your unique situation may vary.

In conclusion, BizTalk and AppFabric Caching can work together.  Will caching work in scenarios such as a message re-play and de-duper?  Absolutely!  But for me, given the criticality of most BizTalk installations, I would use caution on using AppFabric Caching with BizTalk at least for now.

Join The BizTalker-The Newsletter of BizTalkGurus.com

It has been over a year since my list issue of The BizTalker.  With all the excitement in the air about BizTalk in the cloud and AppFabric, I figured now is a good time to bring back the newsletter.  I am planning to send out a new issue monthly with the next one in just a few days.  I am reaching out to various 3rd parties to try to get some freebies to give away for future newsletters. 

In the next issue you will have a chance to win a copy of my new book plus an exclusive discount to order it online. 

If you are not a member of The BizTalker, sign up now at: http://www.biztalkgurus.com/newsletter/index.aspx

The BizTalker Volume 12 – Sending to a WCF Receive Adapter

Last week I released volume 12 of The BizTalker.  This issues’ feature article was written by Dwight Goins of QuickLearn and talks about working with the WCF Adapter in BizTalk 2006 R2. 

Also covered is the new BizTalkBlogs.com and the new training available on BizTalk-Training.com

You can view the newsletter online or to make sure you don’t miss out simply subscribe!