BizTalk and AppFabric Caching: Problem or Problem Solver?

In the recent edition of The BizTalker Newsletter I featured the below article.  Read the article below or join the newsletter to have it delivered to your email.  I welcome and comments. feedback, or thoughts on this subject.


Server AppFabric contains two core pieces – Hosting and Caching. Today, I’m going to focus on the Caching piece of AppFabric.
AppFabric Caching is a new offering from Microsoft as part of the Server AppFabric collection of technologies for the Application Server role inside Windows. AppFabric Caching is Microsoft’s Distributed Cache solution for on-premise, fast, in-memory, and durable object caching.

A lot of community bloggers have been blogging about the use of AppFabric Caching with BizTalk Server. A few of the best posts are:

  • Using Windows Server AppFabric Caching for Storing SSO & Configuration data by Abdul Rafay – This shows how to Cache SSO data inside BizTalk.
  • Use AppFabric Cache to cache your WCF Service response by Mikael Hakansson – This shows how to Cache WCF response messages inside BizTalk.
  • Using AppFabric Cache in BizTalk by Todd Dhaeyer – This shows how to use AppFabric to Cache reference data and even prevent double messages.

Is Server AppFabric Caching really solving problems or creating new ones?

Rather than diving too deep into AppFabric Caching, let me summarize with a list of Do’s and Don’ts.

AppFabric Rules of Thumb – Don’ts:

  • Never put anything in the Cache that MUST not be lost – it is a non-transactional cache, not a database.

  • Never install AppFabric Caching along with other server products.  It will take up all of the available resources of the server including the entire available RAM so it should only be installed on a dedicated server.

  • Avoid making a connection to the Cache Cluster with each request.  The initial connection to the Cache Cluster is more expensive than reading and writing to the Cache. 

AppFabric Rules of Thumb – Do’s:

  • Use the Cache for fast, in-memory access to serializable .Net Objects that may or may not actually be in the Cache.

  • To achieve fault tolerances in the Cache, simply add a second node to the Cache Cluster (in most cases).

  • Use the Cache to reduce the load, performance, and potentially the license costs of more expensive products like SQL Server.

  • As demand grows, add more servers to the Cache Cluster to increase the available RAM for Caching.

The guidelines seem simple enough, so what is my hold out on BizTalk and AppFabric Cache?

Using BizTalk and AppFabric together makes awesome demos and can simplify some problems like working with reference data.  The current release of AppFabric Caching is designed to run alone on its own server since it could take up all available RAM.  This would not be a good situation on a production BizTalk Server.

My main concern with AppFabric Caching is if you have a separate caching layer along with your BizTalk Server, why not just use SQL rather than the Cache? 

  • Lower Cost – AppFabric Cache is included with the Windows License but you need extra hardware to run it on its own.

  • Better Performance – AppFabric Caching would reduce the load on SQL but I would like to see better data on SQL connection vs. Cache connection costs.  Although, if the existing SQL Cluster was already reaching capacity, moving more items away from that would be a plus. 

  • Lower Latency – In my mind, reducing latency is the biggest factor for using a Cache.  Cache would reduce the latency vs. a call to the database.  I would say in most BizTalk Solutions an external call to a SQL Server would probably not be the bottle neck given other BizTalk tuning parameters.  

  • Mission Critical Data – SQL wins out here (at least for now) because it has a true transactional backend data store. 

I love all three of the AppFabric community demos shown above.  At this stage, I would be comfortable using AppFabric Cache for reference and SSO data – not so much for removing double messages unless I was OK with maybe missing one now and then (but then what’s the point?).  I would still look to leverage SQL for double message removal given the need for true transactions.  Like everything – your unique situation may vary.

In conclusion, BizTalk and AppFabric Caching can work together.  Will caching work in scenarios such as a message re-play and de-duper?  Absolutely!  But for me, given the criticality of most BizTalk installations, I would use caution on using AppFabric Caching with BizTalk at least for now.

Server AppFabric for the BizTalk Developer Video Session

If you consider yourself a hard core BizTalk Developer, then this session is for you!

A few weeks ago I presented a session on Server AppFabric to the BizTalk User Group in Sweden.  This session was geared to showing how similar Workflow 4.0 and AppFabric are to BizTalk – conceptually at least.  The goal of this session was to show how us BizTalk guys and gals can quickly pick up Workflow 4 and AppFabric because we already understand the concepts.

I would highly recommend this 60 minute session to all fellow BizTalkers out there!

This session is available on Channel 9 – http://channel9.msdn.com/Blogs/MSCOMSWE/Tech-Overview-WCFWF-Server-AppFabric-BizTalk-Conference-Stockholm

The code and slides are available for download at https://www.biztalkgurus.com/media/p/29973.aspx.

At the same conference, I gave a session which covered Windows Azure Platform AppFabric.  This 30 minute session covers a real-world service bus solution and a walk through of the code behind the solution.  It is available at http://channel9.msdn.com/Blogs/MSCOMSWE/Pattern-5–Remote-Message-Broadcast-BizTalk-Conference-Stockholm and the code and slides can be downloaded at https://www.biztalkgurus.com/media/p/29975.aspx.

 

If you are looking for other sessions from the multi-day conference in Sweden, below is a full list.  We covered a wide-range of technologies from AppFabric to StreamInsight.

Day 1 (Sessions from September 8th, 2010)

Welcome and Introduction

Choosing The Right Tool in the Application Platform
Discuss the challenge of choosing the right technology for a given situation and present a decision framework for guiding evaluation.

Tech Overview: SQL Server
Look at the core components of SQL Server that are used to build applications (e.g. SSIS) and when to use them.

Tech Overview: BizTalk Server
Discuss what BizTalk is and when to use it.

Tech Overview: WCF/WF, Server AppFabric
Highlight key capabilities in WCF and WF and benefits offered by Windows Server AppFabric.

Tech Overview: Windows Azure Platform
Discuss Microsoft’s cloud offering and best usage scenarios.

Pattern #1 – Simple Workflow
Evaluate scenario that involves aggregating data from multiple sources and presenting a unified response.

Day 2 (Sessions from September 9th, 2010)

Pattern #2 – Content Based Routing
Consider options for effectively transmitting data to multiple systems that perform similar functions.

Pattern #3 – Human Workflow with Repair and Resubmit
Showcase using workflow 3.5 to send customer details to an AppFabric hosted workflow 4.0 Workflow Service.  This workflow service controls the payment collection process and allows for updated information on a user to be sent back into the same running workflow instance from SharePoint.

Pattern #4 – Cross Organization Supply Chain
Evaluate how to build a supply chain to integrate systems in a PO scenario.

Pattern #5 – Remote Message Broadcast
Demonstrates a scenario where a traditional polling solution is augmented to support real-time updates.

Pattern #6 – Complex Event Processing
Addresses click stream analysis and creating actionable events from user and system behavior.

Book Sample Code, Our Commercial, and Repair & Resubmit Solution Video

I have posted online all the sample code from my book, “Applied Architecture Patterns on the Microsoft Platform”.  You can download it at https://www.biztalkgurus.com/media/p/30260.aspx.  This code contains various samples using BizTalk, BizTalk ESB Toolkit, Server AppFabric, SharePoint 2010, .Net 4.0 Workflow, SQL, Stream Insight, Azure, SQL Azure, Platform AppFabric Service Bus, Server AppFabric Caching, and more.  Of course, you will get the most benefit of this code using it alongside with the chapter of the book that walks through it. 

The book is available now on Amazon.com and PacktPub.com. 

Thanks to our friends in Sweden we now have a Commercial for our book!  I have not seen many commercials for Technology books, but this video gives some context of the “why” behind our book.  You get to hear from Richard, Ewan, and myself about why we wrote the book and why we picked to release the book at the European BizTalk Conference in Stockholm, Sweden to a room full of top notch, hard-core BizTalk people.

This video is available at https://www.biztalkgurus.com/media/p/30247.aspx and on YouTube. 

 

If you are looking for more detailed information on the types of solutions covered in the book, I have put together a video walking through Chapter 10 – Repair and Resubmit with Human Workflow.  This solution uses SharePoint 2010 to store customer information.  This information is processed by a .Net 3.5 workflow that sends the data to a Server AppFabric hosted .Net 4.0 workflow service for processing. 

This video is available at https://www.biztalkgurus.com/media/p/30249.aspx and on YouTube. 

European BizTalk Conference AppFabric and Workflow 4 Session Slides and Code Downloads

In early September 2010, I got the honor to present three sessions on AppFabric and Workflow 4.0 to the BizTalk Users Group in Sweden at the European BizTalk Conference. While my sessions were not related to BizTalk, they are designed to show how easy it is for someone with BizTalk skills to learn AppFabric and Workflow 4. 

I have posted the session slide decks and sample code (including the code generated from my 100% LIVE demo). 

I did three sessions:

  • Server AppFabric and Workflow 4.0 Overview – This includes some slides on AppFabric strengths and weakness compared to BizTalk and shows how simple it is to set up Correlation in Workflow 4.

 

  • Server AppFabric with Human Workflow – This uses SharePoint 2010 and Workflow 3.5 with AppFabric and .Net 4 Workflow (Chapter 10 from my book) to allow data to be corrected and resubmitted to the same workflow process.

 

  • Platform AppFabric with Remote Message Broadcasting – This talks about using Azure Platform AppFabric to broadcast messages through the Service Bus to destinations on numerous networks (Chapter 11 from my book).

The video recording of the sessions should be available soon and I’ll post the links once I have them.