MockingBird v1 is now formally released. With the squeeze in available time for personal projects, it’s taken me nearly 6 months since the beta and only recently did the site see any check-ins. There’s quite a few updates, and enhancements to the suite. Actually the beta was pretty stable with hardly any bugs that needed […]
[Source: http://geekswithblogs.net/EltonStoneman]
Following on from the sample for processing Excel uploads with nServiceBus, I have a comparable version using BizTalk on MSDN Code Gallery here: BizTalk and nServiceBus Excel Upload. The BizTalk (2006 R2) sample uses a FILE receive port with a simple pipeline component to disassemble the Excel file into separate messages. A SQL send port subscribes to the messages, and with an outbound map calls the AdventureWorks stored procedure to insert each product.
Processing Excel in BizTalk is nothing new, but I wanted to do a comparison against the nServiceBus example, and also see how BizTalk and nServiceBus could be integrated. My original thinking was that the trigger in the nServiceBus solution relies on a FileSystemWatcher, which is less reliable and less flexible than BizTalk’s FILE adapter. A hybrid solution could use the FILE adapter to receive and parse the upload, then send each row as an AddProduct message to MSMQ, which the nServiceBus handler subscribes to.
Running this on the same environment (a Windows Server 2003 VM running under VirtualBox in Ubuntu 9.04), the BizTalk solution processes the 3,500 row Excel file in 3 mins 10 seconds (compared to 4m 15s for the distributed NSB running with 5 threads), and the 12,000 row file in 11m 14s (compared to 14m 0s). I was surprised to find the BizTalk solution running more quickly, as the original NSB was using non-recoverable messaging, so all messages were in memory, while BizTalk had the latency of writing to the message box.
Performance Comparison
This is not an intended to be a thorough benchmark of NSB and BizTalk – both sample projects are basic, untuned implementations, and the tests are on a single box rather than two or three. But for comparison, I ran the 3,500 row upload repeatedly under different configurations:
1. NSB with distributor – host and distributor using 20 threads
2. NSB with distributor – host and distributor using 20 threads, recoverable messaging
3. NSB without distributor – host using 20 threads
4. NSB without distributor – host using 20 threads, recoverable messaging
5. BizTalk FILE receive and SQL send
6. NSB parsing Excel file, BizTalk subscribing to AddProduct messages – MSMQ receive using batches of 100
7. BizTalk parsing Excel file, NSB without distributor subscribing to AddProduct messages – host using 20 threads
Which gave these averaged results:
Boosting the number of threads dramatically improved the NSB performance, and removing the distributor halved the duration. I would expect that running the distributor on a separate node would yield similarly good results on each processing node. The hybrid BizTalk/NSB configurations were the slowest – to be expected, as you have the latency of the message box and the latency of MSMQ saving to disk. NSB configurations without recoverable messaging were only marginally slower than the recoverable version, which seems incorrect based on Udi Dahan’s performance benchmark, so I’ll need to look into that further.
There’s plenty of scope for improving performance in both solutions. BizTalk can be endlessly tuned (the 2004 guidelines still apply as a good starting point). By modifying the NSB solution to send the AddProduct messages in one opration and using 80 threads for the handler, duration fell to 25 seconds – 140 messages per second.
Integrating BizTalk and nServiceBus
Despite having the worst performance, integrating BizTalk and nServiceBus is a viable option which may be very useful in some cases. It would allow you to leverage BizTalk’s adapter suite and mapping functionality to join LOB systems into an nServiceBus estate. Integration is actually very straightforward. By default NSB uses MSMQ, so to publish messages from BizTalk to an NSB subscriber just means configuring an MSMQ send port with the expected queue, and mapping your message to NSB format – which is an envelope containing one or messages:
<?xml version=“1.0“ ?>
<Messages xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance“ xmlns:xsd=“http://www.w3.org/2001/XMLSchema“ xmlns=“http://tempuri.net/ExcelUpload.Messages“>
<AddProduct>
<BatchId>7a80fed4-9d30-44fe-95b0-08218c5f328e</BatchId>
<RegistrationIndex>58</RegistrationIndex>
<RegistrationsInBatch>254</RegistrationsInBatch>
<BatchSourcePath>E:\ExcelUpload\1.0.0.0\Binaries\Drops\ProductUpload.xls</BatchSourcePath>
<OriginatorDestination>ExcelUpload.Client.InputQueue@WIN2003R2-VM</OriginatorDestination>
<Name>new product 58</Name>
<ProductNumber>xlu-np-58</ProductNumber>
<SafetyStockLevel>100</SafetyStockLevel>
<ReorderPoint>20</ReorderPoint>
<StandardCost>10</StandardCost>
<ListPrice>15.5</ListPrice>
<DaysToManufacture>60</DaysToManufacture>
<SellStartDate>2005-02-27T00:00:00.0000000</SellStartDate>
</AddProduct>
</Messages>
The MSMQ adapter lets you specify Recoverable and Transactional flags, so your BizTalk-generated messages have the same durability options. When an NSB handler is listening at the queue, it processes the BizTalk messages in the same way as NSB messages.
Publishing NSB messages to BizTalk is trickier, as the Send and Publish methods from NSB check to see if there are any subscribers before they write to the queue (compare this to the Notify method in RhinoServiceBus). If a BizTalk MSMQ receive port is the only handler, no subscribers will be registered with NSB and the message won’t be published. So you need to either call Send with a named queue (Bus.Send<AddProduct>(“ExcelUpload.AddProductService.1.InputQueue”, m =>…), or send a subscription message from BizTalk to register a subscriber. In the the second option you’d need to specify the message type in the subscription:
<?xml version=”1.0″ ?>
<string>ExcelUpload.Messages.StartBatchUpload, ExcelUpload.Messages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null</string>
Running the Sample
(If you haven’t seen the original NSB sample, have a read as it details the pre-requisites).
To run the BizTalk solution, download ExcelUpload.BizTalk.Binaries.zip, unzip it and:
%u00b7 Copy ExcelUpload.PipelineComponents.dll to your pipeline components directory – e.g. C:\Program Files\Microsoft BizTalk Server 2006\Pipeline Components
%u00b7 If you haven’t done so from the previous sample, run uspInsertProduct.CREATE.sql
%u00b7 Run uspInsertProduct.CREATESYNONYM.sql
%u00b7 Import the MSI into BizTalk
%u00b7 Run the MSI
%u00b7 Start send port SendInsertProduct.SQL (you’ll need to change the connection details)
%u00b7 Enable receive location ReceiveStartUpload.FILE.XLS (monitors c:\drops\ExcelUpload)
%u00b7 Drop an Excel file
As before, you’ll need to clear down the database between runs if you drop the same file repeatedly.
If you want to run the hybrid solutions, you should use the binaries in the new download rather than the original, as these run without a distributor. Run start.cmd and there are three windows – Client (the file watcher), Host (the StartBatchUpload handler) and AddProductService (this is the AddProduct handler).
To run configuration 6 – NSB parsing the file and BizTalk handling the AddProduct messages:
%u00b7 Run start.cmd
%u00b7 Kill the AddProductService console
%u00b7 Disable receive location ReceiveStartUpload.FILE.XLS
%u00b7 Enable receive location ReceiveAddProduct.MSMQ
%u00b7 Unenlist send port SendAddProduct.MSMQ
%u00b7 Start send port SendInsertProduct.SQL
%u00b7 Drop an Excel file
To run configuration 7 – BizTalk parsing the file and NSB handling the AddProduct messages:
%u00b7 Run start.cmd
%u00b7 Kill the Host and Client consoles
%u00b7 Enable receive location ReceiveStartUpload.FILE.XLS
%u00b7 Disable receive location ReceiveAddProduct.MSMQ
%u00b7 Start send port SendAddProduct.MSMQ
%u00b7 Unenlist send port SendInsertProduct.SQL
%u00b7 Drop an Excel file
Source Code
The source and referenced assemblies are in ExcelUpload.BizTalk.Source.zip. There is a VS 2008 solution for the nServiceBus components, and a VS 2005 solution for the BizTalk components.
Paolo Salvatori and I delivered a session titled ‘Customizing and Extending the BizTalk WCF Adapters’ and Paolo has created a number of fantastic blog entries around this subject. His latest entry goes through much of the content that was delivered in the session and he has included a number of links at the bottom for more WCF Adapter information and samples.
This week at PDC, the product team announced BizTalk Server 2009 R2, and previewed some capabilities of a version beyond that. This marks a continuation of the rhythm of releases every 18 – 24 months that we’ve seen over the past decade.
Note that everything I say here is based on this first announcement, as is subject to change as the product moves closer to release.
Areas of enhancement that were discussed were:
Platform alignment.
This is to be expected. BizTalk is built on the Microsoft stack, and the stack is continually evolving. As such, platform synchronization needs to happen in order to align with the latest release of the Visual Studio, Windows Server, SQL Server and the .NET framework itself.
Productivity Improvements
BizTalk Server, just like most software products, ships with a “average configuration” that will work for “most customers” in “most scenarios”. If you’ve been working with BizTalk at an infrastructure level, you know that there are a lot of knobs and dials you can adjust to optimize performance for your particular requirements. However, knowing which dial to turn requires knowledge and experience. The dashboard will embody that knowledge and experience, raising the configuration experience to a higher level and allowing you to more easily choose a group of settings. PowerShell access to management tasks means you will be able to use PowerShell to perform standard administrative tasks, in addition to WMI and the other existing options. The new management pack for Systems Center Operations Manager will extend the capabilities and visibility we already have with the existing management pack. Although it was not specifically called out, I believe this is also related to the PowerShell announcement, as you can invoke PowerShell commandlets from the Management Pack.
B2B Scenarios made easy
Perhaps the most attention-getting part of the presentation is the demonstration of the new mapper prototype. As any BizTalker knows, the mapper is a visual tool that allows you to connect elements on the left with element on the right by dragging a line between them. For many BizTalkers, they deal with simple schemas, and the user experience is just fine. However, when working with large schemas, such as those common in the world of EDI, you can end up with something that I’ve heard people call “black maps”, where you have so many linking lines that the middle of the mapper becomes a solid black mass. In the current version, we have the ability to move links onto up to 20 pages, and that helps, but sometimes that’s not enough.
This is actually a very hard problem to solve, and the team has done a great job. Here are some screen shots.
For comparison, here is the mapper as we know it today:
The map above shows all the links, which results in a lot of clutter in the middle that does not matter given the elements we see on the left and right. The screen shot below shows how this clutter is still shown, but it’s “faded”. Only links to the elements currently in view are in solid black. Not shown in the screen shot is the “auto scroll” feature, clicking on a link will cause the schemas to scroll so that the relevant parts of the schemas come into view.
One the innovations I thought was a great idea was the “relevance tree”. In this mode, you only see elements in the schema that are mapped, the non-relevant parts of the schema are collapsed, as is shown in the following screen shot:
Other features include:
So, lots of work has been done to improve the productivity of people working on large complex maps, and I’m impressed at how significant an impact some of these seemingly simply changes will have.
There is more to talk about from the announcement, but those are the key points as they relate to BizTalk Server 2009 R2. As I said at the outset, this is the first announcement, so everything is subject to change as we progress through the development cycle. This announcement shows Microsoft’s continued commitment to the BizTalk platform, and BizTalk’s role as a key component of the overall data platform strategy.
You can watch the roadmap presentation online at http://microsoftpdc.com/Sessions/SVR15. This blog post only covers the BizTalk Server 2009 R21 portion of that presentation, I’ll talk about the other things show in a separate blog post.
At PDC this week we had a booth to showcase the interoperability that WCF gives you through its support for the WS-* protocols. In particular, we highlighted an Apache incubator project called Stonehenge.
At Stonehenge we work together with a growing list of WS-* vendors (WSO2, Sun, SpringSource, and soon Progress) to build samples and prove out interoperability between the stacks.
At the PDC booth we had a demo of the soon to be released ’M2’ version of the StockTrader Sample application. Check out the Channel 9 video here.
We also talked about Stonehenge at the ApachecCon conference in Oakland a couple of weeks ago where Apache celebrated their 10 year anniversary. If you are in the mood for some geek entertainment you might want to see this from the Lightening Talks.
If you are doing cross-platform interop with WCF, I would encourage you to get involved in Stonehenge. You can contribute by suggesting important test scenarios and/or helping build and test one of the samples. To get started go to the project wiki: Apache Stonehenge and subscribe to the dev email list.
To learn about other interop activities at PDC check out the Interoperability Team Blog.
There is a great article in MSDN Magazine this month by Michele Bustamante on using WIF to implement claims-based security in WCF Services: http://msdn.microsoft.com/en-us/magazine/ee335707.aspx.
It’s gets to the right level of detail to make it crystal clear. Spend all weekend reading the docs, or spend an hour with Michele’s article? Hmmm easy choice!
Tuesday 11/17/09 was a very exciting day for the BizTalk Server product team as well as for BizTalk developers attending the PDC 2009 conference in LA.
We have announced the release of BizTalk Server 2009 R2 in 2010 and we have laid out the roadmap for BizTalk Server future. (You can watch the PDC session here.)
The roadmap includes details around the following future releases:
Here are the highlights of each version:
BizTalk Server 2009 R2:
Latest Application Platform Support for:
BizTalk Server 2009 R2 will also provide new tools to increase the productivity of developers and IT professionals in particularly but not exclusively for B2B scenarios here are some examples:
Business to Business Scenarios
Productivity Enhancements
At PDC we have demonstrated an early prototype of the new BizTalk Mapper hosted in Visual Studio 2010 (beta) .
BizTalk Server vNext (beyond 2009 R2)
This release will focus on the following main themes:
At PDC we presented a demo that highlighted the first themes: Platform Alignment & Enterprise Connectivity.
This demo using a preview of an early prototype for WF (.NET 4.0) integration within BizTalk included the following scenarios:
Expect more details, screenshots and videos of the demos to be available soon
Hi guys, I’ve been busy over the recent months always working and planning new ways
to train, deliver and impart knowledge to *you* in a variety of different ways.
Truth be told our Region with respect to SharePoint Adoption and Knowledge, has been
ahead of the curve worlwide – that you guys “SharePoint-ies”.
With the release of SharePoint 2010 soon-ish, we’ve been busy – to bring you the best
value and the most effective training for your time.
For the release of SharePoint 2007 some numbers:
The big congrats goes to you! For being driven, demanding better,
demanding more detail and better knowledge, you demand that the bar is raised and
meet and excel that challenge.
Breeze has partnered with Microsoft & Excom, allowing us to each focus on our
strengths, delivering to you the best possible offering in this space.
I’m pretty excited to announce our SharePoint 2010 Series running in Melb
+ Sydney initially (+ others soon)
(running at Microsoft Offices)
A Special Blog Reader offer: Would you like me to Webcast
these sessions for those whom can’t make it?
Add a comment on this Post to let me know – I’ll get back to you with details.
(we are giving away goodies and offers but I
can’t tell you too much)
Here’s the formal blurb: —— snip ——
Microsoft Australia, is pleased to invite you to a FREE Seminar on SharePoint
2010.
Melbourne – December 2, 2009
Sydney – December 11, 2009
Delivered by Breeze & EXCOM Education, this seminar is the first
in a series that are designed to get you across the new functionality of SharePoint
2010 and importantly how your business can benefit.
SharePoint 2010 is the business collaboration platform that enables you to connect
and empower people through formal and informal business communities, within the enterprise
and beyond, and to manage content throughout the information lifecycle.
This next release of SharePoint has some exciting new enhancements for IT Professionals
and Developers we think you will want to see for yourself!
This seminar is for all SharePoint enthusiasts from IT Professionals, Developers to
Business Decision Makers and Information Workers. Bring along your questions to the
unveiling of SharePoint 2010!
Seats are limited. Register NOW !!!!
For more information and to register, click HERE
This week Microsoft have announced details about BizTalk 2009 R2 at the Professional Developers Conference. Since BizTalk 2009 was released this year in around May this is a very tight timeframe for the next release. What is exciting about it from my perspective is that Microsoft have released BizTalk 2009 to align with the latest […]