Technology Updates from PDC 2009 and BizTalk 2009-R2

by community-syndication | Nov 23, 2009 | BizTalk Community Blogs via Syndication

So PDC is over and boy did we get a lot of announcements with regards to our Connected System’s Technologies. We learned that the upcoming “Dublin” application server for WCF and WF components will formally be named Windows AppFabric. We also learned that Azure.NET Server has been renamed to Azure AppFabric. We also learned about the roadmap for BizTalk. You can expect BizTalk 2009-R2 will be arriving sometime at the end of next year. It will deliver enhancements in the follow areas:

• Platform Alignment – We will be supporting the latest servers and tools
• VS 2010
• Windows Server 2008-R2
• SQL Server 2008-R2
• Productivity Improvements
• A new dashboard to apply and manage performance parameters (today, tuning a BizTalk server is tricky to do as you have to be aware of a number of registry keys, database based parameters etc)
• PowerShell based capabilities for management tasks
• Updated SCOM object model to better reflect BizTalk artifacts
• B2B
• New Mapper enhancements to make it easier to work with and understand complex maps
• An FTPS adapter will be provided right out of the box.
• RFID
• Support for event filtering right out of the box

During the session, the team demo’d the new Mapper enhancements and got a great reaction from the crowd. For any of you who have ever struggled to keep your sanity when working with a large map, you’ll likely love these simple enhancements.

Now, at the PDC session (which you can view online http://microsoftpdc.com/Sessions/SVR15. ) they did more than just speak about 2009-R2. They also demoed some very-very early bits from v.NEXT+1 (i.e. the one after 2009-R2).  There are a number of key features that will be shipping in that version, the first and foremost being Windows Workflow (WF) integration into BizTalk.  With this integration, you will be able to choose whether or not to use classic BizTalk orchestrations or WF based workflows.  By integrating these two technologies, you can now do things like use the BizTalk mapper from within WF.  I encourage you to check out the recording of the session to see the full roadmap and product demos.

Cheers and keep on BizTalking

Peter

LIDNUG: Online chat with me Monday Nov 23rd

by community-syndication | Nov 23, 2009 | BizTalk Community Blogs via Syndication

Removing an InfoPath template part from the controls section

by community-syndication | Nov 22, 2009 | BizTalk Community Blogs via Syndication

Hi all

The other day I was given the task of updating an InfoPath template part that was
in use on a laptop, because a newer version of this template part was available.

Upon opening the InfoPath client, I saw this:

There were two template parts, and in this case they are named “TemplateGroup1” and
“TemplateGroup2”. I had a new version for the “TemplateGroup1” template part. I clicked
on “Add or Remove Custom Controls” and got this screen:

As you can see, the “TemplateGroup1” does not show up, which I thought was weird.
So, I tried clicking on “Add” to just add the new version of the “TemplateGroup1”,
but that gave me this error:

The custom control, <TemplateGroup1> (urn:schemas-microsoft-com:office:infopath:templategroup1:-myxsd-2009-11-22t19-43-32),
is already installed. Remove the existing custom control, and then try installing
again.

So, I was at a loss I couldn’t remove the existing version, and I couldn’t upload
a new version.

Finally, I discovered what had happened. The user has an entry in the registry like
this:

The key “IPCustomControlsFolder” is placed in the “HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\InfoPath\Designer”
path of the registry. All template parts you put into this folder are automatically
added to the controls of InfoPath.

So I found that path, and deleted the “TemplateGroup1” template part, and everything
was fine.

So, what I now know is, that there are two ways of adding new template parts to be
used by InfoPath:

1. Add them manually inside the InfoPath client (or the toolbox in VS.NET)
2. Add the right registry key to the registry, so you have a repository of template parts.
   This is especially useful for a repository on a shared network folder that can be
   used for all employees.

So if you ever have trouble removing a template parts from the custom controls section,
look for the registry key.

—

eliasen

My recap of the PDC 2009 preview of the BizTalk V.Next prototype

by community-syndication | Nov 22, 2009 | BizTalk Community Blogs via Syndication

Prior to joining Microsoft, I can’t tell you how many times I heard “will there be a convergence between BizTalk orchestrations and Windows Workflow Foundation? When do I use what? Are BizTalk Orchestrations going away?”. Since joining Microsoft, I’ve heard it even more. In fact, it has got to be the number one question I hear, and I’m sure that’s also true for many of you.

Last week at PDC 2009, Sriram did two things, he announced BizTalk Server 2009 (as I blogged about here), but also, he demonstrated some early prototype work that was being done around bringing together BizTalk, Windows Workflow Foundation, and Windows Server AppFabric. That announcement of R2 and the sneak preview of V.Next are significant enough that I decided to do a separate blog post about each.

Before we start though, let’s set expectations. What was shown in terms of WF/BizTalk integration was a very early prototype. This is NOT something you’ll be able to download next week, and put into production next month. This is early-stage code showing the direction the team is heading in. Have said that, if you follow my blog, you know that’s exactly the stage I like to get involved at :), and I feel it’s very important to keep an eye on the horizon and know what direction you’re heading in.

There are actually a few aspects to what was shown.

First and foremost, was WF integration into BizTalk. Let’s consider some of the motivations behind this:

• a common ask from customers is a low-latency solution (not all integrations require it, but in some cases, it can be a must-have). BizTalk is a reliable messaging platform, once a message is received, it is never, ever, lost. That reliability has a cost, as it means that there has to be durable storage (in our case SQL Server), and no matter how well tuned, persisting a message in durable storage has a latency cost associated with it
• sometimes people aren’t sure which way to go as there are four ways to compose services on the Microsoft stack (usually capabilities lead to a decision, but there can be overlap), with no easy migration path between them:
  1. using BizTalk orchestration
  2. using Windows Workflow Foundation
  3. using BizTalk ESB itineraries
  4. using custom code

Of those four, custom code is an option, but it puts a lot of responsibility on the developer to do “plumbing”. If or example you compose 3 services, and service #2 is unavailable, what do you do? You need to code protectively to respond to that. Or, service #2 returns a fault, what does that mean in terms of the overall composition and flow? Personally, I think a more efficient way to go about this is to let something else worry about the minutiae of the plumbing, which is what we get with the integration server capabilities of BizTalk Server.  So, although there will always be times where it is the right way to go, let’s leave custom code off the list for the rest of this post.

That leaves us with three options, two of which are  BizTalk options, and then WF which is currently completely outside BizTalk. As you probably know, with WF 3.5, if you wanted to host a workflow, you had two choices: use the hosting capabilities in SharePoint, or write your own host (which can be a daunting task depending on far you go towards being a fully featured enterprise-grade host). With .NET 4, that changes with the announcement of Windows Server AppFabric (which I blogged about here). What Sriram showed was a preview of what it make look like if WF were integrated into BizTalk.

From a BizTalker and SOA perspective, I really like the idea of having WF integrated into BizTalk, as it makes it centralizes the service composition mechanisms, and to some degree, helps mitigate the “when to use what” question as that can be reduced to a set of “common” guidelines. However, what he showed goes well beyond just using BizTalk projects as a shell from which to get at the WF designer, the integration goes much deeper than that, even in this early prototype. Bringing WF into BizTalk doesn’t change BizTalk, it extends it.

Here’s look at this from a Visual Studio perspective, notice the XAMLX files in a BizTalk project. Notice also the workflow designer. This is the same workflow designer that people will be used to working with Visual Studio 2010.

Bringing WF into the BizTalk environment brings benefits to the WF developers too. One of those is mapping, the screen shot below shows how you can have either a data contract or a schema on both source and destination sides, and regardless of which you pick, the experience is the same. That also means you get all of the new BizTalk Server 2009 R2 mapper goodness that I blogged about here. From there, as Sriram shows at about the 45 minute mark of his presentation, it’s a one-click operation to generate a workflow custom activity to invoke that map, and you can then work with in the WF designer by just dragging it from the palette onto the design surface.

From a deployment perspective, there was a crude prototype shown, this is obviously not what it would look like in final release. The thing to notice here is the fact that the workflow is being hosted in the BizTalkServerApplication host. So, if you follow that logic a step further (rampant speculation on my part!), then that means you could use the standard BizTalk host instance model to have certain workflows running in certain hosts, and then you have instances of those hosts on selected machines.

There’s a lot of cool stuff in this video, and it points to an interesting convergence of technologies in a future release of BizTalk. It also raises a bunch of questions for me, and adds to a long wish list (which I’ve communicated to the team :)). The bottom line though is that WF integration will give BizTalk developers the ability to handle low latency scenarios in a standard framework supported way, and it also add value for WF developers, as it provides them with the possibility of migrating their workflows into BizTalk and gaining additional capabilities as they do so.

What about Windows Server AppFabric and Windows Azure AppFabric?

What remains to be seen is how this fits into AppFabric, something the team is exploring now. An obvious one to me (rampant speculation!) is that AppFabric is just another host for the workflow (as it could be if the workflow were not part of a BizTalk project), be it Windows Server AppFabric or Windows Azure AppFabric, however perhaps getting there through the BizTalk deployment model and taking things like transformation activities with it. Even today, with BizTalk Server 2009, we have integration from BizTalk into Windows Server AppFabric (and also in the other direction) by going through WCF. So, the integration exists, and will only get deeper in future releases.

Finally.

The last point I want to make is something that may or may not be clear from these two BizTalk announcements. The announcements show Microsoft’s firm commitment to the 10,000+ current BizTalk customers, as well as to the product itself. The capabilities planned for BizTalk Server 2009 R2 and the capabilities shown last week for the release beyond that show the level of commitment and investment that Microsoft is making in the platform. As an insider who’s been involved with BizTalk since before the first release, I can tell you that the degree of enthusiasm and excitement in the product team is very, very real. They have great vision, and the resources they need to execute on that vision. The future keeps getting brighter, this has been a fun ride, and it looks like we have even more fun ahead.

The video of all of this is at http://microsoftpdc.com/Sessions/SVR15. The things I’ve talked about in this post are in the second half of the video.

Interview Webcasts from PDC09

by community-syndication | Nov 22, 2009 | BizTalk Community Blogs via Syndication

Video, Slides and Code Available from A Practical Exploration of the BizTalk BRE ACSUG Meeting

by community-syndication | Nov 21, 2009 | BizTalk Community Blogs via Syndication

If you are interested in the BizTalk Business Rules Engine here’s one for you: we have recorded the Auckland Connected Systems User Group meeting from last Thursday by Jeffrey Chilberto on ’A Practical Exploration of the BizTalk Business Rules Engine’. It’s a very hands on presentation that goes over basic and advanced topics – exposing […]

MockingBird v1 RTM

by community-syndication | Nov 21, 2009 | BizTalk Community Blogs via Syndication

MockingBird v1 is now formally released. With the squeeze in available time for personal projects, it’s taken me nearly 6 months since the beta and only recently did the site see any check-ins. There’s quite a few updates, and enhancements to the suite. Actually the beta was pretty stable with hardly any bugs that needed […]

Processing Excel uploads with BizTalk and nServiceBus

by community-syndication | Nov 21, 2009 | BizTalk Community Blogs via Syndication

[Source: http://geekswithblogs.net/EltonStoneman]

Following on from the sample for processing Excel uploads with nServiceBus, I have a comparable version using BizTalk on MSDN Code Gallery here: BizTalk and nServiceBus Excel Upload. The BizTalk (2006 R2) sample uses a FILE receive port with a simple pipeline component to disassemble the Excel file into separate messages. A SQL send port subscribes to the messages, and with an outbound map calls the AdventureWorks stored procedure to insert each product.

Processing Excel in BizTalk is nothing new, but I wanted to do a comparison against the nServiceBus example, and also see how BizTalk and nServiceBus could be integrated. My original thinking was that the trigger in the nServiceBus solution relies on a FileSystemWatcher, which is less reliable and less flexible than BizTalk’s FILE adapter. A hybrid solution could use the FILE adapter to receive and parse the upload, then send each row as an AddProduct message to MSMQ, which the nServiceBus handler subscribes to.

Running this on the same environment (a Windows Server 2003 VM running under VirtualBox in Ubuntu 9.04), the BizTalk solution processes the 3,500 row Excel file in 3 mins 10 seconds (compared to 4m 15s for the distributed NSB running with 5 threads), and the 12,000 row file in 11m 14s (compared to 14m 0s). I was surprised to find the BizTalk solution running more quickly, as the original NSB was using non-recoverable messaging, so all messages were in memory, while BizTalk had the latency of writing to the message box.

Performance Comparison

This is not an intended to be a thorough benchmark of NSB and BizTalk – both sample projects are basic, untuned implementations, and the tests are on a single box rather than two or three. But for comparison, I ran the 3,500 row upload repeatedly under different configurations:

1. NSB with distributor – host and distributor using 20 threads

2. NSB with distributor – host and distributor using 20 threads, recoverable messaging

3. NSB without distributor – host using 20 threads

4. NSB without distributor – host using 20 threads, recoverable messaging

5. BizTalk FILE receive and SQL send

6. NSB parsing Excel file, BizTalk subscribing to AddProduct messages – MSMQ receive using batches of 100

7. BizTalk parsing Excel file, NSB without distributor subscribing to AddProduct messages – host using 20 threads

Which gave these averaged results:

Boosting the number of threads dramatically improved the NSB performance, and removing the distributor halved the duration. I would expect that running the distributor on a separate node would yield similarly good results on each processing node. The hybrid BizTalk/NSB configurations were the slowest – to be expected, as you have the latency of the message box and the latency of MSMQ saving to disk. NSB configurations without recoverable messaging were only marginally slower than the recoverable version, which seems incorrect based on Udi Dahan’s performance benchmark, so I’ll need to look into that further.

There’s plenty of scope for improving performance in both solutions. BizTalk can be endlessly tuned (the 2004 guidelines still apply as a good starting point). By modifying the NSB solution to send the AddProduct messages in one opration and using 80 threads for the handler, duration fell to 25 seconds – 140 messages per second.

Integrating BizTalk and nServiceBus

Despite having the worst performance, integrating BizTalk and nServiceBus is a viable option which may be very useful in some cases. It would allow you to leverage BizTalk’s adapter suite and mapping functionality to join LOB systems into an nServiceBus estate. Integration is actually very straightforward. By default NSB uses MSMQ, so to publish messages from BizTalk to an NSB subscriber just means configuring an MSMQ send port with the expected queue, and mapping your message to NSB format – which is an envelope containing one or messages:

<?xml version=“1.0“ ?>

<Messages xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance“ xmlns:xsd=“http://www.w3.org/2001/XMLSchema“ xmlns=“http://tempuri.net/ExcelUpload.Messages“>

<AddProduct>

<BatchId>7a80fed4-9d30-44fe-95b0-08218c5f328e</BatchId>

<RegistrationIndex>58</RegistrationIndex>

<RegistrationsInBatch>254</RegistrationsInBatch>

<BatchSourcePath>E:\ExcelUpload\1.0.0.0\Binaries\Drops\ProductUpload.xls</BatchSourcePath>

<OriginatorDestination>ExcelUpload.Client.InputQueue@WIN2003R2-VM</OriginatorDestination>

<Name>new product 58</Name>

<ProductNumber>xlu-np-58</ProductNumber>

<SafetyStockLevel>100</SafetyStockLevel>

<ReorderPoint>20</ReorderPoint>

<StandardCost>10</StandardCost>

<ListPrice>15.5</ListPrice>

<DaysToManufacture>60</DaysToManufacture>

<SellStartDate>2005-02-27T00:00:00.0000000</SellStartDate>

</AddProduct>

</Messages>

The MSMQ adapter lets you specify Recoverable and Transactional flags, so your BizTalk-generated messages have the same durability options. When an NSB handler is listening at the queue, it processes the BizTalk messages in the same way as NSB messages.

Publishing NSB messages to BizTalk is trickier, as the Send and Publish methods from NSB check to see if there are any subscribers before they write to the queue (compare this to the Notify method in RhinoServiceBus). If a BizTalk MSMQ receive port is the only handler, no subscribers will be registered with NSB and the message won’t be published. So you need to either call Send with a named queue (Bus.Send<AddProduct>(“ExcelUpload.AddProductService.1.InputQueue”, m =>…), or send a subscription message from BizTalk to register a subscriber. In the the second option you’d need to specify the message type in the subscription:

<?xml version=”1.0″ ?>

<string>ExcelUpload.Messages.StartBatchUpload, ExcelUpload.Messages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null</string>

Running the Sample

(If you haven’t seen the original NSB sample, have a read as it details the pre-requisites).

To run the BizTalk solution, download ExcelUpload.BizTalk.Binaries.zip, unzip it and:

%u00b7 Copy ExcelUpload.PipelineComponents.dll to your pipeline components directory – e.g. C:\Program Files\Microsoft BizTalk Server 2006\Pipeline Components

%u00b7 If you haven’t done so from the previous sample, run uspInsertProduct.CREATE.sql

%u00b7 Run uspInsertProduct.CREATESYNONYM.sql

%u00b7 Import the MSI into BizTalk

%u00b7 Run the MSI

%u00b7 Start send port SendInsertProduct.SQL (you’ll need to change the connection details)

%u00b7 Enable receive location ReceiveStartUpload.FILE.XLS (monitors c:\drops\ExcelUpload)

%u00b7 Drop an Excel file

As before, you’ll need to clear down the database between runs if you drop the same file repeatedly.

If you want to run the hybrid solutions, you should use the binaries in the new download rather than the original, as these run without a distributor. Run start.cmd and there are three windows – Client (the file watcher), Host (the StartBatchUpload handler) and AddProductService (this is the AddProduct handler).

To run configuration 6 – NSB parsing the file and BizTalk handling the AddProduct messages:

%u00b7 Run start.cmd

%u00b7 Kill the AddProductService console

%u00b7 Disable receive location ReceiveStartUpload.FILE.XLS

%u00b7 Enable receive location ReceiveAddProduct.MSMQ

%u00b7 Unenlist send port SendAddProduct.MSMQ

%u00b7 Start send port SendInsertProduct.SQL

%u00b7 Drop an Excel file

To run configuration 7 – BizTalk parsing the file and NSB handling the AddProduct messages:

%u00b7 Run start.cmd

%u00b7 Kill the Host and Client consoles

%u00b7 Enable receive location ReceiveStartUpload.FILE.XLS

%u00b7 Disable receive location ReceiveAddProduct.MSMQ

%u00b7 Start send port SendAddProduct.MSMQ

%u00b7 Unenlist send port SendInsertProduct.SQL

%u00b7 Drop an Excel file

Source Code

The source and referenced assemblies are in ExcelUpload.BizTalk.Source.zip. There is a VS 2008 solution for the nServiceBus components, and a VS 2005 solution for the BizTalk components.

Follow Up from TechEd Europe

by community-syndication | Nov 20, 2009 | BizTalk Community Blogs via Syndication

Paolo Salvatori and I delivered a session titled ‘Customizing and Extending the BizTalk WCF Adapters’ and Paolo has created a number of fantastic blog entries around this subject.  His latest entry goes through much of the content that was delivered in the session and he has included a number of links at the bottom for more WCF Adapter information and samples.

BizTalk Server 2009 R2 Announced

by community-syndication | Nov 20, 2009 | BizTalk Community Blogs via Syndication

This week at PDC, the product team announced BizTalk Server 2009 R2, and previewed some capabilities of a version beyond that. This marks a continuation of the rhythm of releases every 18 – 24 months that we’ve seen over the past decade.

Note that everything I say here is based on this first announcement, as is subject to  change as the product moves closer to release.

Areas of enhancement that were discussed were:

• Platform Alignment – support for the following products and technologies:
• Visual Studio 2010
• Windows Server 2008 R2
• SQL Server 2008 R2
• Productivity Improvements
• Single dashboard to apply and manage performance parameters
• Out-of-the-box support for Event Filtering and Delivery (RFID)
• PowerShell access to management tasks
• New System Center Operation Management SCOM Pack object model to better reflect BizTalk artifacts
• B2B Scenarios Made Easy
• Mapper enhancements to make complex mapping easier to create and maintain
• FTPS to provide secure transactions between businesses
• Updated B2B accelerators for latest protocol versions

Platform alignment.

This is to be expected. BizTalk is built on the Microsoft stack, and the stack is continually evolving. As such, platform synchronization needs to happen in order to align with the latest release of the Visual Studio, Windows Server, SQL Server and the .NET framework itself.

Productivity Improvements

BizTalk Server, just like most software products, ships with a “average configuration” that will work for “most customers” in “most scenarios”. If you’ve been working with BizTalk at an infrastructure level, you know that there are a lot of knobs and dials you can adjust to optimize performance for your particular requirements. However, knowing which dial to turn requires knowledge and experience. The dashboard will embody that knowledge and experience, raising the configuration experience to a higher level and allowing you to more easily choose a group of settings. PowerShell access to management tasks means you will be able to use PowerShell to perform standard administrative tasks, in addition to WMI and the other existing options. The new management pack for Systems Center Operations Manager will extend the capabilities and visibility we already have with the existing management pack. Although it was not specifically called out, I believe this is also related to the PowerShell announcement, as you can invoke PowerShell commandlets from the Management Pack.

B2B Scenarios made easy

Perhaps the most attention-getting part of the presentation is the demonstration of the new mapper prototype. As any BizTalker knows, the mapper is a visual tool that allows you to connect elements on the left with element on the right by dragging a line between them. For many BizTalkers, they deal with simple schemas, and the user experience is just fine. However, when working with large schemas, such as those common in the world of EDI, you can end up with something that I’ve heard people call “black maps”, where you have so many linking lines that the middle of the mapper becomes a solid black mass. In the current version, we have the ability to move links onto up to 20 pages, and that helps, but sometimes that’s not enough.

This is actually a very hard problem to solve, and the team has done a great job. Here are some screen shots.

For comparison, here is the mapper as we know it today:

The map above shows all the links, which results in a lot of clutter in the middle that does not matter given the elements we see on the left and right. The screen shot below shows how this clutter is still shown, but it’s “faded”. Only links to the elements currently in view are in solid black. Not shown in the screen shot is the “auto scroll” feature, clicking on a link will cause the schemas to scroll so that the relevant parts of the schemas come into view.

One the innovations I thought was a great idea was the “relevance tree”. In this mode, you only see elements in the schema that are mapped, the non-relevant parts of the schema are collapsed, as is shown in the following screen shot:

Other features include:

• the “search” box that you see on top of the mapper canvas. Typing something in the search box will find and highlight it whether it’s a functoid name or element name. In addition, if there’s a match that’s on a page that’s not currently shown, then that tab gets highlighted to give you a visual cue that lets you know the non-visible page also has at least one match
• you can move links between pages by a right-click context menu operation

So, lots of work has been done to improve the productivity of people working on large complex maps, and I’m impressed at how significant an impact some of these seemingly simply changes will have.

There is more to talk about from the announcement, but those are the key points as they relate to BizTalk Server 2009 R2. As I said at the outset, this is the first announcement, so everything is subject to change as we progress through the development cycle. This announcement shows Microsoft’s continued commitment to the BizTalk platform, and BizTalk’s role as a key component of the overall data platform strategy.

You can watch the roadmap presentation online at http://microsoftpdc.com/Sessions/SVR15. This blog post only covers the BizTalk Server 2009 R21 portion of that presentation, I’ll talk about the other things show in a separate blog post.