My wish list for next version of BizTalk

The 2009 version of BizTalk Server is soon to be released, but we have been asked to give feedback to the product team on what features we’d like to see in future releases. My plan was to focus on smaller things that I think would make a big difference (all though I let my self go a bit further at the end).

If you think I’ve missed something, please let me know and I’ll make sure to pass it forward.

Deployment

  • In the 2009 version, as the BizTalk projects became C# projects we were given a better support for MSBuild and TFS Build. The possibility to build to and on a server that has not got Visual Studio installed is really great from a deployment perspective. However, since we have not been given any tooling for the actual deployment/undeployment, it requires lots of work to make it useful. I'd really like MS to either create a supported Deployment API or make the Microsoft.BizTalk.Deployment library more "public" by documenting it.
  • The bindingsfiles are one of the key components for automated deployment. First I'd like for you to make use of the Application property in the BindingInfo class. The problem is today that when you create your custom tasks for deployment/undeployment you would need to store the information about the Application and its references somewhere else since it's not to be found in the bindingsfiles.
  • Bindingsfiles are all together a very difficult thing to manage in different environments. A tool for this where we could edit and manage our bindingsfiles, would increase our productivity immensely.

Low Latency

  • There has been a lot of talk about  this. I don’t really think of it as a big issue anymore as Dublin is on its way. However, one small thing that would REALLY make a difference would be to move the polling interval setting from the group to the host.

Pipelines

  • In 2006 we were given the possibility for configuring pipelines at runtime. This was a really good thing, where we don’t need to have hundreds of almost identical pipelines. Although I'd really have like this feature to be implemented with a “real” property box that supported UI editors.
  • To be honest, I don't really like the notion of pipelines at all, and would rather see them as default settings of components, which I could select and use as-is or alter at runtime by adding or removing components.

BAM

  • Use table partitioning rather than the views and tables we have today. This way we could make better use of indexes and boost the performance. This feature where implemented in SQL Server in 2005catch up!
  • BAM administration in the admin console
  • Simple BAM reports in the admin console

EDI

  • Better schema support
  • Message Archiving

Adapter improvements

FILE Adapter

  • Supports regular expressions
  • Support for FTPS

SMTP Adapter

  • Remove the mail address from the URI, so that we can make better use for the adapter in a messaging scenario.

Misc

  • Enforcing a Tracking host
  • Failed message routing for orchestrations
  • Code windows for functoids and the expression should be made sizeable
  • Project templates for pipeline components and Functoids.
  • Improved visualization for mappings

Future features

  • Simulation and Orchestration debugging from VS
  • Self service portal. Almost 90% of all all issues that comes to our support is about -"Where is my f***** message?". Providing a BAM driven portal where these rude "customers" can find this information themselves, would greatly boost the productivity of our dev team.

Email is BACK

I must have changed my password to my email server a LONG time ago, because it has been ages since I have been getting emails from my blog.

I finally took time to look into this and have reconfigured it to send me emails again.

If there are any unanswered emails out there, please let me know and I will answer them.

Sorry, as I do really want to hear from you!

Walkthrough: Composite Operations with the new WCF-based SQL Adapter

The new SQL Adapter in the WCF Adapter Pack 2.0 supports composite operations, that is performing multiple SQL operations in response to a single input message. The purpose of this blog post is to provide a walkthrough of how this works. I am using the public beta of BizTalk Server 2009, the public beta of the WCF Adapter Pack 2.0 and released version of SQL Server 2008. See below for an important caveat, as well as a link at the end of the post to my test solution.

To start, as with most things in BizTalk’s contract-first world, we need a schema. In order to do this, choose “add generated items” from a BizTalk project in Visual Studio. Then, choose “Consume Adapter Service”. If you don’t see that Visual Studio template, then you haven’t installed the Adapter Pack, as that’s where it comes from.

Next, select the sqlBinding specify a server, and press “Configure”. Set the client credential type to Windows (assume appropriate SQL login rights), and then on the URI tab, specified the server and database to use:

Af6ter doing this, press “Connect”, and the metadata will be populated.

For the purpose of this walkthrough, I have created 2 stored procedures: the first one inserts a record into a table, the second returns all rows in that table. Those are shown in the UI below.

Note that in the category we have “Procedures” and “Strongly-Typed Procedures”. The distinction is that “Procedures” will create un-typed schemas, whereas “Strongly-Typed Procedures” will generate schemas that you can work with inside BizTalk for mapping, promoting properties, etc.

The “Filename Prefix” will be used as a prefix for all the generated schemas.

After that was configured, I clicked OK and all the schemas were generated for me.

Next step is that you need to create a composite schema that will define the message you send to the adapter. I’m not quite sure why this one wasn’t generated for me, it’d be nice (hint hint), but it’s trivial to do.

How I did this for the walkthrough:

  • create a new schema
  • rename the root to SQLMsg (or whatever you like, this is unimportant)
  • add a sibling record called SQLMsgResponse (this name does matter, it is the name of the request, with “Response” appended)
  • add two child records under SQLMsg, and another two under SQLMsgResponse (names don’t matter, they’ll get renamed below)
  • right-click the topmost “<schema>” node, and in the “Imports” property, add the “CompositeTypedProcedure.dbo.xsd” schema
  • in the first child under SQLMsg, set the “Data Structure Type” property to InsertIntoDestination (this is a reference that you just imported above)
  • in the second child under SQLMsg, set the “Data Structure Type” property to SelectAllDestination
  • in the first child under SQLMsgResponse, set the “Data Structure Type” property to InsertIntoDestination (this is a reference that you just imported above)
  • in the second child under SQLMsgResponse, set the “Data Structure Type” property to SelectAllDestinationResponse

Your schema should now look like this:

The, create an instance of the new composite schema to use as a test message, and populate the request. Here’s mine:

I then created a simple orchestration that would receive a request, call the adapter, and persist the response from the adapter. The request and response messages are of the type we just created in the composite schema:

Build and deploy the solution. After deploying it, note that there was a binding file generated along with the schemas, which is awesome, as this means you don’t need to manually create the send port. So, import the binding file which exists in you Visual Studio project.

HOWEVER pretty big caveat here. after importing the binding, you need to change the action mapping. If you use the default value, it will fail. You need to replace what is generated with the magic keyword “CompositeOperation”. This tells the adapter that it needs to call multiple operations, which it will resolve based on the schemas and namespaces. I believe the reason this works the way it does is that it allows you to import multiple operations in a single pass, and then use some subset of those operations in a composite operation, thereby enabling re-use of the generated schemas to potentially cover multiple different combinations of composite operations. Either way, watch out for this one. The error message tells you exactly what the problem is, however it won’t tell you about the keyword.

As an aside, and for the benefit of those who have not worked with this adapter yet, here are the binding configuration properties you have access to:

Then:

  • create an inbound file drop location
  • create an outbound file drop folder
  • bind everything
  • start the application
  • drop your instance doc into the file drop location, triggering the orchestration

Lastly, here’s the output file:

In closing, I think this is an awesome new capability, and I am really liking the new SQL adapter. In case you haven’t heard, the old SQL adapter is being deprecated, so you really should be working with this one going forward.

You can download my test solution here.

Technorati Tags: BizTalk,SQL Adapter,BizTalk Server,BizTalk Adapter Pack

Breaking the WSS Top Link Bar in two with jQuery

Since I got a fan of the jQuery Javascript library, usually I can’t resist showing off the power of this library in my SharePoint development courses at U2U. For example two weeks ago I was in sunny Cyprus talking about SharePoint, ASP.NET AJAX and jQuery and I told my students something in the line of “with jQuery you can change pretty much anything in the SharePoint UI by just making use of Javascript”. Promptly one of my students asked me if I could show how to break the Top Link Bar of a SharePoint site into two parts. I really like challenges in my SharePoint courses, but I couldn’t conquer this challenge on the spot, it took me a couple of hours in my hotel room to get this to work. Now that I’ve got some time to polish the code a little bit, I want to share the code with you! 🙂

First, let’s talk about the issue that this little jQuery script is going to solve: you probably know if you create subsites in SharePoint sites, those subsites are shown (by default) in the Top Link Bar of the parent site. So the more subsite you’ve got the more items this menu is showing. Life is all good until there are too many items to show in the menu so it’s get too big to fit on the screen. You won’t get an error or something like that of course, but the browser will give the menu the space it needs by adding horizontal scrollbar the page. The screenshot below illustrates this behavior: the top link bar has too many menu items making it pretty hard to access for example the Site Actions menu (you need to scroll to the right).

So how can this be solved with the help of jQuery? Well besides a very powerful DOM Selectors API, the jQuery library also has a DOM Maniplation API. This Manipulation API can change the HTML that’s rendered in the browser, by adding elements, removing elements etc. The idea is to write a Javascript function that adds a second Top Link Bar to the page’s DOM. The easiest way to accomplish this is to just copy the existing Top Link Bar entirely:

$("#zz1_TopNavigationMenu").clone(true).insertAfter($("#zz1_TopNavigationMenu")).attr("id", "zz1_TopNavigationMenuCopy");

This jQuery script will:

  1. $("#zz1_TopNavigationMenu")
    select the element with ID zz1_TopNavigationMenu
  2. .clone(true)
    clone that element
  3. .insertAfter($("#zz1_TopNavigationMenu"))
    insert the cloned element after the original menu
  4. .attr("id", "zz1_TopNavigationMenuCopy");
    set the ID attribute to a new value to be able to identify it uniquely

The result of this function is a page that shows two identical menu bars:

Now the only thing that’s left to do is to remove the unnecessary menu items from both menus:
var nrOfItems = ($("#zz1_TopNavigationMenu > tbody > tr > td").length + 1) / 3;
var splitIndex = (Math.round(nrOfItems / 2) – 1) * 3;
$("#zz1_TopNavigationMenu > tbody > tr > td:gt(" + splitIndex + ")").remove();
$("#zz1_TopNavigationMenuCopy > tbody > tr > td:lt(" + (splitIndex + 1) + ")").remove();

The first two lines calculate the number of items in the menu (each menu item consists of three td elements) and the index of the td element where the table should be “split”. The third line removes all the td elements of the original menu with and index higher than the calculated one. The last line removes all td elements of the copied menu with and index lower than the calculated one.

Now, let’s put everything together and get this to work in a real SharePoint site! The first thing to do is to make sure your SharePoint site loads the jQuery library. This can be done in a couple of ways, for example using the jQuery component of the SmartTools for SharePoint project on CodePlex (read my previous blog posts on jQuery in SharePoint for more info). Secondly the Javascript discussed above should be loaded, and once again there are a couple of ways to do this. For production scenarios I’d recommend to build a Feature using  a delegate control that loads the Javascript function (just like the SmartTools jQuery component does), but for testing you can also do this in a plain Content Editor Web Part (or, God forbid, using the SharePoint Designer tool). So just add a Content Editor Web Part to a page of your SharePoint site, and copy/past the following piece of code in it:

<script>
$(document).ready(function() {
  // Calculate where to split the tables of the menus
  var nrOfItems = ($("#zz1_TopNavigationMenu > tbody > tr > td").length + 1) / 3;
  var splitIndex = (Math.round(nrOfItems / 2) – 1) * 3;
  // Make a copy of the TopNavigationMenu
  $("#zz1_TopNavigationMenu").clone(true).insertAfter($("#zz1_TopNavigationMenu")).attr("id", "zz1_TopNavigationMenuCopy");
  // Remove items from original menu
  $("#zz1_TopNavigationMenu > tbody > tr > td:gt(" + splitIndex + ")").remove();
  // Remove items from copied menu
  $("#zz1_TopNavigationMenuCopy > tbody > tr > td:lt(" + (splitIndex + 1) + ")").remove();
});
</script>

When done, the SharePoint page will now display a Top Link Bar, split into two. Make sure you test the code in a WSS site (a Team Site for example), because Publishing sites (e.g. a Collaboration Portal has another menu, see the remark at the end of this post).

Important remark: the code discussed in this article is just an example for demonstration purposes, if you plan to use this code I strongly recommend to test it in various web browsers and check if it meets your performance goals. The identifiers of the Top Link Bar in the code are uses in WSS sites (e.g. a Team Site), SharePoint Publishing sites generate menus with other ID’s and or structures, so you have to tweak the Javascript. For example: the following code works for a Collaboration Portal using the default.master.

$(document).ready(function() {
  // Calculate where to split the tables of the menus
  var nrOfItems = ($("#zz1_TopNavigationMenu *.zz1_TopNavigationMenu_5 > tbody > tr > td").length) / 3;
  var splitIndex = (Math.round(nrOfItems / 2) – 1) * 3;
  // Make a copy of the TopNavigationMenu
  $("#zz1_TopNavigationMenu").clone(true).insertAfter($("#zz1_TopNavigationMenu")).attr("id", "zz1_TopNavigationMenuCopy");
  // Remove items from original menu
  $("#zz1_TopNavigationMenu *.zz1_TopNavigationMenu_5 > tbody > tr > td:gt(" + splitIndex + ")").remove();
  // Remove items from copied menu
  $("#zz1_TopNavigationMenuCopy > tbody > tr > td:lt(2)").remove();

  // Remove first item (current site) from copied men
  $("#zz1_TopNavigationMenuCopy *.zz1_TopNavigationMenu_5 > tbody > tr > td:lt(" + (splitIndex + 1) + ")").remove();
});

Technorati Tags: sharerpoint,wss,moss,navigation,menu,jquery,javascript,ajax

.NET Services – Cloud Interoperability

Speaking of standards – I’m thrilled to report that we will release the “M5” (Milestone 5) CTP (Community Technology Preview – think Beta) for .NET Services (part of the Azure Services Platform) tomorrow!  For those who aren’t familiar with this effort, here’s the primer Almost two years ago, we introduced these services – Service Bus (secure messaging across networks and firewalls), Access Control (user access to web apps and services across multiple standards-based identity providers), and Workflow (for orchestrating and routing Service Bus messages).  From the beginning, .NET Services was designed for multi-cloud, multi-platform use. Developers can use the .NET Services in conjunction with ANY programming language (using support for industry-standard protocols, or via available SDKs for .NET, Java and Ruby) on ANY platform to create or extend federated applications. A good overview of .NET Services is available here.


This milestone contains enhancements to all of the services including expanded support for standards like REST, ATOM, SOAP and HTTP.  As I mentioned previously, we demonstrated at MIX cloud to cloud interop in action.  Specifically, we showed how the Access Control Service and Service Bus could be integrated with a Python Application deployed into Google App Engine using just two lines of code. As always, feedback from developers is critical to us. So, please take time to sign up for the CTP, and tell us what you think. We’re on our way to commercial availability later this year and we need your help to get there.


If you haven’t already, you can follow our cloud efforts by adding @Azure on Twitter.

Practical Oslo Examples Starting to Emerge

In recent days there’s been some interesting movement in Oslo-land:

  • Doug Purdy and Chris Sells did a presentation at Mix09 that showed a domain-specific language for invoking RESTful services, and how to use M to create RESTful services
  • BizTalk MVP Yossi Dahan has published (Part 1, Part 2, Part 3) details about his domain-specific approach to BizTalk deployments
  • Kris Horrocks has started blogging, and posted his domain-specific approach to interacting with X10 home automation

What these three things have in common is that this is the start of practical examples of how you can author a DSL using M, and how the DSL could be used to simplify the effort required to do things (ie: reduce the amount of code you need to write). Up until now it’s been interesting to watch, but there hasn’t been much in the way of practical examples. The fact that these are starting to emerge speaks to the increasing maturity of the project.

The path to learning this stuff starts with M, then you move on to creating DSLs using MGrammar, and then the final phase is creating a runtime that is actually doing something with the data produced by your DSL. This is where Doug and Chris’s presentation (I highly recommend you click the link above and watch it) really resonates well. The “Murl” effort they’ve been working on (available here) makes it really easy to create a REST client.

The sample also shows how extensible Intellipad is, by adding a new “mode” that enables actually USING the DSL (parsing *and* executing) from inside Intellipad and calling RESTful services. Very cool! The code you see below is the syntax of the DSL used to call REST services.

But, in my opinion, the most exciting part of their presentation was the MService part, which is a (work-in-progress, not released) way to create a RESTful service, using a DSL written in M. The screen shot below shows the service definition in the left-hand pane, and the generated SQL in the right-hand pane (red arrows indicates some of the relationships between the two).

They saved the file out to c:\inetpub\wwwroot, and made a REST request (from Intellipad, using Murl-mode, just because they could :)). There’s a handler that IIS routes the request for a “*.m” resource request to. It’s an entirely self-bootstrapped operation, if the service sees no storage it will create the storage.

M is all about writing down data, and the ultimate output is structured data in the form of MGraph. Where this becomes tangible and payoff is realized is when we use a runtime to do interesting things with that data, and MService is the best example of this that I’ve seen thus-far. It’s a tantalizing view of how developers will create applications in the future.

Technorati Tags: Oslo,M,DSL

Moving beyond the “Manifesto”

As you might expect, several of us spent most of Thursday and Friday last week in conversations with developers, standards body members and other vendors regarding open standards for cloud computing and how we get there collaboratively. Being in this industry for so many years, I remember a time when new technologies and platforms did not produce much interest in standards and interoperability. It was great this time around to see broad support for openness in the cloud and transparency on the approach to interoperability.  I was also happy to see a number of community-driven efforts spin up last week, which will provide enormously valuable feedback in defining the desired end-state. It’s important for everyone to take a step back and remember this isn’t about vendors; it’s about developers and end-users.


As I indicated on Wednesday night, Microsoft welcomes the opportunity for open dialogue on cloud standards. To that end, we have accepted an invitation to meet on Monday at 4pm in New York at the Cloud Computing Expo with other vendors and members of standards bodies.  From our perspective, this represents a fresh start on the conversation – a collaborative “do-over” if you will.