ESB Toolkit 2.0 Ships (YEAH!!!), I’m presenting it tonight at the San Diego .NET user group Connected Systems SIG

I’m REALLY happy to say the the ESB Toolkit 2.0 has been released. As part of the current launch tour I’m doing, I’m telling people that in my opinion the best part of BizTalk Server 2009 is the ESB Toolkit. If you’re a BizTalk dev, you really want it check it out. And, if you’re one of those unfortunates that had installation issues, it’s OK to come back now and try again. I installed a build a couple of weeks ago, took me about 15 minutes total. The team invested a LOT of effort in this area, and it shows.

By sheer coincidence (OK, so I knew rough timeframe, but not to the day :)), I’m doing an overview presentation of it, and showing some really cool dynamic messaging demos, at a user group meeting in San Diego tonight. Meeting will be at 6:30 (pizza at 6:00) at the Microsoft office in La Jolla.

The official team blog announcement is available here.

You can get the bits here.

I’ll be posting more info about this within the next few days, and, if you’re in San Diego tonight, come on by!!

BizTalk Server: ESB Toolkit 2.0 Released. UDDI issues fixed?

Yesterday, Microsoft released the final version of ESB Toolkit 2.0 (the word ‘Guidance’ has been dropped) which you can download from http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=bc86cf1e-ef29-4b19-95f7-388f64555090#tm. I have had half an hour today to look at the documentation. A preliminary read seems to suggest that, between the release of the CTP and the final release, Microsoft scrapped the old UDDI resolver and have introduced a completely new version that complies with the UDDI standards and which addresses the issues raised in a blog article which I published earlier this year.See http://geekswithblogs.net/cyoung/archive/2008/11/12/126975.aspx. However, please be aware that I need time to properly digest the functionality changes, and won’t be able to do that for another week or so. I will report back when I have had more time to investigate.

BizTalk Adapter Pack Migration Tool

Migration tool released – if you want to consider using the BizTalk WCF adapters.

http://www.microsoft.com/downloads/details.aspx?FamilyID=e68b8fd1-5c8a-499a-8237-4dc23a8342c5&displaylang=en

Overview

The WCF-based adapters shipped with the BizTalk Adapter Pack 2.0 are the recommended solutions to integrate various LOB systems with BizTalk Server. The Adapter Pack consists of LOB adapters for SAP, Siebel, Oracle DB, Oracle EBS, and SQL Server. They are built on the WCF LOB Adapter SDK framework. However these are not the first adapters to be released by Microsoft for integration.

The BizTalk Adapters for Enterprise Applications is a suite of adapters based on the older BizTalk Adapter Framework and was released with BizTalk 2006 and BizTalk 2009. Among others, this suite has adapters for SAP, Siebel, and Oracle DB systems. BizTalk Server has a native SQL adapter.

With the release of BizTalk 2009, all these adapters are deprecated in favour of the WCF-based adapters in the Adapter Pack as the latter set has more features and support more LOB versions.

Announcing the Availability of the BizTalk ESB Toolkit 2.0

As we have pre-announced during several BizTalk sessions at TechEd North America last month, the BizTalk ESB Toolkit 2.0, formerly known as the ESB Guidance 2.0 – has been released today to the web. It is available on the new ESB page in the BizTalk Developer Center on MSDN.

This toolkit extends BizTalk Server 2009 capabilities to support a loosely coupled and dynamic messaging architecture. It functions as middleware that provides tools for rapid mediation between services and their consumers. Enabling maximum flexibility at run time, the BizTalk ESB Toolkit 2.0 simplifies composition of service endpoints without “hard-wiring” them and also provides management of service interactions at enterprise scale.

Why did we change the name ?

The BizTalk ESB Toolkit 2.0 has been renamed from “ESB Guidance2.0” to reflect the fact that it is now provided as a BizTalk Server 2009 value-add with a better support model. This will hopefully help customers develop mature ESB implementations that will be ready for enterprise-wide deployments.

How does this change support and community engagement?

The BizTalk ESB Toolkit 2.0 Connect site has been created to log bugs with the BizTalk ESB Toolkit Team and to provide updates to additional tools over time. Once you log a bug, someone from the BizTalk ESB Toolkit team will respond to you within five days with an acknowledgment and status. Any future updates or tools will be provided through the Microsoft Download Center.

In addition, an ESB Toolkit Forum is provided on MSDN. These forums target online communities-in particular BizTalk MVPs and other BizTalk community lists. Best-effort assistance will be offered through a closely managed forum. That is, any fixes and responses to questions in the forums are best effort. We will continue to leverage the community to provide peer assistance, although with the capability of issuing fixes if necessary.

What’s new with the ESB Toolkit 2.0?

The BizTalk ESB Toolkit 2.0 provides both architectural enhancements and new capabilities over the previous ESB Guidance 1.0. For more information, see SOA and Web Services section in the New Features in BizTalk 2009 Web page (rename will propagate soon ).

What does the BizTalk ESB Toolkit 2.0 provide?

The BizTalk ESB Toolkit 2.0 provides key building blocks that are required for implementing a comprehensive service-oriented infrastructure (SOI) including:

  • Endpoint run-time discovery and virtualization.
    The service consumer does not need to be aware of the service provider location and endpoint details; a new or modified service provider can be added to the ESB, without interruptions to the service consumer.
  • Loosely coupled service composition.
    The service provider and service consumer do not need to be aware of service interaction style.
  • Dynamic message transformation and translation.
    The mapping definition between distinct message structure and semantics is resolved at run time.
  • Dynamic routing.
    Run-time content-based, itinerary-based, or context-based message routing.
  • Centralized exception management.
    Exception management framework, services, and infrastructure elements that make it possible to create, repair, resubmit, and compensate fault messages that service consumers or BizTalk components submit.
  • Quality of service.
    An asynchronous publish/subscribe engine resolves different levels of service availability and provides high availability, scalability, and message traceability for ESB implementations.
  • Protocol transformation.
    Providing the ability for service provider and service consumer to interact via different protocols including WS-* standards for Web Services. For example, a service provider can send an HTTP Web Service request, which will result in sending a message via the BizTalk SAP adapter.
  • Extensibility.
    Providing multiple extensibility points to extend functionality for endpoint discovery, message routing, and additional BizTalk Server adapters for run time and design time.

How to get started with the BizTalk ESB Toolkit 2.0?

Using PowerShell in BizTalk Post-Processing Scripts

[Source: http://geekswithblogs.net/EltonStoneman]

Often in BizTalk deployments you need to do additional work after installation. Typically your full install process may need to:

  • Install BizTalk artifact assemblies to the GAC
  • Install application dependencies to the GAC
  • Register an application source name in the registry, for logging to the Event Log
  • Create FILE send or receive locations on the local filesystem
  • Add application store configuration settings to Enterprise Single Sign-On (SSO)
  • Add log4net configuration settings to BTSNTSvc.exe.config

You can achieve this with a single BizTalk installer by configuring resources and post-processing scripts, and exporting an MSI from the application. Various scripting languages are supported in BizTalk installations (batch files, VBScript etc.), except the most logical – PowerShell, which gives first-class support for the filesystem, the registry, XML files and .NET objects. You can still use PowerShell by including scripts as resources, and using a batch file as the post-processing script, which acts as a harness to call the PowerShell scripts.

This walkthrough addresses all the points above. The completed BizTalk application is on MSDN Code Gallery here: BizTalk PowerShell Deployment Sample – import and install the MSI to deploy with the PowerShell script, or browse the ZIP file to see the scripts and resources.

1. Install BizTalk artifact assemblies to the GAC

This is straightforward, set the resource option “Add to the global assembly cache on MSI file install” to true – this happens by default if you add a BizTalk Assembly resource in the Administration Console:

Using the command line though, this is not the default option so you need to explicitly set -Options:GacOnInstall in BTSTask:

btstask AddResource

-ApplicationName:PowerShellSample

-Type:BizTalkAssembly

-Options:GacOnInstall

-Source:PowerShellSample.Schemas.dll

-Destination:%BTAD_InstallDir%\PowerShellSample.Schemas.dll

2. Install application dependencies to the GAC

As 1), except the resource type is System.BizTalk:Assembly (in BTSTask you can omit “System.BizTalk”). The command requires the same flag to add to the GAC on install:

btstask AddResource

-ApplicationName:PowerShellSample

-Type:Assembly

-Options:GacOnInstall

-Overwrite

-Source:.\Dependencies\SSOConfig.dll

-Destination:%BTAD_InstallDir%\Dependencies\SSOConfig.dll

In this case, I’m installing the SSOConfig assembly (from SSO Config Tool) which provides static .NET classes for accessing the SSO application configuration store. The Overwrite flag is set in case the resource already exists in another application.

3. Register an application source name in the registry, for logging to the Event Log

To log to the Application event log with your own source name, you need to add a registry key with the app name, and the name of the handler:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\Application\PowerShell.Sample

In PowerShell, this is done in using the New-Item cmdlet to create the key, and New-ItemProperty to set the key value:

New-Item -Path ‘HKLM:\SYSTEM\CurrentControlSet\Services\Eventlog\Application\PowerShellSample’ -Force

New-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Services\Eventlog\Application\PowerShellSample’ -Name ‘EventMessageFile’ -PropertyType ExpandString -Value ‘C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\EventLogMessages.dll’ -Force

(HKLM: is a PowerShell drive mapped to HKEY_LOCAL_MACHINE, and the -Force flag overwrites existing values).

To execute the PowerShell script on install, we need a batch file which BizTalk can run as a post-processing script. The batch file is very simple, just separating install and uninstall logic to individual PowerShell scripts, and redirecting script output to a log file:

cd “%BTAD_InstallDir%\Deployment”

if “%BTAD_InstallMode%” == “Install” ( powershell “.\PowerShellSample.Install.ps1” >> PowerShellSample.Install.ps1.log )

if “%BTAD_InstallMode%” == “Uninstall” ( powershell “.\ PowerShellSample.Uninstall.ps1” >> PowerShellSample.Uninstall.ps1.log )

Both the CMD and PS1 files need to be added as resources to the BizTalk application. The PS1 files are of type BizTalk:File, and the CMD harness is of type BizTalk:PostProcessingScript:

btstask AddResource

-ApplicationName:PowerShellSample

-Type:File

-Source:.\Deployment\PowerShellSample.Install.ps1

-Destination:%BTAD_InstallDir%\Deployment\PowerShellSample.Install.ps1

btstask AddResource

-ApplicationName:PowerShellSample

-Type:PostProcessingScript

-Source:.\Deployment\PowerShellSample.PostProcessing.cmd

-Destination:%BTAD_InstallDir%\Deployment\PowerShellSample.PostProcessing.cmd

4. Create FILE send or receive locations on the local filesystem

If you need to create static file locations, the same New-Item cmdlet is used with the filesystem provider. Specify the full path for the directory and any intermediate directories will be created if they don’t exist. Use the -Force flag to suppress warnings if the directory already exists:

New-Item -Path ‘c:\receiveLocations\x\y\z’ -ItemType Directory -Force

Note that the resources in the BizTalk application are copies rather than references, so if you modify your PS1 files, you’ll need to update the resource (in the Administration Console, select the resource and use Modify… Refresh, or re-run the BTSTask command).

5. Add settings to Enterprise Single Sign-On (SSO)

If you’re using SSO to store group-wide application config, you can create or export an XML file of the settings using SSO Config Tool. We add the .ssoconfig file as a File resource to the application, then in the install script use PowerShell to call a .NET method to import the settings using the SSOConfig.SSOApplication class. The SSOConfig assembly is a resource which has already been deployed to the GAC by this point in the installation:

[Reflection.Assembly]::Load(‘SSOConfig, Version=1.1.0.0, Culture=neutral, PublicKeyToken=656a499478affdaf’)

$configPath = [IO.Path]::Combine($env:BTAD_InstallDir, ‘Deployment\PowerShellSample.ssoconfig’)

$app = [SSOConfig.SSOApplication]::LoadFromXml($configPath)

$app.SaveToSSO()

Note that the PowerShell script has access to all the environment variables set by BizTalk on the install – accessed by prefixing $env: to the variable name, as we do here to get the installation directory from the installer ($env:BTAD_InstallDir).

6. Add settings to BTSNTSvc.exe.config

Modifying XML is straightforward in PowerShell. We want to configure an Event Log appender in the BizTalk service config file by adding the following XML:

<configSections>

<section name=”log4net” type=”log4net.Config.Log4NetConfigurationSectionHandler, log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821″ />

</configSections>

<log4net>

<appender name=”Sixeyed.CacheAdapter.EventLogAppender” type=”log4net.Appender.EventLogAppender, log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821″>

<param name=”LogName” value=”Application”/>

<param name=”ApplicationName” value=”Sixeyed.CacheAdapter”/>

<layout type=”log4net.Layout.PatternLayout”>

<conversionPattern value=”%date [%thread] %logger %level – %message%newline” />

</layout>

</appender>

<logger name=”Sixeyed.CacheAdapter.Log”>

<level value=”WARN” />

<appender-ref ref=”Sixeyed.CacheAdapter.EventLogAppender” />

</logger>

</log4net>

The Get-ItemProperty cmdlet can read the BizTalk install path from the registry, then Get-Content reads the file – casting it to XML for subsequent processing:

$installPath = Get-ItemProperty -Path ‘HKLM:\SOFTWARE\Microsoft\BizTalk Server\3.0’ -Name ‘InstallPath’

$btsConfigPath = [IO.Path]::Combine($installPath.InstallPath, ‘BTSNTSvc.exe.config’)

$xml = [xml] (Get-Content $btsConfigPath)

On a fresh install, the config file is quite bare and doesn’t include a <configSections> element, so in that case we need to add both <configSections> and <log4net> nodes. We can’t guarantee that other solutions haven’t already modified the config file though, so <configSections> may exist, and <log4net> may also exist – in which case, we just need to add our specific appender and logger values (log4net allows you to define multiples of these in config, and we specify names which we can expect to be unique).

To achieve this, the script checks for each element first, creates it if it doesn’t exist, then adds the specific settings:

$configSections = $xml.SelectSingleNode(‘configuration/configSections’)

if ($configSections -eq $null)

{

$configSections = $xml.CreateElement(‘configSections’)

$firstChild = $xml.configuration.get_FirstChild()

$xml.configuration.InsertBefore($configSections, $firstChild)

}

$log4netSection = $configSections.SelectSingleNode(‘section[@name=”log4net”]’)

if ($log4netSection -eq $null)

{

$log4netSection = $xml.CreateElement(‘section’)

$log4netSection.SetAttribute(‘name’, ‘log4net’)

$log4netSection.SetAttribute(‘type’, ‘log4net.Config.Log4NetConfigurationSectionHandler, log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821’)

$configSections.AppendChild($log4netSection )

}

Finally the updates are saved over the original file:

$xml.Save($btsConfigPath)

Limitations

The main limitation with any post-processing script, is that the target environment selected for the install is not available. If you have multiple bindings files, the environment selected at runtime is only alive for the duration of the MSI import – the install has no reference to it, and there’s no record made in the management database (not that I can see, please correct me if there is). This means you can’t switch your script based on environment (e.g. to use different SSO config settings for System Test and Production). If that’s a serious restriction you may prefer to create different MSIs per-environment in your build process, each containing the correct bindings file and scripts.

Specific to this approach, you need to have PowerShell installed on all the target machines, and configured to allow script execution (by default, scripts are not permitted to execute, as a security measure). Hopefully this is becoming the norm. Security also needs to be considered – the sample app writes to the registry and to SSO, so the installing context needs to have explicit permissions. The BizTalk installer runs under a separate security context from the installing user (by a trial-and-error process, this is NT AUTHORITY\ANONOYMOUS LOGON in my Server 2003 VM), so if you’re amending SSO you’ll need to set your SSO Administrators group membership correctly.

Benefits

The completed PowerShell scripts should be straightforward to read and maintain. All the post-installation requirements are implemented using a single technology, and many of the functions are reusable and could easily be parameterised and moved to a central script. The script is easy to test outside of the installer runtime, either manually using a batch file as a test harness (which sets up the relevant environment variables and then calls the post-processing file), or worked into an automated unit test.

The approach is not limited to BizTalk installations, so similar tasks for .NET deployments which are currently done with custom assemblies or Wix script can be isolated in the same way. With BizTalk and .NET installs using the same technology, you’ll build up a library of high-quality, reusable PowerShell scripts.

I also like having the scripts deployed as part of the install, so in combination with the log files, you can see exactly what’s been done to your environment and modify if necessary.

Extensions

With native cmdlets and community scripts, together with WMI, XML and .NET code, you can achieve any desired functionality with PowerShell scripts, and have them rapidly developed and tested. So you can easily add code to update version numbers in config files, remove your assemblies from the GAC on uninstall, access performance counters etc. And PowerShell scripts are just plain text so you can extract them into a T4 template and generate different scripts for different environments in your build process.