BizTalk 2009 thoughts…

by community-syndication | Dec 15, 2008 | BizTalk Community Blogs via Syndication

Have you had a chance to play
yet ?

Many folks have noted that a lot of the Visual Studio elements that have been present
within BizTalk to support the development experience are no longer…quite so BizTalk
specific!  BizTalk projects now build upon C# projects, and thus a lot of the
differences that you used to see in navigating property pages, compilation settings
and build mechanics are now gone.  This is a very good thing – it allows you
to leverage skills you already have on your team.

MSBuild support is now first class.  Everyone who went through the trouble to
install (and invoke) DevEnv.exe on their build server in order to build BizTalk projects
will be glad to know that this is no longer required.  (Builds can be ever so
much faster when you aren’t relying DevEnv…)

Because of the close relationship with C# projects, you can now have C# artifacts
directly in your BizTalk projects.  Many people have noted that when you “Add
new item…”, C# classes aren’t offered as an option.  The product group has
explained that this is because currently, the designers (such as the Orchestration
designer) are unable to provide intellisense for types that are within the same assembly. 
So, you are required to use “Add existing item…”.

(This sort of reminds me of the interaction between
pipelines and schemas – the former require fullly qualified assembly information at
run time, which they won’t get if you combine these two artifacts in the same assembly.)

If this limitation persists, will the feature get used?  I tend to think so. 
There are often cases where the smart thing to do in an orchestration is to delegate
to a component…but the work you do in that component is, at times, so specific to
the orchestration that it makes sense for them to be co-located for deployment and
organizational purposes.  What do you think ?

The support for unit testing is extremely welcome – check it out.  Debugging
maps is great as well, but I’m now (often) partial to the external
XSLT approach these days.

BizTalk Adapter Pack v2 CTP

by community-syndication | Dec 14, 2008 | BizTalk Community Blogs via Syndication

BizTalk Adapter Pack team has released a CTP. This is  the 5^th CTP and the first one available publicly to everyone to try.  The specific enhancements in this CTP  are mostly to the OracleDB adapter(included in the BAPv1) to which we added features like support for polling stored procedures and composite operations. In addition we have addressed some feedback we got from the other adapters .

To re-iterate here is the complete list of features BAPv2 is adding . The downloads are available internally at http://adapters and on the connect site

BizTalk Adapter Pack V2 – Themes

New Adapters

%u00b7           Oracle EBS Adapter

%u00d8  Browse and search metadata for each Oracle application

%u00d8  Support Interface tables, Interface views

%u00d8  Support for PL/SQL APIs

%u00d8  Concurrent Programs, Request Sets

%u00d8  Support for polling tables, Stored Procedures

%u00d8  Notification support

%u00d8  App context initialization

%u00d8  Supports REF cursors, User Defined Types, LOB types, RECORD types

%u00d8  Support for PL/SQL Tables, boolean data types

%u00d8  Support for MLS EBS installations

%u00d8  Microsoft Office SharePoint Server Integration

%u00d8  Versions supported – ODP.NET  11.1.0.6.21 (available only on 32bit), Oracle EBS 11.5.9, 11.5.10 and 12

%u00b7           SQL Server Adapter

%u00d8  Insert/Update/Delete/Select operations on tables and views

%u00d8  Invoke (SQL and CLR) Stored procedures, scalar valued functions , table value functions

%u00d8  Supports execution of generic T-SQL statements, via the ExecuteReader(), ExecuteNonQuery() and ExecuteScalar() operations

%u00d8  Configurable polling – polling statement, polling interval, receive multiple result sets

%u00d8  Support for x86 and x64  platforms

%u00d8  Support for new data types in SQL2005 and SQL2008 (notably FILESTREAM, varbinary (max))

%u00d8  Support for UDTs

%u00d8  Ability to pass table value parameters

%u00d8  Composite operations – invoke operations on multiple tables and call any number of stored procedures in the same transaction

%u00d8  SQL query notifications

%u00d8  Ability to directly call stored procedures which used the ’for xml’ style supported by the BizTalk 2006 SQL adapter to ease backward compatibility

%u00d8  Versions supported – SQL 2000, SQL 2005, SQL 2008

Enhancements to the existing V1 Adapter Pack

• Include hot fixes built since the V1 Adapter Pack released


• Support new Microsoft Platform releases – BTS 2009, VS 2008, Windows Server 2008


• Oracle DB Adapter:

%u00d8  Polling support for tables, Stored procedures

%u00d8  UDTs, PL/SQL tables  (from Beta onwards)

• SAP Adapter:

%u00d8  Support for Table Types

%u00d8  Customization of adapter behavior when “special” values obtained for DATS/TIMS/NUMC

%u00d8  SAP ADO Provider Enhancements:

o   Support for SSRS

o   Support for SAP Queries

o   Ability to specify the result set to give to SSIS when executing BAPIs

• Siebel Adapter:

%u00d8  MVG operations (associate, dissociate, query) in design time enabled

%u00d8  Bounded pick list fields supported per recommended approach

%u00d8  Context specific custom messages generated for exceptions

First Look at BizTalk 2009 Build and Developer Experience

by community-syndication | Dec 13, 2008 | BizTalk Community Blogs via Syndication

We had a very useful brown bag session with the BizTalk Product Team this week and went through some of the new developer improvements. Fortunately not much of this was under NDA so I thought id share some of what I learnt and my thoughts on this. Ill just keep this in the simple format below:

BizTalk Project Files are now MsBuild

A new BizTalk project file is basically an extension of a C# project file. It looks very similar to a normal MsBuild based project file see the Picture below.

From this picture you can see lots of familiar stuff. You can see from the following picture from the project file that a BizTalk project file imports the C# MsBuild targets as well as some new BizTalk ones by default.

I believe BizTalk projects have their own project type guid in the project file system which indicates them as a certain type of project file. You can see this above in the MsBuild script where the Project type Guids indicate this project contains C# and BizTalk stuff. I would assume here you can also add the project type guid for MsTest projects and then start putting tests in this project also. In practice you probably wouldn’t do this, but it illustrated whats possible with project files now being in MsBuild.

BizTalk Projects Can have non BizTalk Artefacts in them

That’s right you can now add other artefacts than BizTalk ones. Some examples of this might be you have a C# class, a text or xml file in your project. Because of what I said above about the C# targets this means if you include a .cs file in your BizTalk project then it will be compiled and made available just like any other normal class. There are a couple of questions and limitations relating to this however:

1. In the current CTP you can not Add New Item and add a C# file etc to the project. This is expected to be in future releases, but for now you will need to add the item to the file system then add it as an existing item.
2. Because the BizTalk project file extends on C# if you add a VB file it will not by default be compiled. If however you add the VB MsBuild targets it should be possible to compile these files also. Essentially its just an MsBuild script so you can do what ever you can do with .net projects.

Assembly Info

BizTalk projects now have AssemblyInfo.cs type files like any normal C# project. This makes things simpler for manipulating these kind of files. See the pic below, if you notice there is a new assembly attribute called BizTalkAssemblyAttribute which I guess indicates the assembly is a BizTalk assembly.

No more Deployment/Development

One thing which makes me very happy is that the pointless configuration difference between C# and BizTalk projects is now gone. You can see this below.

It would be nice if the Deploy tick box wasn’t checked by default also. This is a useful setting for small demo type projects but in real projects you are unlikely to deploy your BizTalk assemblies through the Visual Studio IDE. I guess now we are based on MsBuild it might be possible to create a new project template where this is not checked (possibly a small community project here).

Compile BizTalk Solutions without Visual Studio

In the installation of BizTalk there is an option to install Project Build components. This means you can use these components to build your solution without Visual Studio. This is what a number of people have been asking for to help them implement the good practice on a Build server of compiling code without needing Visual Studio.

Strong Naming Change

The way BizTalk projects are strong named has changed and is now the same as for C# projects. From the picture below you can see this is set through the project property editor and it sets information in the MsBuild project file. Again another handy improvement in that it is just the same as C#.

Common Project Settings

As you can see from the below picture you can see the BizTalk project also has the same settings which are commonly used in C# projects. You can set the highlighted sections so you can set warnings and constants etc.

One of the more interesting things however though is that you can setup the XML documentation for the project. At present this will only do documentation for things like C# files within the project, but I guess in the future this is something that either Microsoft or the community may look at as a way of incorporating BizTalk Documenter style functionality into the compilation process.

Project Migration

When you migrate your project to BizTalk 2009 there is a migration wizard similar to the one for migrating .net code. You can use this or you can also call DEVENV from the command line with the /UPGRADE switch to migrate your code. You seem to get the normal features of the Visual Studio migration wizards. One limitation here is I believe the project migration does not change the project configurations from the old Development/Deployment modes so you will have to do this manually. And also apparently you can migrate from BizTalk 2006 or BizTalk 2006 R2 to BizTalk 2009.

Unit Tests

As I have mentioned in some other recent posts there is a new Unit Tests property as shown in the picture below. This will make the compilation process generate some additional code to help you unit test various BizTalk artefacts. If you would like some more information about how to use these features please refer to my updated BizTalk Testing Guidance on the following link:

http://geekswithblogs.net/michaelstephenson/archive/2008/12/12/127828.aspx

Improved Build Performance

Apparently when BizTalk projects are compiled now the MsBuild will be able to tell if things have changed and only compile what has changed. Maps and Schemas seem to be complied to a .cs file before being compiled into the assembly a bit like Orchestrations could be if you made that tweak in earlier versions of BizTalk. This means MsBuild is able to detect the changes to these files and only recompile them as needed. This should in theory improve the overall solution compilation time. This could make a big difference on those projects where you end up with lots of assemblies.

This is most of what I remember from the session which was really useful, thanks to the guys who put this together.

Performance Story about Traceflag T1118 and the cluster registry checkpoint service

by community-syndication | Dec 13, 2008 | BizTalk Community Blogs via Syndication

I’ve just got back from a week working with a customer. The customer was a large retailer using BizTalk to perform processing over AS2.  They had just installed a new QA environment and wanted the operational environment to be optimized for their application.  I had a gotcha with SQL Clustering which I wasn’t aware of and I wanted to share.

As with any BizTalk installation, I started from the ground up and switched their SQL Storage from Raid 5 (which is very bad for write performance – which BizTalk does a lot of) to RAID 10.  Over time in the lab I’ve also found that changing the NTFS cluster size can gain performance improvements.  Typically we use 64KB for data LUNS (which host the BizTalk Database MDF files) and 4KB NTFS cluster size for log file LUNS (LDF’s).

Optimizing TEMPDB is another of the optimizations that we tend to do.  Specifically splitting temp db into multiple equal size files, 1 per Logical Proc available on the server.  To reduce contention further on SQL Server and prevent almost all single page allocations Trace Flag T1118 is then applied.

I used SQL 2005 configuration manager to make the change by opening the tool, clicking SQL Server 2005 Services and then double clicking SQL Server (instance name).  In the SQL Server (Instance name) properties dialog box on the advanced tab I added the following text at the end of the existing string: -T1118.  Crucially what I’d forgotten to include was the semi colon required.  So what I’d actually done was change the following

…-lc:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\mastlog.ldf

to

…-lc:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\mastlog.ldf-T1118

In other words the SQL Server would not attempt to load a log file for the Master database with name mastlog.ldf-T1118. Still unbeknown that I had made this error we rebooted the cluster so an OS hotfix could take affect. 

When the server rebooted SQL Server would not start and the following error was in the SQL Error log

Can’t open file “c:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\mastlog.ldf -T1118”

We removed the T1118 from the Configuration Manager properties used previously and attempted to bring the SQL Service back online.  It failed again with the sam error in the error log. We examined the SQL configuration manager properties and the -T1118 is still appended to the Master log file name.  We try this several times even going into the registry directly (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\Parameters) to try and change it.  But no matter what we do it stayed there.

 Those of you who are cluster gurus reading this are probably laughing because you know what has happened.  If SQL is clustered it is the WIndows Clustering Service which controls its execution and manages based on the clsuter config which node it will start on.  In order that SQL Server functions correctly when failed over between servers, the cluster service stores a checkpoint registry on the cluster quorum disk.

In this case when the change was made and the cluster was rebooted, this incorrect configuration was stored on the quorum disk. Because changes to the checkpoint can only be made when the service (in this case SQL) is online the change I was making to remove T1118 was never getting saved and the incorrect config was consistently erasing our right config.  Talk about frustrating.

Fortunately there is a work around on this, which one of my colleagues in the Clustering Support Team was able to help me out with.

 1. Disable the cluster checkpoint for the specific registry key:

c:\> cluster res “SQL Server” /removecheck: “Software\Microsoft\Microsoft SQL Server\MSSQL.\MSSQLSERVER” 

2. Correct the trace flag using SQL configuraiton manager or the registry

3. Re-enable Cluster checkpoint for the specific registry key:

c:\> cluster res “SQL Server” /addcheck: “Software\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLSERVER”

Duplicate SOAP Subscription on Dynamic Request-Response Port

by community-syndication | Dec 12, 2008 | BizTalk Community Blogs via Syndication

Joy of joys, more subscriptions problems to debug this week with some unexpected results.
The first sign of problems were persistence errors:
Microsoft.XLANGs.Core.PersistenceException: Exception occurred when persisting state to the database. —> Microsoft.BizTalk.XLANGs.BTXEngine.PersistenceItemException: A batch item failed persistence Item-ID dbbd66a3-d748-4051-a238-fea9509efcf4 OperationType MAIO_CommitBatch Status -1061151949 ErrorInfo The message found multiple request response subscriptions. A message can only be […]

Incoming version MBV 10.00 : Query custom properties support included and Gui interface redesigned

by community-syndication | Dec 11, 2008 | BizTalk Community Blogs via Syndication

Hello,

Long time I did not write a post about MBV !  two reasons mainly : I was quite busy in my BizTalk Tech Lead job and I released until now on my blog only new minor MBV builds for which I considered that dedicated posts were not maybe justified; so I just made them available on my classic download link http://blogs.technet.com/jpierauc/pages/msgboxviewer.aspx  listing quickly their new small features and the fixes.

I had took time however to think about an interesting major feature I could include in MBV to give to the tool again more flexibility for users and more attraction, and I think it can justify a post this time to present it :-).

I will make available very soon (probably next week) a new major build (10.00)  containing I think an interesting new feature that I present below.
This build is not available immediately as I have small tests to do yet before and I would prefer that an important Access Denied  bug found very recently is solved definitively (see below for more details) before.


I had so noticed two main areas of improvement in MBV :

1)      Until now, it was not possible to enter at runtime some properties per query (like a “sort order” prop for example) . The sort order properties  for “Send Ports/RL/Orchs”  or “Tables Size” queries were well present in previous versions but were just some options in the ”Options” tab and were specific to only these queries.
I thought also that it could be interesting to be able to specify some properties at runtime for any type of query contained in MBV before to start an analyze

I decided so to add support of query custom properties in  my Health Check engine that MBV use (I will write soon another post presenting you MBV Archirtecture and the HC engine that I developed which can be used very easily in any .NET app to do a custom Health Check).

Each query can have so 0 to N custom properties with following attributes :

a.       ID
b.      Caption
c.       Default Value
d.      Value
e.      ReadOnly flag
f.        Mandatory Value flag

When a such property is defined for a query, its ID can be used in the query body or query title to be replaced dynamically during collect execution, using syntax “%u00e0 la macro” %<MYPROPID>%

This possibility allow so now to have parameterized queries with possibility then to enter quickly parameters value before to start a collect.
This new build of MBV  that I hope to release next week will allow so to enter very easily value for each custom property present for a query (if they are not read only) depending of the query selected in the Query ListViews.

2)  Until now, I have to admit that Gui interface of MBV tool (so the Gui tool version “MsgBoxViewer.exe” ) was quite “heavy” and confusing with lot tabs and fields everywhere present in the form  and

That’s why, with advises of Jean Severino my colleague, I redesigned the interface to be more easy to use, more intuitive, and less loaded I think :

a   . I removed the User Roles support as it was rarely used
b.      I grouped the output files Path fields and periodic collect options in the OPTIONS tab
c.       I removed the numerous Queries output Tabs keeping only two:  one displaying the Header and one displaying the Summary Report
d.      For each query you select (not just check), its standard and custom properties are displayed immediately on the right side and you can enter quickly values for custom properties.

Once I will make available this version on this blog, you could see concretly this possibility selecting for example one of these queries :   “Send Ports”, “Db Space”,  “BizTalk Perfmon counters”, “Last Svc Instance” etc.  you will see then the corresponding custom propertis displayed on the right side of the dialogbox and can change them if you want

Each value you enter in a custom property will be saved into the MBV XML Settings file when you exit the tool 


Other Features in build 10.00:

BizTalk Testing Guidance – Revisited

by community-syndication | Dec 11, 2008 | BizTalk Community Blogs via Syndication

I’ve updated my BizTalk Testing Guidance with some new articles based on what we learned today about BizTalk 2009. Please find the latest index below:

BizTalk Testing Series – Testing Pipelines and Pipeline Components in BizTalk 2009

by community-syndication | Dec 11, 2008 | BizTalk Community Blogs via Syndication

A while back in my series of articles about testing with BizTalk I covered a technique for testing pipelines and pipeline components. The background to the approach is still valid in BizTalk 2009, but the technique is improved. To review the previous article refer to the following link:

Pipeline Components – http://geekswithblogs.net/michaelstephenson/archive/2008/03/30/120852.aspx

Pipelines – http://geekswithblogs.net/michaelstephenson/archive/2008/03/30/120853.aspx

In this post although we could do testing of pipelines and pipelines components using the excellent pipeline component test library. While that component is excellent for testing, I think the new model provides a simpler option and to be honest with Pipeline Component Test Library it sometimes took a little bit of fiddling with setting up the pipeline wrappers to get your test to work. My gut feeling is that the new approach will just be that bit more straightforward.

I will extend on the sample from the previous post about testing schemas. So our solution now looks like the below picture containing a pipeline

1. Configure your BizTalk Project for Tests

Just to reiterate this step from the previous post you need to ensure the enable unit testing property is set for the BizTalk project.

1. Adding references to your test assembly

Again reiterating this step, in your test project you will need to add a couple of references, the main one is the new Microsoft.BizTalk.TestTools assembly which is in the Visual Studio Add Reference list. This will allow you to use these new testing features. You would also need to add references to the pipeline assemblies highlighted in the below picture

1. The test code

In the below code you can see that I will construct an instance of the TestableReceivePipeline which is what my custom receive pipeline will derive from. I can then provide documents and schema information and then execute my pipeline. Unfortunately the documentation of these objects doesn’t seem to be complete in the initial CTP so Im guessing there are a few more ways you can configure these objects, but they will give you an easy way to test send and receive pipelines.

1. What about Pipeline Components

In the Pipeline Component Test Library there was the ability to be able to construct a pipeline wrapper in code so you could simulate adding your custom pipeline component to different types of pipeline. I’m not sure that you can do this in the new features of BizTalk 2009, however you can just create a pipeline artefact and add your component to that. The other cool feature is you might not (haven’t tried this) even have to add a new project for the BizTalk artefact as you can have BizTalk and C# files in the same project.

Summary

As you can see this new technique appears like it will be a simpler way to encourage testing.

The sample for this demo is available at the following location: http://www.box.net/shared/7rb81bcxsm

If you have problems accessing it then let me know.

BizTalk Testing Series – Testing Maps in BizTalk 2009

by community-syndication | Dec 11, 2008 | BizTalk Community Blogs via Syndication

A while back in my series of articles about testing with BizTalk I covered a technique for testing maps. The background to the approach is still valid in BizTalk 2009, but the technique is improved. To review the previous article refer to the following link:

http://geekswithblogs.net/michaelstephenson/archive/2008/03/30/120851.aspx

In this post although we could do testing of maps, to be honest it was a little bit of a pain. In the newly released BizTalk 2009 CTP there have been some improvements in how you can do some of the testing and I would like to demonstrate this. I will walk through the steps to test your schema.

I will extend on the sample from the previous post about testing schemas. So our solution now looks like the below picture containing an additional schema and a new map.

1. Configure your BizTalk Project for Tests

Just to reiterate this step from the previous post you need to ensure the enable unit testing property is set for the BizTalk project.

1. Adding references to your test assembly

Again reiterating this step, in your test project you will need to add a couple of references, the main one is the new Microsoft.BizTalk.TestTools assembly which is in the Visual Studio Add Reference list. This will allow you to use these new testing features.

1. The test code

In the below picture you can see that I am doing two things. The first is that I create an instance of the new TestableMapBase class which my map derives from. I can use this to execute my map by passing the details of an input file. The results are written to the output file.

I’m also using the schema validation to confirm that the message is valid that comes out. I think there are also properties of the TestableMapBase object which can do this.

In practice when I use this technique I will probably execute the map with a known input file and then compare the results of the output file with a predefined example output file so rather than just validating the schema I’m actually also ensuring the content is exactly as expected. You can do this with a binary comparison of the file.

Summary

As you can see this new technique just simplifies what we had been doing in BizTalk 2006 R2 and is a very nice interface to support this testing.

The sample for this demo is available at the following location: http://www.box.net/shared/yro0poto3m

If you have problems accessing it then let me know.

BizTalk Testing Series – Testing Schema in BizTalk 2009

by community-syndication | Dec 11, 2008 | BizTalk Community Blogs via Syndication

A while back in my series of articles about testing with BizTalk I covered a technique for testing schema. The background to the approach is still valid in BizTalk 2009, but the technique is improved. To review the previous article refer to the following link:

http://geekswithblogs.net/michaelstephenson/archive/2008/03/30/120850.aspx

In the newly released BizTalk 2009 CTP there have been some improvements in how you can do some of the testing and I would like to demonstrate this. I will walk through the steps to test your schema.

Before I go into the details, we start with a simple project with two projects. The first is a BizTalk project with a schema and the second is a C# test project. The following steps need to be taken to setup your tests

1. Configure your BizTalk Project for Tests

The first step is to open the properties for your new BizTalk project. If you go to the Deployment section you can see highlighted in the below picture the setting which will turn on the generation of some additional code to allow you to test BizTalk artefacts.

1. Adding references to your test assembly

In your test project you will need to add a couple of references, the main one is the new Microsoft.BizTalk.TestTools assembly which is in the Visual Studio Add Reference list. This will allow you to use these new testing features.

1. The test code

The following code snippet shows how you can create an instance of your schema which derives from the new TestableSchemaBase object. This has the new ValidateInstance method which allows you to pass in the path to a file containing the message you wish to validate.

Summary

As you can see this new technique just simplifies what we had been doing in BizTalk 2006 R2 and is a very nice interface to support this testing.

The sample for this demo is available at the following location: http://www.box.net/shared/2ojysf5ys3

If you have problems accessing it then let me know.