BizTalk 2006 – Deployment Beta 2 Issue

Both I and others have been experiencing an issue in BizTalk 2006 beta 2 (and previous versions) whereby if you attempt to deploy a BizTalk assembly in Visual Studio that references another BizTalk assembly that is also being referenced from another BizTalk assembly you can end up in a situation where the other referencing assembly is un-deployed.


Figure 1 – Deployment Options in Visual Studio

Did that make sense? Well let’s explain that in a little more detail.


Let’s say you have BizTalk Project 1, BizTalk Project 2 and BizTalk Project 3. BP1 and BP2 both reference BP3 like so:


Figure 2 – Project References

Now, if you wish to redeploy BP1 from within Visual Studio you will end up with BP1 and BP3 deployed, but BP2 removed. Why?


Well, when deploying BP1 any referenced assemblies will also be redeployed as part of its redeployment. This redeployment is simply to remove the assembly and install the one in your project. So, BP1 will initially remove and install BP3.


Now, the complexity comes from the fact that BP3 is referenced by BP2 in our example, so this also has to be redeployed. However, as BP2 is not related to our deployment of BP1 i.e. there are no references; it cannot be installed, simply removed.


Therefore, BP2 is removed, BP3 is removed and then installed and BP1 is removed and installed. This leaves us with only BP1 and BP3 remaining.


Obviously this is a simple annoyance within the deployment process in Visual Studio and if you know what happens under the covers then you can cater for it, or simply use another mechanism to deploy in your development environment such as from within BizTalk Administrator. I do believe that there should be a warning included in the Visual Studio Error List window to inform you what has occurred however.


A bug that I believe I have come across that is related to this issue is that when you do perform a deployment scenario as described above, you could be left with some referential integrity issues in BizTalk Administrator.


Figure 3 – BizTalk Administrator Resources

In Figure 3 you can see that the three assemblies appear to be deployed, even though I have just said that BP2 is removed. That is the issue, it really isn’t deployed and if you attempt to remove it you will receive an error like the following:


Figure 4 – Remove Error

I have obviously raised this with Microsoft, so hopefully this inconsistency in BizTalk Administrator will be fixed. However, I am not sure how they are going to approach the underlying problem. Whether they will provide an option to deploy in a different way, report on what occurs therefore raising awareness, or simply update the documentation.


A point to note that I have not mentioned before, you cannot simply redeploy BP2 again, as it would then cause the same process to happen, but this time in reverse. So you would end up with BP2 and BP3 deployed, but this time BP1 would be missing.


Happy BizTalking!

New BizTalk 2006 5 day Course -> Building BizTalk 2006 Solutions Best Practices

I am in the process of writing a new BizTalk 2006 course as below:


Preamble


BizTalk is all about messaging and how messages are processed and routed between a companies internal applications (for example: Sap/Mainframe/Peoplesoft/ Database/JD Edwards/Web Services etc.) and how messages are exchanged with trading partners (B2B). BizTalk provides an excellent framework for implementing many common messaging patterns such as:
a) Scatter/Gatherer pattern
b) Splitter pattern
c) Aggregator pattern
d) Message Broker pattern
e) etc.
The objective of this course is to provide students with all the necessary tools to implement messaging patterns using BizTalk 2006.


Below are some of the topics with labs for the course:


a) Various methods to Create Schemas for XML and Flat files messages.
b) Mapping using the BizTalk Mapper, custom XSLT and other alternatives
will be discussed to perform complex transformations.
c) An introduction into building Custom Adapters
d) An introduction into building Custom Pipeline Components
e) Interacting with the BizTalk Messaging Engine
f) Creating and using Custom Pipelines
g) Orchestrations.
h) Using xpath in Orchestrations.
i) Web Services and BizTalk
j) Error Handling with BizTalk
k) Debugging BizTalk Solutions
l) Deploying BizTalk Solutions
m) Business Activity Monitoring and BizTalk
n) Business Rules Engine
o) Implementing Common Enterprise Integration Patterns with BizTalk.
Some of the patterns discussed and put into practice are:
Message Broker
Content Based Routing
Splitter Pattern
Normalizer Pattern
Aggregator Pattern
 
Additionally the new features of BizTalk 2006 will be highlighted and used in Labs.


It is quite a bit for a five day course (not sure if all of it will make it in). From my experience using the product and teaching a BTS 2004 course for the past year and a half, the above is what I would like to cover in the course.


If you have a suggestion for additional topics for the course, please reply to this entry. I would love to hear them.


Thanks,


Matt.


 

Check out the new Backup / Restore stuff in BETA2

Hi all. I am still here. Just get a bit busy and disappear for a while. 🙂 🙂 So one of the cool things in Pathfinder is some updated backup and restore features. It is still based around the same log shipping story, but we have automated almost all of the steps and vastly improved our documentation to try and make this easier. We are still working on making it better with things like a UI and support for alternatives instead of just log shipping, but hopefully you will find the new docs easy to understand and the new procedure easier to do. I am in the process of trying to find a way to back port this to BTS 2004 but in the mean time, a lot of the work can be used as “samples” of how to automate the process for 2004. This includes a more finalized version of the scripts I have included below for the automated restoration. They are vbs files so slight modifications for 2004 can be done (I know the TDDS connection string is a slightly different format in 2004 so that will need to be changed). Also, it has some automation for restoring to the mark and you can use the way I import data to populate the LogShippingDatabases table as an example as this manual process can be error prone (I was on the phone with a customer today who had a typo while performing this step which caused failures). The underlying technology for our log shipping story is the same from 2004 to 2006, we have just tried to make it a lot easier. I highly recommend everyone check out the new docs on this feature and try it out on 2006 and then see how you can use it on 2004 while I work to officially backport it. Please send feedback as we continue to work to make this feature better for you. Hope you all have a happy holiday season.


 


Lee

Check out the new Backup / Restore stuff in BETA2

Hi all. I am still here. Just get a bit busy and disappear for a while. 🙂 🙂 So one of the cool things in Pathfinder is some updated backup and restore features. It is still based around the same log shipping story, but we have automated almost all of the steps and vastly improved our documentation to try and make this easier. We are still working on making it better with things like a UI and support for alternatives instead of just log shipping, but hopefully you will find the new docs easy to understand and the new procedure easier to do. I am in the process of trying to find a way to back port this to BTS 2004 but in the mean time, a lot of the work can be used as “samples” of how to automate the process for 2004. This includes a more finalized version of the scripts I have included below for the automated restoration. They are vbs files so slight modifications for 2004 can be done (I know the TDDS connection string is a slightly different format in 2004 so that will need to be changed). Also, it has some automation for restoring to the mark and you can use the way I import data to populate the LogShippingDatabases table as an example as this manual process can be error prone (I was on the phone with a customer today who had a typo while performing this step which caused failures). The underlying technology for our log shipping story is the same from 2004 to 2006, we have just tried to make it a lot easier. I highly recommend everyone check out the new docs on this feature and try it out on 2006 and then see how you can use it on 2004 while I work to officially backport it. Please send feedback as we continue to work to make this feature better for you. Hope you all have a happy holiday season.


 


Lee

BizTalk Retrospective

While I was at the San Francisco launch of Visual Studio 2005, SQL Server 2005 and BizTalk Server 2006, it struck me what an important milestone the event was in BizTalk’s life cycle. For the first time, BizTalk has been elevated to a high-profile position among Microsoft’s developer toolset. For many developers that have never really had any exposure to BizTalk, this makes it real and should get their attention. If you’re a BizTalk developer, then you’re in a great position right now as demands for you skills are about to go ballistic. Scott Woodgate just did an excellent post on his blog about this.


In fact, if you’re a REALLY good BizTalk developer (architect type with multi-years BTS experience that at night dreams about pipeline components and messaging patterns) and you’re looking for a new gig working with a highly-talented team, I’d love to hear from you. Drop me a note and CV at my first name, plus a “.“, plus my last name, at “Neudesic“, plus a “.com“.


This seems like a great time for me to share something I came across while doing an “architectural dig” (that’s what I call the every-year-or-two cleanup of my office). I don’t know how many of these CDs were run, but I don’t think it was very many.


Yes, that’s “copyright 1999”.


What a long way we’ve come, in a short period of time, from this CD to the current gig I’m on in the Bay area. We’re building an ESB to support SOA and EAI, featuring 56 procs of BizTalk. I can’t imagine how I would do this without BizTalk. I suppose it _could_ be done, but it’d take a TON of custom app dev, cost orders of magnitude more, and take a very looong time to deliver. We’re building it with a relatively small, fast moving team and the end result will be highly extensible, scalable, and reliable.


BizTalk developers…. The next couple of years are going to hold a lot of opportunities. Enjoy the ride!

BizTalk 2006 – Using External Assemblies Bug

One bug that currently exists in all pre-release versions of BizTalk 2006 is the way an external assembly reference is held onto from within a map. If you call an external assembly and use the “Test Map” option to test your map, you have to restart Visual Studio before any modifications to that external assembly is reflected in the map.


In my scenario I have two projects in my solution – one c# Class Library that contains a class to be called from a map, the other a BizTalk project containing the map to use. The map solely uses an external XSLT file and the external call is made in a fashion similar to that below:


<xsl:variable name=”fragment” xmlns:externalHelper=”http://schemas.mycompany.com/MyDept/MyApp” select=”externalHelper:MyExternalCall(node())”/>


This works as expected until you change the c# class (recompile and re-GAC), as any changes are not reflected when testing the map. I have tried restarting all sorts of processes to get these changes to take effect. The only solution I have found is to literally close Visual Studio and start it again.


During development this is certainly a pain in the neck, especially when you are forced to use a rather slow virtual machine. I am waiting to see whether the product team implement a fix prior to the full release, but I cannot see it making it unfortunately, as it is hardly high priority.


So why did I post this? Hmm, partly a reminder, a warning and a medium for voicing my displeasure 🙂

New functions in BizTalk 2006 Rule Editing

There is an excellent blog article writen by Richard Seroter on how to enable static methods in the BizTalk 2006 rule engine. Seems that for the time being you have to get into the registry. I wonder why these essential functions are always hidden? Are there any caveats?

You can read it here

Let’s see if it will make it into the the final release. This would eliminate the need for all these simple .NET conversion functions that are needed currently with BizTalk 2004. No need to wrap it in an IFactCreator, and no need to create an instance to test the rules.

BizTalk, MIB, Acord and 5 minutes of Marketing

My blog entries are usually technical in nature. This entry is more of a marketing entry discussing how BizTalk 2004/2006 can be used to implement the following specific business scenario:


1) An insurance underwriter requires information (for example -> medical history) about an insurance applicant.


2) An internal application captures applicant information provided by the underwriter such as:
a) Applicants First Name/Last Name/ Middle Name.
b) Birth date.
c) Birth place.


3) The internal application then sends this information to a company called MIB. The underwriters query parameters (Last Name/ First Name etc) are then used to query a data repository at MIB. A result set of the query is then returned to the internal application from MIB. This result set contains information such as the medical history of the applicant.


4) The results from MIB are then displayed to the insurance underwriter to aid in the insurance application process.


Presently many North American insurance companies send/receive information to/from MIB over a dial up connection. Flat Files are sent and received back from MIB.


By December of 2006, all users of the current dial up process are to switch over to a new process provided by MIB as below:


All insurance underwriter requests are to be transmitted to MIB using a secure HTTP connection. The requests are to be sent in XML messages that conform to the Acord schemas. A synchronous HTTP response (XML) message will then be sent back over the same secure HTTP channel as the request. This response message contains the results of the query.


Below is a description of how the above new process can be implemented by BizTalk 2004 or BizTalk 2006.


1) Create a new BizTalk project that contains the Acord XSD schemas as below:




2) Typically a mainframe application may dump the underwriter requests to disk in a flat file or different format.
A configured BizTalk FTP or File Receive Port is then configured to pick up the files.
 
Note: The original request may be delivered by a different transport or stored to intermediate location as below:



HTTP
MSMQ
MQSeries
Sql Server
Oracle
DB2
Host Applications  -> IBM, mainframe zSeries (CICS and IMS) and/or midrange iSeries (RPG)
Host Files -> Data adapter for host file systems on IBM mainframe zSeries VSAM datasets and midrange iSeries physical files
etc.



BizTalk contains many out of the box adapters that can be used to receive messages from all the above transports/locations and others not listed. If the needed adapter is not provided out of the box, then the BizTalk Adapter framework can be used to author your own custom adapter. Third party adapters can also be purchased.


3) Create internal XSD schemas to represent the internal messages.
For example, an XSD schema using flat file extensions would be created  to represent flat files produced by the mainframe application.


4) Create BizTalk map(s) to transform the internal flat files to Acord XML messages.


5) Create the necessary BizTalk orchestrations for the business process. 


6) Install a certificate on the BizTalk machine for the MIB secure HTTP requests using the Certificate Snap-in.


7) Create a BizTalk Solicit-Response Send port with a configured HTTP adapter for the MIB secure HTTP requests. The Send port is configured to use the installed certificate from step 6).



8) Create BizTalk map(s) to transform the Acord XML response messages to internal messages.


The above pretty well covers all the required BizTalk objects.


So why use BizTalk for the above MIB request/response loop or a similar business process:


1) This is what BizTalk server does. It integrates a companies internal applications together and also provides B2B capabilities.


2) A number of turnkey solutions can be purchased that do the same communication loop with MIB. These turnkey solutions are probably more expensive and proprietary than a BizTalk Solution. There is little coding involved to build the above BizTalk solution and there is complete control over the process as a developer(s) is building it and maintaining it. For changes or modifications to the BizTalk process, a configuration change is sometimes all it takes.


For BizTalk 2006 pricing go to HERE
For BizTalk 2004 pricing got to HERE


I have heard claims that it took more than 1000 man hours to complete the above process in BizTalk. This is completely untrue as it took me a fraction of 1000 hours to implement the above MIB communication loop using BizTalk.


3) The BizTalk server that hosts the above process can be used to host other developed Application Integration or
B2B business processes.


4) BizTalk server is increasing exponentially in popularity. More and more developers have BizTalk experience.


5) BizTalk provides many useful out of the box features such as:


a) Business Activity Monitoring
b) Rules Engine
c) Fault Tolerance/Load Balancing
d) Tracking
e) etc. etc.



Note: I do not mean to sugar coat the process. The BizTalk maps for the MIB requests and response messages are somewhat complex. The complexity of the maps can be attributed directly to the large size of the Acord XSD schemas. There are larger XSD schemas. In my opinion, the HL7 XSD schemas are more complex then the Acord XSD schemas.

BizTalk Server 2006 Beta2 is out – What’s new in WSS Adapter?

Microsoft BizTalk Server 2006 Beta2 is now available on Microsoft Beta Place. If you have not signed up already for BizTalk 2006 Beta program you can follow the instructions that I posted in this previous post in order to download the bits. If you have already signed up for BizTalk Beta 1 then you can just go to https://beta.microsoft.com, sign in with your account and download the latest BT 2006 beta.


The setup/configuration is pretty much the same so you can follow the steps from my previous post but now you have to use the RTM version of Visual Studio 2005 and Windows SharePoint Services 2003 with SP2.


BizTalk 2006 Beta 2 is a production-ready build, and of course upgrade from Beta 2 to RTM will be supported.


So, what’s new feature wise in Windows SharePoint Services adapter included in BizTalk 2006 Beta 2 when compared to previous Beta 1? See below:



  • new macros %Filename% and %Extension% are supported in all send port/receive location fields that accept macros. These macros are replaced at runtime with the filename and extension values computed from the WSS.Filename context property of the message being processed. All messages that are received from SharePoint have this property automatically initialized to the filename of the SharePoint file that generated the BT message. As a result these macros will be replaced with the filename and extension of the original SharePoint file (if the message originated from SharePoint), or with the filename and extension defined in the orchestration/custom pipeline if the orchestration/custom pipeline specifies a value for the WSS.Filename context property of the message. Unlike the rest of the WSS adapter macros, these two macros cannot be used in WSS adapter context properties values that are defined in an orchestration/pipeline. This feature addresses a part of Edgardo’s well justified complaint that he made a while ago . The other part of the complaint (about Versioning support) was not possible to address in this release but it might be addressed by a future release.
  • the send port “Microsoft Office Integration” setting now supports an “Yes (InfoPath Form Library)” option. If this option is enabled and the XML message is being sent to a SharePoint InfoPath form library, the XML file being created will automatically be linked to the InfoPath solution attached to the SharePoint form library so that the file will open up in InfoPath. When Microsoft Office Integration is set to “Yes (InfoPath Form Library)”, no values are required for the other Microsoft Office Integration fields but the message must be sent to a SharePoint InfoPath Form Library.
  • The behavior for “Optional” setting of “Microsoft Office Integration” send port property has been changed slightly to accomodate the new value “Yes (InfoPath Form Library)”. When Office Integration is set to “Optional”, the adapter will try to use both discovery mechanisms (“Yes” and “Yes (InfoPath Form Library)”). If both mechanisms return a solution, the solution found in the Templates Document Library/Templates Fallback Document Library will be preferred. If no InfoPath solution is found the message will not be suspended, instead it will be saved as-is (no change here).
  • the send port “Overwrite” setting now supports a Rename option. If a file with the same name exists in SharePoint, the name of the file being created will be made unique by appending the message identifier %MessageID% to it. This option is not supported for receive location ‘Archive Ovewrite’ property.