I think my daily "workflow" is about to get more interesting
I’m very excited about our new addition, but boy does it bring to mind all sorts of questions about time management. 😉
Thank God for BizTalk 2006
This past week, I had the joy of helping out a Client with BizTalk 2002. Wow, I haven’t said this in a while! I decided to jump in my time machine and do an install….Windows 2000 with Visual Studio 6.0 and BizTalk 2002. I was surprised that I got this working in a short amount of time with no errors! I fired up the infinite amount of user interfaces for the product and began waking up some of my dormant brain cells pertaining to 2002.
So when you look back at the product, is it really that much different? You have an editor, a mapper, an orchestration designer…wait, disregard my last sentence. All I can say about the 2002 orchestration designer is…Yuck. I then was overjoyed to bring up the Messaging Manager and navigate my way through the Channels and Ports…boy, this was a confusing product, but I will admit that we did get it to work!!! All I can say is Thank God for BizTalk 2006! What a difference in just a few years. I just love looking at old VB 6.0 code….On Error Resume Next and fall through and not catch an error and wonder why it doesn’t work and blame it on BizTalk!
I was able to help out our Client and get them moving in the right direction. They had some interesting issues, but didn’t know where to look to resolve these issues. I’ll make a statement here and I don’t think I’m going to shock anyone:
A MAJORITY OF BIZTALK ISSUES THAT I COME ACROSS ARE NOT RELATED TO BIZTALK AT ALL, BUT RATHER EDUCATION ON THE PRODUCT AND IT’S CAPABILITIES.
Installing BizTalk does not mean that you automatically have a good design in place and are free from bad coding. Do you agree?
Whole buncha stuff
So it has been a while. In the mean time, WE SHPPED!!!! It is such a great feeling. I have been on this team for almost 7 years now and this release is defnitely the best (one would of course hope that every release is better than the previous but you never know 🙂 :). I am super happy to have this out there and hope you are all going to be really happy with it and we are already working on really cool stuff for the next release. There is rarely a break from inovating. 🙂 🙂 However, I did manage to take two weeks recently and go to New Zealand which is an absolutely amazing country. You Kiwis really know what is up. Even if a town has only 5 buildings (one of which will always sell fish and chips and the other will have postcards of sheep), you can always find an information bldg which will tell you everything to do, book it for you and send you on your way. It is the best country for tourists. I went surfing, caving, kayaking, glacier hiking, tramping, river surfing, bungy jumping and lots more. It was awesome. I am also apparently now a Hurricanes fan cause the Crusaders win too much and the Blues, well, they are the blues. 🙂 🙂 I think I should spend more time with customers in New Zealand … we’ll see.
My role on the team has changed a bit and now I have a team to help work on all the great features. Sumitra S has been working with me for a while now and I am going to work on getting her to post to this blog as appropriately. She rocks and along with a bunch of work on the messagebox core functionality she wrote the (internal) OM which exposes all of the data and operations in the Admin MMC Group Hub Page. Also joining us now is Adrian Hamza after working on the Share Point adapter most recently He has his own blog which you should all checkout at http://blogs.msdn.com/ahamza. He is great and we are really looking forward to his work on the core engine. Craig C has also joined the team and will be helping define some of the long term direction we are going towards. Should make an awesome team. 🙂 🙂 🙂
So now maybe I will give you some good technicall details so that you are not just reading fluff. 🙂 One thing I have really wanting to post are some learnings I have had with using Yukon and 2 tips that could really help some people avoid the gotchas that we hit when porting code directly that must run on both Yukon and Shiloh (I’d post more but it is 1 am).
1) After a number of conversations with the optimizer team, we discovered a hidden gem in the use of WITH SCHEMABINDING on UDFs. Our UDF simply took in three datetime variables and did a somewhat complex comparison of them and returned 1 or 0. It did not touch any tables and so did not affect any data. In Yukon there is a new feature / attribute associated with a UDF called SystemDataAccess. This property can be seen via:
SELECT OBJECTPROPERTYEX(OBJECT_id(’<MyFunction>’), ‘SYSTEMDATAACCESS’)
Without the WITH SCHEMABINDING option, this property is always set to 1 so the optimizer assumes that you might access the data and will add sort / spool to your plan to protect itself from data changes. Adding the schemabinding hint, causes the function to be parsed when it is created and then you can see this property set to 0 and the sort / spool disappears. It is really quite nice and can be a very large perf improvement which I believe will be KB’d at some point. In our case, this is a major gain in the performance of dequeue for large queue sizes. Thanks to the SysRepublic guys for discovering the perf issue with large queues and triggering this discovery. You will find that our peformance for large queues is better on Yukon in some scenarios because of this .
2) Always owner qualify tables and sprocs / udfs which you execute. SQL plan cache access actually uses this as part of its lookup in the cache and missing this can cause a certain amount of extra cache misses and plan (re)compiles resulting in higher CPU utilization and potentially lower performance. What I mean is instead of “SELECT * FROM Foo” you would write “SELECT * FROM dbo.Foo” assuming the owner of Foo is the dbo. Unfortunately this is something we found too late and it actually didn’t make it into our RTM bits as it was not a functional bug and was such an extensive change that we could not risk our RTM date since we had reached our perf goals. We are pushing to QFE this, though, and I will let you know if / when this happens. All it means is that in certain scenarios, on Yukon, we would use more CPU than on Shiloh.
These two things are just so wierd that I figured I would post about them as case one could be a big gain for those who do this (which might not be a lot of you) and case two is probably all of us and these are fixes that can do which have no effect on Shiloh but can have large gains on Yukon. Hope this helps some of you and I expect the Yukon guys will KB this stuff at some point (as well as investigating why #2 is currently worse than Shiloh).
Next time, I will post something cool about BizTalk, but now I have a coulpe more things to catch up on before I go to sleep. 🙂
Thx
Lee
ENTSSO and Password Synchronization
A long time ago, I commented that I was surprised that the Enterprise Single Sign-on
(ENTSSO) service that came with BizTalk Server 2004 (and Sharepoint Portal Server
2003) was so manual based: Users needed to keep their credentials and mappings updated
by hand, and even so using very awkward console applications.
Apparently, however, this was an scenario in mind but, I think, not implemented in
the V1 version of ENTSSO that shipped with BizTalk 2004. However, Password Synchronization
was added to ENTSSO for version 2.0, which came out with Host
Integration Server 2004!
As I understand it, the model around password synchronization is having adapters hooked
into ENTSSO that can notify it when either a password change has been made in the
Active Directory (or another system for which an adapter exists) so that the ENTSSO
can update the password stored in the Credentials Database (SSODB), and even forward
that notification to other systems so that full synchonization can be done. This feature
is included in the ENTSSO version of BizTalk Server 2006, by the way, though the stuff
necessary to support has to be installed explicitly.
The way I ran into this information was while researching the ISSOPSAdmin interface
of the ENTSSO API in the BizTalk documentation, which is the programmatic interface
you can use to configure and manage password synchronization adapters:
namespace Microsoft.EnterpriseSingleSignOn.Interop
{
[InterfaceType(0)]
[Guid(“C35718F9-C35C-4cd4-8978-2B4CE1792F1B”)]
[CoClass(typeof(SSOPSAdmin))]
public interface ISSOPSAdmin
{
void AssignAdapterToAdapterGroup(string adapterName, string adapterGroupName);
void AssignApplicationToAdapter(string applicationName, string adapterName);
void ClearDampingTable();
void ClearNotificationQueues(string adapterName);
void GetAdaptersForAdapterGroup(string adapterGroupName, out string[]
adapters);
void GetApplicationsForAdapter(string adapterName, out string[]
applications);
void RemoveAdapterFromAdapterGroup(string adapterName);
void RemoveApplicationFromAdapter(string applicationName);
void SetAdapterProperties(string adapterName, IPropertyBag properties);
}
}
You can find out more about ENTSSO and about the password synchronization mechanism
in this MSDN WebCast: Enteprise
Single Sign-on integrated with Microsoft BizTalk Server 2004 and Microsoft Host Integration
Server 2004. It has some good scenarios that show where ENTSSO can be used, as
well as some good demos.
Participation Oriented Web Sites Part II
Participation Oriented Web Sites Part II
In my previous post I introduced the conceptual model and logical model behind Participation Oriented Web Sites.
In this post I would like to concentrate in the mashup social arena by looking at examples and implications of how people reuse existing web content to support BYOC
Let us start by looking at the Participation Oriented Architecture of the MashUp EcoSystem
As you can see this architecture is built upon existing SOAs and data services and utilities.
Put it simply as of today the internet has over 8 trillion web sites, the majority of them are individualistic, disconnected, unique in some ways.
The idea behind mashups is that there is a shift in the decrease of the entropy of the web content towards a more structured sense, where entropy is a measure of disorder/chaos.
So what is the secret to aggregating the best content from the top few websites out of the trillions available already, not mentioning the near quadrillion images available aswell?
Take a look at the best web 2.0 sites of 2005
Take a look at the web 2.0 awards of 2005
Next take a look at the following web 2.0 site usage analysis
As you visit each site individually you will notice that there is a hierarchical trend of influence.
1st most inflential characteristic to any succesful web 2.0 site is the Users naturally.
2nd most infleuntial characteristic to any succesful web 2.0 site is the Data the users provide via BYOC.
The least most infleuntial characteristic to any succesful web 2.0 site is the Functionality engine that users leverage to BYOC.
So the advice to any web 2.0 bandwagoner, is plan very carefully.
Concentrate initially on your social market, and then focus on the data services that they can provide.
When you have the first two questions answered, then focus on the technologies and how best to leverage them.
So we now have an understanding of where the internet has been, and where it is now in the unstructured/structured arena and we have seen how people play the pivotal contributing part towards reuse of web content.
But is there going to be a point in time in the near future where human behaviours will be captured and automated?
Will the internet evolve towards a state of decrease in entropy on its own momentum(ie everthing will become more ordered and categorised) and will this be enabled by automating systems to provide the ultimate in UKM Universal Knowledege Management?
Will the future websites automate BYOC ie autoaggregate content themselves. ie Will there one day be a automashed web site? what do we call that Web 3.0?
Human Participation Oriented Web Sites ie Web 2.0 => Auto Participation Orinted Web Sites ie Web 3.0
hmmm 2010 maybe or even sooner?
It’s that time of night again where my brain goes into the twilight zone, so i’ll stop speculating here for now.
BTW When will dotnetjunkies sort out the user comment issue, that has been sabotaged by the comment spam filters?
Donny Mack long time no hear friend, so what’s the eta for the fix for this content management software bug?
Did I not ask this same question this time last year?
I would love to get back my user comments visible on the blog itself.
Currenty I can only get them by the email link.
Customer Reviewers Needed for Transacional Adapter Whitepaper
I have been working the past month on a whitepaper entitled, “Developing a Transactional BizTalk Adapter Using the Microsoft Base Adapter Classes“. It is almost ready for customer review and I would greatly appreciate any feedback before I finalize it. My asking for your comments is not to have you check the validity of my technical content or give me grammatical recommendations. Rather, I want comments like:
- “Please explain topic X to me more. I have never understood that and Microsoft has not done a good job of that”.
- “Can you add a section on Y? I feel it would really help other developers as I really struggled in this area since the documentation is lacking”.
That’s the kind of stuff I want from developers so Microsoft can better serve you and make adapter development easier for our customers. Let this paper be a means to share any lessons learned in your development process to benefit others like you.
If you are interested please click on the “EMAIL” link at the top of this page and send me your email address. I will most likely send you directly the version to review by this Thursday, April 6th. Again, it will not be in an edited or finished state at that point but you are reviewing for missing or expanded content anyway so it should not matter.
Thank you for your assistance on this!
Kind regards
Mike
BizTalk Server 2006 runs great under Microsoft Virtual Server 2005 R2!
Today, Microsoft announced that Virtual Server 2005 R2 can be downloaded free of charges. BizTalk Server 2006 works great under Virtual Server 2005 R2. When we were developing BizTalk Server 2006, I ran all tests under Virtual Server and never saw any…(read more)
How To Use BizTalk Messaging for Grouping, Sorting, and Debatching
In a past edition of The BizTalker, I talked about how easy it is to use custom XSLT inside the BizTalk Mapper to sort data. Sorting data inside the map can be doing using inline XSLT like this:
<xsl:for-each select=”Record”>
<xsl:sort select=”Id” data-type=”number” order=”ascending”/>
<xsl:element name=”Customers”>
<xsl:element name=”Id”><xsl:value-of select=”Id” /></xsl:element>
</xsl:element>
</xsl:for-each>
Sometimes sorting just is not enough. What if you also needed to group your data and break it up into single messages? This can be done several different ways including using xpath, .net components, or something external to BizTalk all together; with none of them really being ideal.
Another non-ideal way to accomplish grouping and debatching is to use pure BizTalk Messaging! Yes, pure messaging can group and debatch your data. Now, it is a little strange and takes a few hops through the message box, but it is a relatively simple solution to implement in a short amount of time.
So, how does it work?
Pass 1 will apply a map on the Receive Port to sort the data. The Send Port will apply another map to group the data.
The grouping map using two Functoids. One determines if the current record and the past record have the same Id. The second does the grouping.
The code in these Scripting Functoids looks like this:
True / False
string newNode = “”;
string pastValue = “X”;
public string MakeNewNode(string currentValue)
{
if (pastValue != currentValue)
{
pastValue = currentValue;
newNode = “true”;
}
else
newNode = “false”;
return newNode;
}
Grouping
<xsl:template name=”GroupByID”>
<xsl:param name=”param1″/>
<xsl:param name=”param2″/>
<xsl:if test=”$param1=’true'”>
<xsl:element name=”Group”>
<xsl:for-each select=”//Customers[Id=$param2]”>
<xsl:element name=”Customer”>
<xsl:element name=”Id”><xsl:value-of select=”Id”/></xsl:element>
<xsl:element name=”Name”><xsl:value-of select=”Name”/></xsl:element>
</xsl:element>
</xsl:for-each>
</xsl:element>
</xsl:if>
</xsl:template>
Pass 2 will apply an envelope to split the data inside the Receive Port and publish single messages to the message box.
Download the Sorting, Grouping, & Debatching Sample Code.
See the ReadMe.txt for set up information. Also note that I needed two projects for this solution. That is because the envelope schema needs to be in a separate project due to the way pipelines read the schema collections.
Professional BizTalk Server 2006
As promised here’s my draft Table of Contents for my upcoming Professional BizTalk Server 2006 book.
This TOC is subject to change and I haven’t gone into full detail in each of the chapters for obvious reasons but it’s hopefully enough to whet your appetite! As ever I’m really keen to get feedback and ideas to make sure it really hits the mark for you all.
The Technical Primer chapter is done and I’ve virtually finished the Business Activity Monitoring Chapter, but there’s still a lot of work to go!
Chapter 1: Technology Primer
In this chapter I cover XML Schema, XML Namespaces, XPath and Serializable classes, these are core development techniques that you need to understand to make the most of BizTalk Server development. This chapter does not cover the technologies in huge depth but gives you enough to understand why certain things happen the way they do when developing a BizTalk solutions (Schema is a constant source of developer headaches!)
Chapter 2: BizTalk Overview
BizTalk Server 2004 Unleashed is a good introduction book so we are not going to go there in this book – this should be a small intro to get the architecture of BizTalk straight in peoples minds (i.e. Messages don’t go straight from Adapters to Orchestrations)
Chapter 3: Adapters
In this chapter we will cover the BizTalk Adapter architecture and detail how it works under the covers (batches, interchanges, etc.) and then drill into each of “in-box” adapters detailing the nuances of each one, tips/tricks, registry keys, configuration, don’t use this in “this scenario”, etc.
Chapter 4: Pipelines
Pipelines are a very powerful and overlooked BizTalk development technique, we’ll discuss how they work, demonstrate some custom dissasembler/assembler components and outline scenarios where they have worked to great effect. Cover XML Splitting, BAM, Resumable interchanges, etc.
Chapter 5: Orchestration
We’ll cover the usual shapes in this chapter and how things work under the covers, a big focus on persistence points which are the key to good performance, transactions, calling .NET components (the right way), serializable classes, atomic scopes, orchestration patterns, XPath function, how to get around the common desire to use XMLNode and other non-serializable classes.
Chapter 6: Business Activity Monitoring
This chapter will drill heavily into BAM and will cover real-world scenarios where BAM has become the centre of the solution and rightly so, when to use TPE, when to use the Event Streams yourself via code and how you go about doing it. Continuation, Relationships, References and the other new to BizTalk 2006 features.
As part of the chapter we’ll build BAM into a fictional solution and demonstrate how you can build a health/support portal all driven from BAM data using SQL Server Reporting Services.
Chapter 7: Rules Engine
This chapter will cover the rules engine, how you can invoke it from code, tuning and how you might go about building a custom rules UI (which quite a few customers end up doing). If possible we’ll cover how the WF rules engine fits moving forward.
Chapter 8: Testing
In this chapter we’ll cover the whole scope of BizTalk Solution testing, we’ll cover how to use BizUnit to unit-test Orchestrations, integration with Team System, “Code Coverage” tooling. How to load test a BizTalk solution, counters to watch out for and how to diagnose issues. Demonstrate how LoadGen can be used to simulate load and avoid all those custom test harnesses we’ve all written!
Chapter 9: Performance and Scalability
In this chapter we’ll cover optimal datacenter design, host structure, disk storage, scale-out, how to optimise your design, adapter specific tuning and things to watch out for, tracking and throttling.
Chapter 10: Low Latency
Customers often wish to develop low latency solutions based on BizTalk which BizTalk out-of-the-box doesn’t support without custom tuning and adopting a number of approaches such as in-line web service calls. We’ll cover the possible approaches and details the pros and cons.
Chapter 11: Debugging and Diagnostics
We’ll cover a variety of developer techniques for diagnosing and debugging BizTalk solutions and how you can build your own logging solutions that aren’t performance bottlenecks (EntLib, BAM, etc.). We’ll also cover the most common issues that we see developers hitting to try and head them off before you hit them and waste time!
Chapter 12: Administration
We’ll cover a variety of Administration topics in this chapter, including how the new Application concept and Administration tool work and demonstrate how Microsoft Operations Manager (MOM) can be used in conjunction with BizTalk to greatly simplify BizTalk administration.
Chapter 13: Tips and Tricks (Working title)
This is a catch-all chapter right now and will probably get re-arranged, we’ll cover how SSO can be used to store config information, clearing BizTalk rigs down, creating a daily build process, etc.
Chapter 14: End to End Scenarios (E2E)
This chapter will cover the End to End Scenarios that have been shipped with BizTalk Server 2006 which are based on customer scenarios and are a great way to see how things should be done.
Chapter 15: BizTalk Accelerators
Hopefully I’ll get to cover the accelerators available today, what they can do and how they work.
Chapter 16: BizTalk Patterns
Cover some popular and useful BizTalk patterns, such as Resource Dispenser, Splitting Messages, FIFO, etc.
Chapter 17: Windows Workflow and BizTalk
We’ll cover what scenarios BizTalk and WF should be used for and some BizTalk futures.
Sorting, Grouping, and Debatching Using BizTalk Messaging
In a past edition of The BizTalker, I talked about how easy it is to use custom XSLT inside the BizTalk Mapper to sort data. Sorting data inside the map can be doing using inline XSLT like this:
<xsl:for-each select=”Record”>
<xsl:sort select=”Id” data-type=”number” order=”ascending”/>
<xsl:element name=”Customers”>
<xsl:element name=”Id”><xsl:value-of select=”Id” /></xsl:element>
</xsl:element>
</xsl:for-each>
Sometimes sorting just is not enough. What if you also needed to group your data and break it up into single messages? This can be done several different ways including using xpath, .net components, or something external to BizTalk all together; with none of them really being ideal.
Another non-ideal way to accomplish grouping and debatching is to use pure BizTalk Messaging! Yes, pure messaging can group and debatch your data. Now, it is a little strange and takes a few hops through the message box, but it is a relatively simple solution to implement in a short amount of time.
So, how does it work?
Pass 1 will apply a map on the Receive Port to sort the data. The Send Port will apply another map to group the data.
The grouping map using two Functoids. One determines if the current record and the past record have the same Id. The second does the grouping.
The code in these Scripting Functoids looks like this:
True / False
string newNode = “”;
string pastValue = “X”;
public string MakeNewNode(string currentValue)
{
if (pastValue != currentValue)
{
pastValue = currentValue;
newNode = “true”;
}
else
newNode = “false”;
return newNode;
}
Grouping