What it takes to be a good software developer

Last week I’ve got such a message from my blog:



—–Original Message—–
From: sandusergiu1984@…
Sent: Thursday, June 29, 2006 1:17 AM
To: Eldar Musayev
Subject: (Random Thoughts and Hints on Software Development) : Hello Eldar!
Importance: High


Can you please tell me what I should learn to be a good software developer? Thank you.


Hm-m-m, tough question. No, seriously. A lot of people proud of being asked about such fundamental thing would gladly go ahead and give a ton of well, you understood what I meant. Anyway, I did not want to do that. If I give an answer, I want to give a real answer, at least as good as I can. So, I postponed answering until I can find the time to give my thoughts on the subject. And, please, understand, these are just my thoughts; there is a lot of other very qualified and decent developers who may have a different opinion.


“Sometimes it’s almost as good as sex”


So, IMHO the first qualification for a good developer is that he or she should enjoy writing the code. As one of the famous hackers of the past said before “Sometimes it’s almost as good as sex”. Actually, I suspect that this is exactly the sentence that made him famous. But anyway, that’s what I am talking about. Let me explain why.


First, software development is a very demanding field. It’s not a 9 to 5 job by definition. Of course, you and your manager should try to keep it 9 to 5 (meaning not policing you to come at 9am, but rather trying to prevent you from staying late and sustaining yourself on company-bought pizzas while fixing the code around the clock), but – alas! – that’s not always  possible. You simply cannot do a good job if you don’t really love to write and handle the code. Otherwise, a couple of milestones or releases and you will hate the job. So, you simply have to enjoy this work to be able to do it long enough. Sure, some do that long enough without, but we are talking about good software developers, right?


Another reason is learning. This is a profession where you have to learn, learn, learn without any breaks forever. Software development defies scientific management of Taylor invented early in the XX century to destroy trade unions of skilled workers like arsenal workers. The very idea of the “scientific management” may be shortly expressed as “look at what those smartasses do, decompose it into simple steps, hire a bunch of cheap dumb guys, teach them the same steps, fire smartasses.” In late 50s “the father of American corporate management” Peter F. Drucker discovered a completely new category of workers whom he called “knowledge workers”. Knowledge workers defy the power of “scientific management”. The problem with knowledge workers is that they know what to do themselves. What’s worse, if you try to decompose what they do, then by the time you teach a bunch of dumb guys the same thing, they already do something different. They know what’s needed, you – the manager – don’t. That’s a challenge for both the manager and the worker. Software development is the prime example of knowledge workers. As a software developer, you have to learn all your life non-stop. There is no break, no finish line, that’s a race to the horizon. Try to learn all your life the things that you don’t like, and you may end up in a mental institution. And if not, you’ll hate your life for sure. You have to love writing code.


There is also a third reason. Did you ever met a person who thinks that he is a great writer, but writes a louse fiction. He annoys his friends and relatives, editors, and simply cannot get where he wants. He does not improve. Beside being  oblivious to his own shortcomings, such guy has a really hard time getting an informative honest feedback on his job. Friend and family lie to him, saying that he does a great job just to be nice. Editors simply reject his manuscripts. How can he learn? We, humans, need immediate feedback to learn effectively. That’s how our neural circuitry works and learns.


Fortunately for us – software developers – computers are much more critical and at the same time more willing readers than whom you can find in a literature field. Sometimes mere milliseconds are required to get your bugs in your face, when your code executes. This is not always pleasant, but it provides the necessary continuous feedback into the neural networks of our brains getting them into the shape to become able to create programs.


Computers are ruthless in exposing defects in your craftsmanship, they don’t care for your ego, they don’t let you save the face, they simply don’t care: they execute exactly what you ordered, and if you ordered a wrong thing, that’s what is going to happen and that’s what everybody will see. Actually, this is very good for us, this helps us to perfect and learn. “Everything that does not kill us, makes us stronger”, but unless you really love writing code, this kills your will to continue.


That’s why you have to enjoy writing the code. That’s why enjoying writing the code is the first requirement for being a good developer.


Of course, everything can be at extreme. A typical hacker enjoys writing code a little too much. Computer and coffee is all he wants in his life. Those guys can never become good software developers because beside the desire to write code, you also need a desire to write good code, a code that other people will use (and hacker doesn’t care for other people). Fortunately, most people pass this phase and come to their senses, often at the same age when raging hormones start to balance better and find other channels for exit.


Technical side


Of course, beside the attitude, there is simply a technical side, skills. You have to learn a set of basic knowledge and skills to start. However, this is not that hard question. Just look at computer science curriculum of a good university, that’s it. Those professors are experts in the area, they know what to teach a student. Don’t invent a bicycle, just look at what they teach.


Of course, college differs form a college and a university differs from a university. Some make a better job, some are not. I am far from endorsing any particular university or a college, but in my humble opinion University of Waterloo in Canada makes a superb job teaching their students computer science and software development. There are other universities in United States and Canada that make as great job teaching their students. Examples include MIT and University of Alberta. If you want a more complete and worldwide list, check ACM competition site here: http://icpc.baylor.edu/icpc/Finals/ And, of course, I am personally fan of my own Alma MaterSt.Petersburg State University in Russia, that ranked 6th in 2006, but was first on multiple occasions in previous years.


Anyway, once you like the field, learning is much easier.


More to it


Still there is more to it. I hardly can give a complete list but here are a few examples:


– ergonomics, usability, perception psychology Should this button be red or gray? How much space menu should take on the screen? How many items can you put into a scroll box? Why actionable control should be visually different from indicators? These are all not the questions about the technical side, but about human ability to handle it.


– What is the main language for the programmer? English (or whatever you use in the office). Communication skills are very important part of software development. Negotiation skills – depending on a team – may be of an utmost importance. Sometimes, either you can convince the team to go your way, or those who cannot write code but skillful at negotiations will have it their way, and you will be stuck with fixing impossible bugs and infinitely patching your code (those guys usually lack the design skills too).


– Psychology, empathy and investigation/interrogation skills. How do you find out what exactly the customers expect from your programs? Even if you have a real customer in front of you, people in sales, marketing and executive positions (who often represent a customer for you) are not cured with the brutality of computers that train you to tell truth, truth and only truth, because any lie (bug) immediately blows into your face, even if you believed it yourself. Actually, they are often trained by superiors and peers in rather opposite things. Sometimes you can meet a customer who can vividly articulate what he wants, and that’s your lucky day. Unfortunately, in many cases you have to dig out the truth, which is not easy. And that’s in the case when you have a customer in the first place.


– And what if you don’t have customers yet? What if you develop a new revolutionary product that nobody (including competition) have thought about before? Of course, you have a business concept by that time, but how do you fill the blanks? A serious help may be marketing – by the way, marketing guys, at least at Microsoft, are great: they are smart, know what they are want and have a sixth sense at what the customer needs. But your marketing guy cannot fill all the blanks too. What it boils down to is that you have to have tons of common sense to make the right decisions. And some marketing knowledge does not hurt too. Understanding how a product for X-ers should differ from a product for Y-ers may be essential. By the way, do you understand this difference?


– Networking, and I don’t mean TCP/IP. Friends, keeping connected, socialized is very important in our profession. I am not very good at that myself, but even so more than half of my jobs in US (including internal transfers at Microsoft) came from networking and referrals from my friends. If you really good at networking, you may need no resume at all. Beside that, it gives you more. How do you track what’s new and worth attention around? Trade magazines usually have no clue, and even when they have, they are still filled with tons of paid ads junk. Friends are your source of upcoming technology alerts, no-bull overview and introduction to new things.


– and that’s not all Feel free to add your thoughts at what is needed for a good software developer in the comments to this post.


So, I am wondering, if I managed to answer the question? I don’t know. I tried my best, if you can do better, do so and leave a link in the comments.

How To Add BizUnit Snippets Into Visual Studios 2005

How To Add BizUnit Snippets Into Visual Studios 2005

Recently I have been working a lot with BizUnit 2.0 (Available on GotDotNet).  This framework provides an easy and customizable way to unit test your end to end BizTalk solutions.  I’ll not get into details about BizUnit right now since this post is focused on Visual Studios 2005 Code Snippets.



Young Jun has developed a set of code snippets for BizUnit 2.0 for use with Visual Studios 2005.  Although not all the BizUnit tasks are included, it is a huge time saver when building unit test files. 



The only problem I had was how to install the snippets.  I found it a little tricky so here are the steps I went through.



Installation:


1.  Download the code snippets.


2.  Unzip them to a folder, in my case C:\BizUnit


3.  Open Visual Studios and locate the Code Snippets Manager under Tools. 


4.  If you do not see it listed, you will need to go to Customize under Tools, look under Commands, Tools, and move the command to the Tools menu.


 


 


5.  Change the Language to XML


 


 


6.  Select the Add button and select the folder with your snippet (in my case C:\BizUnit)


7.  Click OK. 


8.  You are all set.  Now when you open an Xml document inside Visual Studios you will see the BizTalk Snippets available




More to come on BizUnit in later posts. 

MOM 2005 Category added

As fate would have it, I’m going to be delving back into MOM 2005. I figure I may as well blog on it, so I’ve added it as another category.  I’ll be talking about MOM 2005 within the context of MOM monitoring BizTalk Server 2006.  There are of course, many other applications but I’ll start focused and try to be alot of use to a small group of people.


I’m going to be covering the management packs, initial setup, using the agents, configuring and responding to alerts, and also writing scripts for Mom to execute when an alert condition is detected.


I’m also taking requests, so if you have questions about MOM 2005 let me know and I’ll do my best to answer them.

Adding BizUnit Snippets to Visual Studios 2005

Adding BizUnit Snippets to Visual Studios 2005

Recently I have been working a lot with BizUnit 2.0 (Available on GotDotNet).  This framework provides an easy and customizable way to unit test your end to end BizTalk solutions.  I’ll not get into details about BizUnit right now since this post is focused on Visual Studios 2005 Code Snippets.


 


Young Jun has developed a set of code snippets for BizUnit 2.0 for use with Visual Studios 2005.  Although not all the BizUnit tasks are included, it is a huge time saver when building unit test files. 


 


The only problem I had was how to install the snippets.  I found it a little tricky so here are the steps I went through.


 


Installation:


1.  Download the code snippets.


2.  Unzip them to a folder, in my case C:\BizUnit


3.  Open Visual Studios and locate the Code Snippets Manager under Tools. 


4.  If you do not see it listed, you will need to go to Customize under Tools, look under Commands, Tools, and move the command to the Tools menu.


 


 


5.  Change the Language to XML


 


 


6.  Select the Add button and select the folder with your snippet (in my case C:\BizUnit)


7.  Click OK. 


8.  You are all set.  Now when you open an Xml document inside Visual Studios you will see the BizTalk Snippets available


 


 


More to come on BizUnit in later posts. 

SalesForce.com and Service Broker adapters available

AdapterWorx community keeps growing!!!

Today Two Connect decided to make available to the AdapterWorx community the trial versions of the Salesforce.com adapter and Service Broker 1.2 adapters for BizTalk Server 2006. Stay tune because we will have new surprises coming up on the next weeks.

So go ahead download the software, start playing with it and provide feedback.

Share this post: Email it! | bookmark it! | digg it! | reddit!

Extending BizTalk Mapping/Transformation operations into Sql Server

A staple in BizTalk 2004/2006 is to design and implement mapping/transformation operations using the BizTalk mapper. Mapping in BizTalk can be extended and improved by:


a) Using a rich set of Functoids that ship with BizTalk.
b) Using custom XSLT in a map.
c) Writing Custom Functoids that can be used in a map (link)
d) Writing JScript,C# or VB.Net code in a Scripting Functoid.


As mapping operations become more complex, Custom XSLT must often be used to perform the complete transformation or a portion of the transformation.
 
Unfortunately many developers do not know XSLT and in some cases do not have time to learn it. But, most developers do have Sql skills and are comfortable writing Sql Statements that join tables to produce a final result set.


This post will discuss a transformation of a source XML message to a destination XML message using a Sql Server 2005 stored procedure. The code for this example can be downloaded at the end of this entry.


A portion of the source XML message for the transformation is as below.
One or many Applicant nodes are submitted with the message.



A brief description of the <Applicant> node for the above message is as below:
1) <Applicant> nodes contain information such as <ApplicantId>,<FirstName>,<LastName>,<Sex>,<BirthCountryCode> etc.
2) <Applicant> nodes contain 0 to many <NickName> Nodes.
3) <Applicant> nodes contain <TestScore> nodes, that include the results of a test taken.
4) <Applicant> nodes contain <RelationInfoToScores> Nodes that contain IDRefs to relate the ApplicantId to the correct TestScores in the message. The red arrows in the above graphic illustrate the relation.


The Destination XML message for the completed transformation is as below. A separate <ConcatenatedApplicantInfo> node is created for each <Applicant> node in the source message. Information for each Applicant is concatenated with a padding of 25 spaces between items.



The below Sql Server 2005 stored procedure will complete the transformation of the source message to the destination message.


USE [TransformationHelper]


GO


/****** Object:  StoredProcedure [dbo].[Map_ApplicantHierarchical_To_ApplicantFlat]    Script Date: 07/13/2006 17:25:26 ******/


SET ANSI_NULLS ON


GO


SET QUOTED_IDENTIFIER ON


GO


 


CREATE Proc [dbo].[Map_ApplicantHierarchical_To_ApplicantFlat]


 


— Parameter to Accept AllApplicantInfo XML message


— from BizTalk.


@ApplicantXML xml


 


 


As


 


/*


 @Applicant Temp table for Applicant Header Info


 from AllApplicantInfo info XML message.


*/


 


DECLARE @Applicant Table(


    ApplicantId int null,


      FirstName varchar(20) null,


      LastName varchar(20) null,


      MiddleName varchar(20) null,


      Sex char(1) null,


      BirthCountryCode int null,


      BirthProvinceCode int null,


      BirthCountry varchar(20) null,


      BirthProvince varchar(20) null); 


 


/*


 Statement to shred Applicant Header Info from XML message


 into @Applicant Temp table


*/


WITH XMLNAMESPACES (‘http://Applicants’ as ns0)


Insert Into @Applicant


(ApplicantId,


FirstName,


LastName,


MiddleName,


Sex,


BirthCountryCode,


BirthProvinceCode)


Select ShreddedApplicantInfoXML.nCol.value(‘@ApplicantId’,‘int’),


ShreddedApplicantInfoXML.nCol.value((‘(FirstName)[1]’),‘varchar(20)’),


ShreddedApplicantInfoXML.nCol.value((‘(LastName)[1]’),‘varchar(20)’),


ShreddedApplicantInfoXML.nCol.value((‘(MiddleName)[1]’),‘varchar(20)’),


ShreddedApplicantInfoXML.nCol.value((‘(Sex)[1]’),‘char(1)’),


ShreddedApplicantInfoXML.nCol.value((‘(BirthCountryCode)[1]’),‘int’),


ShreddedApplicantInfoXML.nCol.value((‘(BirthProvinceCode)[1]’),‘int’)


From @ApplicantXML.nodes (‘/ns0:AllApplicants/Applicant/Information’) as ShreddedApplicantInfoXML(nCol)


Where ShreddedApplicantInfoXML.nCol.value((‘(FirstName)[1]’),‘varchar(20)’) is not null


 


 


/* In below update statement Join to another table to get Country Name:


 


       Join CountryCodes


            On Applicant.BirthCountryCode = CountryCodes.Id


 


      Replaces Database functoids.


      This increases performance as it cuts out the


      Database functoids that would make multiple round trips to


      the database


*/


 


/* Also in below update statement Call Managed Code to get Province name


 


   BirthProvince = dbo.GetProvince(Applicant.BirthCountryCode,Applicant.BirthProvinceCode)  


     


   This is like a scripting functoid or custom functoid in a BizTalk map.


*/


 


Update @Applicant


Set BirthCountry = CountryCodes.Name,


      BirthProvince = dbo.GetProvince(Applicant.BirthCountryCode,Applicant.BirthProvinceCode)


From @Applicant as Applicant


      Join CountryCodes


      On Applicant.BirthCountryCode = CountryCodes.Id


 


–Below is good for debugging purposes.


–Select * from @Applicant


 


 


/*


 @NickNames Temp table for Info


 from AllApplicantInfo info XML message.


*/


DECLARE @NickNames Table


(ApplicantId int null,


 [Name] varchar(20) null) ;


 


/*


 Statement to shred Info from XML message


 into @NickNames Temp table


*/


WITH XMLNAMESPACES (‘http://Applicants’ as ns0)


Insert Into @NickNames


Select NickNameXML.nCol.value(‘(../@ApplicantId)[1]’,‘int’),


NickNameXML.nCol.value((‘(Name)[1]’),‘varchar(20)’)


FROM @ApplicantXML.nodes (‘/ns0:AllApplicants/Applicant/Information/NickName’) as NickNameXML(nCol)


 


–Below is good for debugging purposes.


–Select * from @NickNames


 


/*


 @TestScores Temp table for Info


 from AllApplicantInfo info XML message.


*/


Declare @TestScores Table


(TestScoreId char(2) null,


Test varchar(20) null,


Score int null);


 


/*


 Statement to shred Info from XML message


 into @TestScores Temp table


*/


WITH XMLNAMESPACES (‘http://Applicants’ as ns0)


Insert Into @TestScores


Select


TestScoresXML.nCol.value(‘(@TestScoreId)’,‘char(2)’),


TestScoresXML.nCol.value((‘(Test)[1]’),‘varchar(20)’),


TestScoresXML.nCol.value((‘(Score)[1]’),‘int’)


FROM @ApplicantXML.nodes (‘/ns0:AllApplicants/Applicant/TestScores’) as TestScoresXML(nCol)


 


–Below is good for debugging purposes.


–Select * from @TestScores


 


 


/*


 @Relation Temp table for Info


 from AllApplicantInfo info XML message.


*/


Declare @Relation Table


(ApplicantId int null,


TestScoreId char(2) null);


 


/*


 Statement to shred Info from XML message


 into @TestScores Temp table


*/


WITH XMLNAMESPACES (‘http://Applicants’ as ns0)


Insert Into @Relation


Select


RelationXML.nCol.value(‘(@ApplicantId)’,‘int’),


RelationXML.nCol.value((‘@TestScoreId’),‘char(2)’)


FROM @ApplicantXML.nodes (‘/ns0:AllApplicants/Applicant/RelateInfoToScores’) as RelationXML(nCol)


 


–Below is good for debugging purposes.


–Select * from @Relation


 


Declare @FinalResults Table


(ApplicantID int not null,


ConcatenatedInfo varchar(8000))


 


Declare @ApplicantId int


Declare @ApplicantInfoText varchar(8000)


Declare @NickNamesText varchar(8000)


Declare @ScoresText varchar(8000)


 


— Below cursor will do the work of concatenating each Applicants


— info together into one node.


Declare cur_ApplicantRows Cursor Local Fast_Forward For


SELECT ApplicantId


FROM  @Applicant


     


OPEN cur_ApplicantRows


FETCH NEXT FROM cur_ApplicantRows INTO @ApplicantId


 


While @@Fetch_status = 0


      Begin


 


            Set @ApplicantInfoText =


            Set @NickNamesText =


            Set @ScoresText =


 


            Select @ApplicantInfoText = FirstName + space(25 len(FirstName)) +


                                                LastName + space(25 len(LastName)) +


                                                MiddleName +  space(25 len(MiddleName)) +


                                                Sex +  space(5 len(Sex)) +


                                                BirthCountry +  space(25 len(BirthCountry)) +


                                                BirthProvince +  space(25 len(BirthProvince))


            From @Applicant


            Where ApplicantId = @ApplicantId


 


            Select @NickNamesText = @NickNamesText + [Name] + space(25 len([Name]))


            From @NickNames


            Where ApplicantId = @ApplicantId


 


            Select @ScoresText = @ScoresText +


                                    TestScores.Test + space(25 len(TestScores.Test)) +


rtrim(str(TestScores.Score)) + space(25 len(rtrim(str(TestScores.Score))))


            From @TestScores as TestScores


                  Join @Relation as Relation


                  On TestScores.TestScoreId = Relation.TestScoreId     


                  and Relation.ApplicantId = @ApplicantId


 


            –Select @NickNamesText


            –Select @ApplicantInfoText


            –Select @ScoresText


           


            Insert Into @FinalResults


            values (@ApplicantId, @ApplicantInfoText + ‘ ‘ + @NickNamesText + ‘ ‘ + @ScoresText)


 


            FETCH NEXT FROM cur_ApplicantRows INTO  @ApplicantId


      End


 


— Below statement will generate the Final Flat Applicant XML Result.


— Return back the final result set


Select 1 as Tag,


      0 as Parent,


      ApplicantID as [ConcatenatedApplicantInfo!1!ApplicantID],


      ConcatenatedInfo as [ConcatenatedApplicantInfo!1!!cdata]        


From @FinalResults as ConcatenatedApplicantInfo


FOR XML EXPLICIT


 


 Discussion of Stored Procedure:


1) The AllApplicantInfo source XML Message is passed to the stored procedure via a parameter of type XML.


2) The AllApplicantInfo source XML Message is then shredded into an number of temporary relational tables:


@Applicant
@NickNames
@TestScores
@Relation


3) Sql Statements in the stored procedure are used to join the temp tables to produce the final <ConcatenatedApplicantInfo> node for each <Applicant> node in the source message.


4) The final destination message is returned using a Select statement via an XML Explicit Clause.


5) The Country Code table is directly available and can be joined to in a set operation in the stored procedure. This eliminates using Database Functoids in a BizTalk Map that would invoke multiple round trips to Sql Server.


6) A new feature of Sql Server 2005 is calling managed code. Managed code in Sql Server 2005 allows complex data manipulation that would be difficult to write using straight Transact SQL. In this example, calling managed code from stored procedure replaces the use of functoids in a BizTalk map. The below managed code was called from the stored procedure to determine the Province from a Province code.


 


using System;


using System.Data;


using System.Data.SqlClient;


using System.Data.SqlTypes;


using Microsoft.SqlServer.Server;


 


public partial class UserDefinedFunctions


{


   


    [Microsoft.SqlServer.Server.SqlFunction]


    public static SqlString GetProvince (SqlInt32 country, SqlInt32 province)


    {


       


        string result = “”;


        switch (country.Value)


        {


            case 1:


                switch (province.Value)


                {


                    case 1:


                        result = “Newfoundland”;


                        break;


                    case 2:


                        result = “PEI”;


                        break;


                    case 3:


                        result = “New Brunswick”;


                        break;


                    case 4:


                        result = “Nova Scotia”;


                        break;


                    case 5:


                        result = “Quebec”;


                        break;


                    case 6:


                        result = “Ontario”;


                        break;


                    case 7:


                        result = “Manitoba”;


                        break;


                    case 8:


                        result = “Saskatchewan”;


                        break;


                    case 9:


                        result = “Alberta”;


                        break;


                    case 10:


                        result = “British Columbia”;


                        break;


                    case 11:


                        result = “Yukon”;


                        break;


                    case 12:


                        result = “NWT”;


                        break;


                    default:


                         result = “Unknown”;


                         break;             


                }


            break; 


            default:


                result = “Unknown”;


                break;  


        }


        return result;


    }


};


 


As below, BizTalk can invoke the stored procedure via the Sql Adapter passing the source message as a parameter and then receiving the destination message as the response.



Sql 2005 has added many enhancements to aid in XML processing.
a) XQuery expressions
An example is below:


for $i in /InvoiceList/Invoice/Items/Item
return
{$i/@Quantity}
{string($i/@Product)}


b) Xml Auto -> Nesting XML auto clauses
c) XML datatype
d) Indexing xml data
e) A subset of the FLWOR syntax
f) Validating XML against a Schema.
g) more..



If you are interested in some free online virtual labs that explore the new Sql 2005 XML capabilities, please click on the below links: 


 


SQL Server%u2122 2005: XML Capabilities (TechNet) Virtual Lab
And:
Using Xquery with SQL Server 2005 XML Data (TechNet) Virtual Lab

Finally:


The not so good things about this method:

1) You have to call Sql server from BizTalk. When using BizTalk maps, the entire
transformation is localized to the BizTalk host process.


The good things about this method:

1) Simplified debugging of the transformation. For example, in the T-SQL you can place in print statements, select statements that return intermediate result sets, debug a stored procedure in Visual Studio, use the Sql Profiler etc.
2) In this example only a stored procedure needs to be deployed to a Sql Server database.
3) In this example, no data is persisted to permanent Sql Server tables. Only temp tables are utilized in the stored procedure.
 
Use this method:


1) As an alternative to writing custom XSLT.
2) If you are comfortable writing Sql Server Stored Procedures.
3) To replace database functoids in a map that incur multiple round trips to a database.
4) To perform transformations on larger messages

Some notes:


1) More than one XML message can be passed to a stored procedure.
2) Transformation operations can be split between the BizTalk maps and stored procedures. For example BizTalk maps can carry out a portion of the transformation and then pass the remainder of the transformation to the stored procedures.
3) For Sql 2000, OpenXML statements can be used to shred XML nodes into relational tables.

Conclusion:


Use the new XML enhancements along with the managed code in Sql 2005 to simplify and to improve the performance of complex BizTalk mapping operations.


Download the sample discussed above HERE. Read the ReadMe.Txt before installing and running.


 

Extending BizTalk Mapping/Transformation operations into Sql Server

A staple in BizTalk 2004/2006 is to design and implement mapping/transformation operations using the BizTalk mapper. Mapping in BizTalk can be extended and improved by:


a) Using a rich set of Functoids that ship with BizTalk.
b) Using custom XSLT in a map.
c) Writing Custom Functoids that can be used in a map (link)
d) Writing JScript,C# or VB.Net code in a Scripting Functoid.


As mapping operations become more complex, Custom XSLT must often be used to perform the complete transformation or a portion of the transformation.
 
Unfortunately many developers do not know XSLT and in some cases do not have time to learn it. But, most developers do have Sql skills and are comfortable writing Sql Statements that join tables to produce a final result set.


This post will discuss a transformation of a source XML message to a destination XML message using a Sql Server 2005 stored procedure. The code for this example can be downloaded at the end of this entry.


A portion of the source XML message for the transformation is as below.
One or many Applicant nodes are submitted with the message.



A brief description of the <Applicant> node for the above message is as below:
1) <Applicant> nodes contain information such as <ApplicantId>,<FirstName>,<LastName>,<Sex>,<BirthCountryCode> etc.
2) <Applicant> nodes contain 0 to many <NickName> Nodes.
3) <Applicant> nodes contain <TestScore> nodes, that include the results of a test taken.
4) <Applicant> nodes contain <RelationInfoToScores> Nodes that contain IDRefs to relate the ApplicantId to the correct TestScores in the message. The red arrows in the above graphic illustrate the relation.


The Destination XML message for the completed transformation is as below. A separate <ConcatenatedApplicantInfo> node is created for each <Applicant> node in the source message. Information for each Applicant is concatenated with a padding of 25 spaces between items.



The below Sql Server 2005 stored procedure will complete the transformation of the source message to the destination message.


USE [TransformationHelper]


GO


/****** Object:  StoredProcedure [dbo].[Map_ApplicantHierarchical_To_ApplicantFlat]    Script Date: 07/13/2006 17:25:26 ******/


SET ANSI_NULLS ON


GO


SET QUOTED_IDENTIFIER ON


GO


 


CREATE Proc [dbo].[Map_ApplicantHierarchical_To_ApplicantFlat]


 


— Parameter to Accept AllApplicantInfo XML message


— from BizTalk.


@ApplicantXML xml


 


 


As


 


/*


 @Applicant Temp table for Applicant Header Info


 from AllApplicantInfo info XML message.


*/


 


DECLARE @Applicant Table(


    ApplicantId int null,


      FirstName varchar(20) null,


      LastName varchar(20) null,


      MiddleName varchar(20) null,


      Sex char(1) null,


      BirthCountryCode int null,


      BirthProvinceCode int null,


      BirthCountry varchar(20) null,


      BirthProvince varchar(20) null); 


 


/*


 Statement to shred Applicant Header Info from XML message


 into @Applicant Temp table


*/


WITH XMLNAMESPACES (‘http://Applicants’ as ns0)


Insert Into @Applicant


(ApplicantId,


FirstName,


LastName,


MiddleName,


Sex,


BirthCountryCode,


BirthProvinceCode)


Select ShreddedApplicantInfoXML.nCol.value(‘@ApplicantId’,‘int’),


ShreddedApplicantInfoXML.nCol.value((‘(FirstName)[1]’),‘varchar(20)’),


ShreddedApplicantInfoXML.nCol.value((‘(LastName)[1]’),‘varchar(20)’),


ShreddedApplicantInfoXML.nCol.value((‘(MiddleName)[1]’),‘varchar(20)’),


ShreddedApplicantInfoXML.nCol.value((‘(Sex)[1]’),‘char(1)’),


ShreddedApplicantInfoXML.nCol.value((‘(BirthCountryCode)[1]’),‘int’),


ShreddedApplicantInfoXML.nCol.value((‘(BirthProvinceCode)[1]’),‘int’)


From @ApplicantXML.nodes (‘/ns0:AllApplicants/Applicant/Information’) as ShreddedApplicantInfoXML(nCol)


Where ShreddedApplicantInfoXML.nCol.value((‘(FirstName)[1]’),‘varchar(20)’) is not null


 


 


/* In below update statement Join to another table to get Country Name:


 


       Join CountryCodes


            On Applicant.BirthCountryCode = CountryCodes.Id


 


      Replaces Database functoids.


      This increases performance as it cuts out the


      Database functoids that would make multiple round trips to


      the database


*/


 


/* Also in below update statement Call Managed Code to get Province name


 


   BirthProvince = dbo.GetProvince(Applicant.BirthCountryCode,Applicant.BirthProvinceCode)  


     


   This is like a scripting functoid or custom functoid in a BizTalk map.


*/


 


Update @Applicant


Set BirthCountry = CountryCodes.Name,


      BirthProvince = dbo.GetProvince(Applicant.BirthCountryCode,Applicant.BirthProvinceCode)


From @Applicant as Applicant


      Join CountryCodes


      On Applicant.BirthCountryCode = CountryCodes.Id


 


–Below is good for debugging purposes.


–Select * from @Applicant


 


 


/*


 @NickNames Temp table for Info


 from AllApplicantInfo info XML message.


*/


DECLARE @NickNames Table


(ApplicantId int null,


 [Name] varchar(20) null) ;


 


/*


 Statement to shred Info from XML message


 into @NickNames Temp table


*/


WITH XMLNAMESPACES (‘http://Applicants’ as ns0)


Insert Into @NickNames


Select NickNameXML.nCol.value(‘(../@ApplicantId)[1]’,‘int’),


NickNameXML.nCol.value((‘(Name)[1]’),‘varchar(20)’)


FROM @ApplicantXML.nodes (‘/ns0:AllApplicants/Applicant/Information/NickName’) as NickNameXML(nCol)


 


–Below is good for debugging purposes.


–Select * from @NickNames


 


/*


 @TestScores Temp table for Info


 from AllApplicantInfo info XML message.


*/


Declare @TestScores Table


(TestScoreId char(2) null,


Test varchar(20) null,


Score int null);


 


/*


 Statement to shred Info from XML message


 into @TestScores Temp table


*/


WITH XMLNAMESPACES (‘http://Applicants’ as ns0)


Insert Into @TestScores


Select


TestScoresXML.nCol.value(‘(@TestScoreId)’,‘char(2)’),


TestScoresXML.nCol.value((‘(Test)[1]’),‘varchar(20)’),


TestScoresXML.nCol.value((‘(Score)[1]’),‘int’)


FROM @ApplicantXML.nodes (‘/ns0:AllApplicants/Applicant/TestScores’) as TestScoresXML(nCol)


 


–Below is good for debugging purposes.


–Select * from @TestScores


 


 


/*


 @Relation Temp table for Info


 from AllApplicantInfo info XML message.


*/


Declare @Relation Table


(ApplicantId int null,


TestScoreId char(2) null);


 


/*


 Statement to shred Info from XML message


 into @TestScores Temp table


*/


WITH XMLNAMESPACES (‘http://Applicants’ as ns0)


Insert Into @Relation


Select


RelationXML.nCol.value(‘(@ApplicantId)’,‘int’),


RelationXML.nCol.value((‘@TestScoreId’),‘char(2)’)


FROM @ApplicantXML.nodes (‘/ns0:AllApplicants/Applicant/RelateInfoToScores’) as RelationXML(nCol)


 


–Below is good for debugging purposes.


–Select * from @Relation


 


Declare @FinalResults Table


(ApplicantID int not null,


ConcatenatedInfo varchar(8000))


 


Declare @ApplicantId int


Declare @ApplicantInfoText varchar(8000)


Declare @NickNamesText varchar(8000)


Declare @ScoresText varchar(8000)


 


— Below cursor will do the work of concatenating each Applicants


— info together into one node.


Declare cur_ApplicantRows Cursor Local Fast_Forward For


SELECT ApplicantId


FROM  @Applicant


     


OPEN cur_ApplicantRows


FETCH NEXT FROM cur_ApplicantRows INTO @ApplicantId


 


While @@Fetch_status = 0


      Begin


 


            Set @ApplicantInfoText =


            Set @NickNamesText =


            Set @ScoresText =


 


            Select @ApplicantInfoText = FirstName + space(25 len(FirstName)) +


                                                LastName + space(25 len(LastName)) +


                                                MiddleName +  space(25 len(MiddleName)) +


                                                Sex +  space(5 len(Sex)) +


                                                BirthCountry +  space(25 len(BirthCountry)) +


                                                BirthProvince +  space(25 len(BirthProvince))


            From @Applicant


            Where ApplicantId = @ApplicantId


 


            Select @NickNamesText = @NickNamesText + [Name] + space(25 len([Name]))


            From @NickNames


            Where ApplicantId = @ApplicantId


 


            Select @ScoresText = @ScoresText +


                                    TestScores.Test + space(25 len(TestScores.Test)) +


rtrim(str(TestScores.Score)) + space(25 len(rtrim(str(TestScores.Score))))


            From @TestScores as TestScores


                  Join @Relation as Relation


                  On TestScores.TestScoreId = Relation.TestScoreId     


                  and Relation.ApplicantId = @ApplicantId


 


            –Select @NickNamesText


            –Select @ApplicantInfoText


            –Select @ScoresText


           


            Insert Into @FinalResults


            values (@ApplicantId, @ApplicantInfoText + ‘ ‘ + @NickNamesText + ‘ ‘ + @ScoresText)


 


            FETCH NEXT FROM cur_ApplicantRows INTO  @ApplicantId


      End


 


— Below statement will generate the Final Flat Applicant XML Result.


— Return back the final result set


Select 1 as Tag,


      0 as Parent,


      ApplicantID as [ConcatenatedApplicantInfo!1!ApplicantID],


      ConcatenatedInfo as [ConcatenatedApplicantInfo!1!!cdata]        


From @FinalResults as ConcatenatedApplicantInfo


FOR XML EXPLICIT


 


 Discussion of Stored Procedure:



1) The AllApplicantInfo source XML Message is passed to the stored procedure via a parameter of type XML.


2) The AllApplicantInfo source XML Message is then shredded into an number of temporary relational tables:


@Applicant
@NickNames
@TestScores
@Relation


3) Sql Statements in the stored procedure are used to join the temp tables to produce the final <ConcatenatedApplicantInfo> node for each <Applicant> node in the source message.


4) The final destination message is returned using a Select statement via an XML Explicit Clause.


5) The Country Code table is directly available and can be joined to in a set operation in the stored procedure. This eliminates using Database Functoids in a BizTalk Map that would invoke multiple round trips to Sql Server.


6) A new feature of Sql Server 2005 is calling managed code. Managed code in Sql Server 2005 allows complex data manipulation that would be difficult to write using straight Transact SQL. In this example, calling managed code from stored procedure replaces the use of functoids in a BizTalk map. The below managed code was called from the stored procedure to determine the Province from a Province code.



using System;


using System.Data;


using System.Data.SqlClient;


using System.Data.SqlTypes;


using Microsoft.SqlServer.Server;


 


public partial class UserDefinedFunctions


{


   


    [Microsoft.SqlServer.Server.SqlFunction]


    public static SqlString GetProvince (SqlInt32 country, SqlInt32 province)


    {


       


        string result = “”;


        switch (country.Value)


        {


            case 1:


                switch (province.Value)


                {


                    case 1:


                        result = “Newfoundland”;


                        break;


                    case 2:


                        result = “PEI”;


                        break;


                    case 3:


                        result = “New Brunswick”;


                        break;


                    case 4:


                        result = “Nova Scotia”;


                        break;


                    case 5:


                        result = “Quebec”;


                        break;


                    case 6:


                        result = “Ontario”;


                        break;


                    case 7:


                        result = “Manitoba”;


                        break;


                    case 8:


                        result = “Saskatchewan”;


                        break;


                    case 9:


                        result = “Alberta”;


                        break;


                    case 10:


                        result = “British Columbia”;


                        break;


                    case 11:


                        result = “Yukon”;


                        break;


                    case 12:


                        result = “NWT”;


                        break;


                    default:


                         result = “Unknown”;


                         break;             


                }


            break; 


            default:


                result = “Unknown”;


                break;  


        }


        return result;


    }


};


 



As below, BizTalk can invoke the stored procedure via the Sql Adapter passing the source message as a parameter and then receiving the destination message as the response.



Sql 2005 has added many enhancements to aid in XML processing.
a) XQuery expressions
An example is below:


for $i in /InvoiceList/Invoice/Items/Item
return
{$i/@Quantity}
{string($i/@Product)}


b) Xml Auto -> Nesting XML auto clauses
c) XML datatype
d) Indexing xml data
e) A subset of the FLWOR syntax
f) Validating XML against a Schema.
g) more..



If you are interested in some free online virtual labs that explore the new Sql 2005 XML capabilities, please click on the below links: 



SQL Server™ 2005: XML Capabilities (TechNet) Virtual Lab
And:
Using Xquery with SQL Server 2005 XML Data (TechNet) Virtual Lab

Finally:


The not so good things about this method:

1) You have to call Sql server from BizTalk. When using BizTalk maps, the entire
transformation is localized to the BizTalk host process.


The good things about this method:

1) Simplified debugging of the transformation. For example, in the T-SQL you can place in print statements, select statements that return intermediate result sets, debug a stored procedure in Visual Studio, use the Sql Profiler etc.
2) In this example only a stored procedure needs to be deployed to a Sql Server database.
3) In this example, no data is persisted to permanent Sql Server tables. Only temp tables are utilized in the stored procedure.
 
Use this method:


1) As an alternative to writing custom XSLT.
2) If you are comfortable writing Sql Server Stored Procedures.
3) To replace database functoids in a map that incur multiple round trips to a database.
4) To perform transformations on larger messages

Some notes:


1) More than one XML message can be passed to a stored procedure.
2) Transformation operations can be split between the BizTalk maps and stored procedures. For example BizTalk maps can carry out a portion of the transformation and then pass the remainder of the transformation to the stored procedures.
3) For Sql 2000, OpenXML statements can be used to shred XML nodes into relational tables.

Conclusion:



Use the new XML enhancements along with the managed code in Sql 2005 to simplify and to improve the performance of complex BizTalk mapping operations.


Download the sample discussed above HERE. Read the ReadMe.Txt before installing and running.


Passing arguments from the command line to downstream functions in Powershell

$args is a special variable, an array of all arguments passed to a function on the command line. But, $args is always treated as an array in PowerShell. And It may cause an interesting side effect when passing $args to a downstream function.
Suppose we define a function as follows:
function t1
{
 “args : $args, $($args.count)”
}

If you call the function in the command line as follows:
PS : Yjhong>C:\t1 r t g
The result will be
args: r t, 2
But, if you call the function in a function as follows:
function t2
{
 t1 $args # my intention is passing the command line arguments that t2 received to the nested function, t1.
}

And, if you run the t2 as follows, you will get a different result as follows:
PS : Yjhong>C:\t2 r t
args: System.Object[], 1
It means that when the t1 function is called in a function, t1 will receive one System.Object[] object consisting of two elements, ’r’, and ’t’

MS BizTalk Server 2006 certification passed

I passed new Microsoft Certified Technology Specialist (MCTS) exam 70-235: Developing Business Process and Integration Solutions using Microsoft® BizTalk® Server 2006. Fresh experience with BizTalk 2006 helped a lot as well as reading documentation, blogs. Interestingly, this exam stands aside of new Microsoft Certified Professional Developer (MCPD) path. It would make sense to include it as an elective option earning credit towards MCPD: Enterprise Application Developer. This exam with the combination of:



  • Exam 70–549: PRO: Designing and Developing Enterprise Applications by Using the Microsoft .NET Framework


  • Exam 70–536: TS: Microsoft .NET Framework 2.0 – Application Development Foundation


  • Exam 70–528: TS: Microsoft .NET Framework 2.0 – Web-Based Client Development


  • Exam 70–529: TS: Microsoft .NET Framework 2.0 – Distributed Application Development

would give well rounded Enterprise Application Developer skill set with emphasis on integration and business automation given that you’ve got to know all that stuff anyways to use BizTalk efficiently. I don’t know what Microsoft planned this certification for but without skills listed above one can’t develop a solid “Business Process and Integration Solutions”. So it does look like an orphan certification right now.