by community-syndication | Feb 18, 2009 | BizTalk Community Blogs via Syndication
Next month, (10, 11 & 12 of March) the Belgian TechDays will be held in Antwerp. Together with my pal Lieven Iliano, I’ll present the SharePoint Pre-conference day on March, 10th. So what is this SharePoint Pre-conference all about? Well were going to talk about .NET 3.5 in SharePoint 2007, think WCF, Silverlight, Linq and WF applied in SharePoint 2007 sites. I’m very excited and I’m sure Lieven and I will be able to entertain you 5 (five!) sessions in a row. For the full agenda, check the TechDays site and I hope to see you there!
by community-syndication | Feb 18, 2009 | BizTalk Community Blogs via Syndication
While creating the EDI Warehouse, for EDI files that were large (10mb transactions), I was seeing 1 of two things with the SQL Adapter:
- Out of memory exceptions
- Taking hours to load data with high sql server usage
In opening a ticket with Microsoft to discover that the SQL adapter casts the message to a string in a certain part of the code, I knew that I could not use it for consistent high volume usage.
I created a Bulk XML Load adapter that accomplishes in 1 minute that previously took 3 hours to load.
This is the interface:
Some of the features are:
If the size of the message is small enough (as defined in the send handler properties) it takes the message and directly loads the data into the database. You can override the direct Stream to Database setting by changing the File Load boolean to True or if the message size is greater than the send handler property (by default set at 10mb)
Enables all of the features of Bulk XML load 4.0
If an error occurs during the load, will create the xml data it is trying to load, the error.xml and a Visual Basis Script to re run the process for further testing (as long as the Debug Flag is set to True).
In the Event Log you see this entry:
This allow to manually run the bulk load to determine the error.
by community-syndication | Feb 18, 2009 | BizTalk Community Blogs via Syndication
So I’ve been speak a lot about version 2.0 of the ESB Guidance and I thought I’d better get some of that content up to the blog. I’ll be posting a series of demos that I’ve built that show how to use the new features and tools in V2. In order to keep the blog entries shorter and easier to get out quickly, I’m just going filming me doing the demo and then I’ll post the AVI file for you to watch. (i.e. pictures and videos being worth a thousand words or so) That saves me from having to take screen shots and to try and verbally explain what I did.
In part one, I’ll be showing how to implement a basic routing scenario. I have a basic web service and test client setup. I want the ESB to route a message from the test client to the web service. I also want the ESB to determine where the web service’s endpoint is by calling out to a UDDI server.
What I’ll be showing is how this can be implemented using the basic routing capabilities of the ESB 2.0. I’ll take you through the process of setting this up from start to finish. We setup the BizTalk components, define the Itinerary (which defines the process flow within the ESB) and we’ll test the end to end process. If any of those terms were new for you (i.e. itineraries, biztalk, uddi etc), then I would recommend you take a bit of time and review the basic concepts of the ESB guidance.
Those of you that worked with the ESB guidance 1.0 will already know how to implement the scenario that I just outlined. However one of the great new features of v2.0 is the the new visual itinerary design tool. It is an incredibly welcomed addition to the guidance as we no longer need to write the raw XML that itineraries are made of.
Due to the size of the file, I’ve had to zip it and split it into 7 files. Be sure to download all of them before and then extra them into the AVI.
File 1
File 2
File 3
File 4
File 5
File 6
File 7
Hopefully this helps those of you who are new to the guidance and helps you to get using it more quickly.
Cheers and Keep on BizTalking…
Peter
by community-syndication | Feb 18, 2009 | BizTalk Community Blogs via Syndication
My first session this year at DevDays09 will be ARC206, with Tiago Pascoal. We’ll be discussing mostly architecture anti-patterns, with loads of real life examples. It’s today at 14:30 on room A03, at Instituto Superior T%u00e9cnico, this year’s venue. Overview to follow later.
by community-syndication | Feb 17, 2009 | BizTalk Community Blogs via Syndication
I just came across a great demonstration of a fantastic new piece of kit.
Siftables they’re called (shot this month) that ‘interact’ and play
with each other on a desk, table what ever.
CubeWorld guys was something I’d had my eye on for a while…..
Siftables takes it far beyond the next level, check it out for yourself.
http://www.ted.com/index.php/talks/david_merrill_demos_siftables_the_smart_blocks.html
by Richard | Feb 17, 2009 | BizTalk Community Blogs via Syndication
During my years of BizTalk development I’ve been warned of a couple of scenarios that the product wouldn’t handle very well. Yesterday another of those scenarios turned out to kind of false and, if done right, not really a problem at all.
The scenario I’m talking about is a batch import of data to SQL Server using the SQL adapter. In my case the data is received as a flat text file containing a large number of rows. These rows should the be places inside a database table as one table-row per row in the flat file.
The common way of dealing with batch incoming data like this is to split (aka disassemble) it in the receive port using the Flat File Disassembler pipeline component (for a good example – look here). Disassembling the data when receiving it is usually good practice to avoid running into OutOfMemoryException when dealing with big messages.
Sometimes the requirements also forces one into reading each row to a separate message to be able to route and handle each messages in a unique way depending of it’s content. If that so – this is a not a good post for you. In this post I’ll discuss the scenario were all the data just needs to go as fast as possible from the text file into a single database table. No orchestration or anything, just a simple batch import.
So, what’s The problem with the batch import scenario?
When I implemented similar batch import scenarios in the past I tried to practice good practice and split the data into separate files that I then filtered to the SQL adapter send port, one by one.
- The received flat file files has been split into thousands of small little message that one by one are sent to the SQL adapter send port.
- The SQL adapter then parses each message into a SQL script that executes a store procedure and the message is finally inserted to the target database.
“So what’s the problem?” you then ask? It’s slow! It’s very slow! Each message gets stored a couple of times in the BizTalk database and each message is sent inside it’s own DTC transaction against the target database. And all this adds up …
And after reading this this interview by Richard Seroter with Alan Smith I also felt I was the only one having the problem either …
There are quite a few people asking about using BizTalk for heavy database integration, taking flat files, inserting the data in databases and processing it. SQL Server Integration Services (SSIS) does a much better job than BizTalk at this, and is worth looking at in those scenarios. BizTalk still has its uses for that type of work, but is limited be performance. The worst case I saw was a client who had a daily batch that took 36 hours to process using BizTalk, and about 15 minutes using SSIS. On another project I worked on they had used BizTalk for all the message based integration, and SSIS for the data batching, and it worked really well.
>
>
Note: As I’ll described later in this post my import scenario went from something like 3-4 hours to 2 minutes (importing 10 MB). Alan talks about a 36 hours (!) import. I don’t know anything more about the scenario he mentions and it might not even be solved using the technique discussed below. Hopefully Alan might comment on the post and give us more details. 😉
How can we get better performing imports using BizTalk?
As the import scenario we described doesn’t involve any orchestration but is a pure messaging scenario and we do all the transformation on the ports we don’t really have to worry about OutOfMemoeyExceptions even though the message is quite big.
**Large message transformation.** In previous versions of BizTalk Server, mapping of documents always occurred in-memory. While in-memory mapping provides the best performance, it can quickly consume resources when large documents are mapped. In BizTalk Server 2006, large messages will be mapped by the new large message transformation engine, which buffers message data to the file system, keeping the memory consumption flat. ([Source](http://www.microsoft.com/technet/prodtechnol/biztalk/2006/evaluate/overview/default.mspx))
>
>
Another reason for splitting the message was for it to work with the SQL adapter. When setting up the SQL adapter to work with a store procedure the adapter expects a message that looks something like the below.
<ns0:ImportDataSP_Request xmlns:ns0="http://FastSqlServerBatchImport.Schemas.SQL_ImportDataSP">
<ImportData Name="Name 1" Value="1"></ImportData>
</ns0:ImportDataSP_Request>
This tells us that the store procedure called is “ImportData” with “Name 1” as the value for the “Name” parameter and “1” as the value for the parameter called “Value” in the stored procedure. So each little separate message would get mapped on the send port into something like this.
What I however didn’t know until I read this post was that the message I send to the SQL adapter port just as well could look like this!
<ns0:ImportDataSP_Request xmlns:ns0="http://FastSqlServerBatchImport.Schemas.SQL_ImportDataSP">
<!-- TWO rows!!! -->
<ImportData Name="Name 1" Value="1"></ImportData>
<ImportData Name="Name 2" Value="2"></ImportData>
</ns0:ImportDataSP_Request>
So basically we can have as many store procedure calls as we want in one single file that then can send to the SQL adapter send port!
Eureka!) __This means that we don’t have to split the incoming file! We can keep it as one big single file and just transform it to a huge file containing separate nodes that we send to the SQL Adapter send port! The SQL adapter will then parse this into separate store procedure calls for us.
__
Is it really any faster?
As the technique above drastically reduced the amount of database communication needed I knew it’d be much faster. Some initial testing shows that an import of a file containing somewhere around 55 000 rows (about 10 MB) into our article database went from 3-4 hours to under two minutes!
See for yourself!
In this sample solution I have a text file containing 2 600 rows. I’ve then created two separate projects in a solutions. One that splits the messages into separate messages (called “SlowImport”) and one that just transforms it and send it as one message to the send port (called “FastImport”). One takes 1:50 minutes and 2 seconds on my development machine … I won’t tell you which one is the faster one …
Test it for yourself and let me know what you think.
by community-syndication | Feb 17, 2009 | BizTalk Community Blogs via Syndication
During my years of BizTalk development I’ve been warned of a couple of scenarios that the product wouldn’t handle very well. Yesterday another of those scenarios turned out to kind of false and, if done right, not really a problem at all.
The scenario I’m talking about is a batch import of data to SQL […]
by community-syndication | Feb 17, 2009 | BizTalk Community Blogs via Syndication
This post is just to call out some work that Bram Veldhoenhas done in linking the Biztalk Software Factoryto the venerable Deployment Framework. One thing to note is that the integration was done to the NAnt version of the framework prior to the latest MSBuild version of the framework (5.x). I guess linking into the […]
by community-syndication | Feb 17, 2009 | BizTalk Community Blogs via Syndication
Originally posted by Nick Heppleston at: http://www.modhul.com/2009/02/17/gotcha-when-exporting-adapter-config-in-binding-files/
This one caught me out today – hope it might help others in the future…
If you export Bindings to get the lovely syntax for the ReceiveLocationTransportTypeData element, make sure that all of the properties in the adapter configuration dialog that you are interested in are populated *before* you do […]
by community-syndication | Feb 17, 2009 | BizTalk Community Blogs via Syndication
I’ll be speaking at the ASP.NET SIG of the San Diego .NET User Group tonight. Abstract is below. Meeting starts at 6:00 at the Microsoft La Jolla office.
This is basically the same presentation Allan Naim and I did recently at the SOA BP conference in Redmond and at an internal Microsoft conference.
If the demo gods are willing (as I expect them to be), I’m planning to do my shinny new demo that ties together work I’ve been doing around Dublin, Azure, WF, WCF, BizTalk 2009 and ESB Guidance 2.0.
Session Abstract:
With so many technologies to choose from, Solution Architects often find themselves unsure of what the right technologies are to address their solution needs. With the recent announcement of Microsoft’s new “Dublin” application server capabilities, architects and developers have more choices than ever and the challenge is selecting and integrating the right technologies in order to ensure that IT is agile, and that business needs are met. The goal of this session is to provide you with a holistic view of Microsoft SOA technologies focusing on building the right architecture to address specific SOA capabilities around integration, messaging, security and management using technologies such as BizTalk Server 2009, ESB Guidance v2, Managed Services Engine, .NET 4.0 and Dublin. We will look at the value each piece adds to the overall picture, and how they can be integrated in a holistic solution.
Technorati Tags: SOA,ESB,Dublin,BizTalk,.NET,WF,WCF
