Flat File Debatching in BTS 2006 – Part II

In earlier part I, we checked out the Customers Record and child elements creation.Lets see how we can create Items and Item fields in the purchaseorder.xsd using Flat File schema wizard.
Lets continue with the wizard:
1. Now we can see the schema with fields under Customers record as below:
                                        
2. Now select Items and click Next.
3. […]

Performance Testing at MS DAY 1

The morning was spent completing the biztalk build and config on the dev boxes being used. The sourcesafe repository was copied over and by mid morning we were up and running with builds on the dev boxes.

During the afternoon we tested each of the 3 designs against a standard “out of the box” config – this ran on a single 4-way biztalk box and a super fast sql box. As a recap, the following are the designs


Async Design – designed to return processing to the payment flow asap, by launching the final accounting writes and summarisation as an async orchestration.


All in one – this design complete the entire process in a single synchronous orchestration – all processing is completed before returning to the payment flow.


Dispatch Message – this design completes the final accounting in a similar way to the Async design, but rather than call the orchestration using a “start”, a message is dispatched as a trigger.


Our initial tests were all based around 10,000 payments in a single file, on a single biztalk box. A harness was built around BAM to extract timings based on the number of accounting message posted between the first payment BAM activity and the last accounting BAM activity being written. This benchmark takes out the “loading” of the messagebox with the payment messages and so gives us a purer “throughput” figure (it also took account of the buffered BAM writing)


During the day we saw a number of problems with rules engines timeouts. Config changes were made to increase translation timeouts and also increase the rules policy cache. We also made some changes to the code that instantiates and executes the policy to improve its use of the cache – this improved performance. We saw significant troughs of performance during what looked like the rules loading. Darren took a number of dumps and these were sent to the US to review.


Our day 1 tests started on 1 box showed performance ranging from 28 to 120 accounting entries per second. The best performing design was the “all in one” at this point, with 120 aeps (accounting entries per second) on a singe box and 169 on a two box config.


At the end of the day we started off 1 million payments with a 2 box config to see if we saw the same type of performance.

 

Summary : Best performance today: 169 eps on 2 boxes.

Overview of deliverables for ESB project

When I first arrived onsite, I was coming in cold. I knew what the consensus opinion was of what an ESB is, but I knew nothing about what the actual requirements were, nor functionality or delivery plans. I stepped into a large project, however my part of it was a blank slate. The infrastructure was being laid down (a very lengthy process as it is a complex environment, both technically and from an organizational perspective), but beyond what was done in the Proof of Concept that turned this into a BizTalk project, there was no code or finalized designs.


 


Perfect: just the way I like it!


 


So, the first thing we needed to do was explore requirements and define the scope.  I’m a firm believer in iterative development with phased deliverables, as are the client’s personnel, so that’s the approach we are taking for what will likely be a very long-term project.


 


After discovery, our list of deliverables breaks down as shown below.


 


What we are calling the “ESB Core Engine” will provide:


 



  • Dynamic routing – the ability for a message entering the ESB to be routed to an endpoint. This could be a protocol hop, or to an application

  • Dynamic transformation – the ability to dynamically transform a document from one type to another (think BizTalk mapper, but dynamic application of maps)

  • Exception management – a standardized and common way to surface and handle exceptions

 


Beyond the core functionality, additional capabilities will be:


 



  • JMS pipeline component (for JMS header promotion and full fidelity preservation of JMS headers as a message passes through BizTalk)

  • “Heartbeat” service that can be used for interop testing

 


This is not a case of “if you build it they will come”, so to accelerate getting developers up to speed, we will include an SDK, which will contain:


 



  • Code samples (for consistency, these will look and feel like the BizTalk SDK samples)

  • Documentation, “A Developer’s Guide to the ESB Core Engine”

 


From an admin/operations perspective, we will have two additional pieces (after Iteration 1):


 



  • Provisioning tool – defines the “on-ramps” and “off-ramps” into the ESB, the message receipt points

  • Engine Central (still looking for a better name here J) – a portal that allows ESB administrators to see metrics about engine performance and ESB usage patterns. We may (or may not) also opt to include some kind of self-provisioning here. This piece is a bit fuzzy at the time this was written, as we have a few possible approaches, and just need some time to think it through further. I will elaborate later, but this could become a core piece of the puzzle.

 


There will be phased deliveries within each of those areas. There will almost certainly be additional areas in the future, but for now, I’m happy with the mix and feel it will provide the infrastructure we need to facilitate interop between the 1,700 applications running in this enterprise.


 


On a side note: I recently saw an “ESB vendor” (remember, so companies would have you believe that an ESB is a product) proudly proclaiming that their ESB consists of 144 “pieces”. As impressive as that number sounds (no doubt one of the competing vendors will come out with one that has 150 pieces!), I’m a minimalist and prefer the “keep it simple” approach. The technology stack that we will be using is:


 



  • BizTalk Server 2004 (switching to 2006 as soon as we can)

  • SQL Server 2000

  • <LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo2; tab-stops: l

BizTalk Server 2006 has RTM’d

BizTalk Server 2006 has RTM’d today … best BizTalk ever, shipped on time and a very high quality server product. See the announcement here.


http://www.computerworld.com/softwaretopics/software/story/0,10801,109944,00.html

http://www.crn.com/sections/breakingnews/breakingnews.jhtml?articleId=183702861

Of course, my favorite BT’06 features are are WSS Adapter, BAS and BAM … because my colleagues and I worked on many of these features … 😉


I’ve been working in BizTalk team for 4 years now, this is my second BizTalk release, and I really like the things that I do here. The product and the people are great, there are MANY great opportunities, the mix of technologies that we use or develop is exciting and challenging. This is one of the best places to be at Microsoft. If you are interested, check this post on how to get your resume posted and join the team http://blogs.msdn.com/biztalk_server_team_blog/archive/2006/03/21/556067.aspx
Normally, I would get borred working on the same product for 4 years, but that was not the case with this product. I decided to stay with the BizTalk team and work on the next version of BizTalk that should be even better. However, I will change teams inside BizTalk, I am moving to the engine/message box team. I have admired the work they’ve done for quite a while now and that seems to be the right place for me going forward.


If you haven’t done this already, take a look at the latest BizTalk release here http://www.microsoft.com/biztalk/default.mspx

Office 2007 Delay? Actually, No.

There has been a lot of confusion over the announcement that Office 2007 would also reschedule its consumer launch date to January 2007 to match that of Vista.  This marketing launch is a different animal than the development and availability date of the Office 2007 product(s).


Microsoft clarifies this with a press release that reads as follows:


—-


Microsoft Confirms Timeline for 2007 Microsoft Office System
Product on track for October RTM.


REDMOND, Wash. – March 24, 2006 – Yesterday, Microsoft Corp. confirmed the timeline for release and availability of its 2007 Microsoft%u00ae Office system. The company remains on track to complete work on the 2007 Microsoft Office system in October of this year and is planning to make the product available to the business customers through the volume licensing program in October 2006. Retail and OEM availability of the product are scheduled to coincide with the retail and OEM availability of the Windows Vista%u2122 operating system in January 2007.


—–


It has been stated that Beta 2 will be available in June, and the above press release specifies October as the RTM and bits available date.  Rock on!

BizTalk 2006 RTM is here!

Congratulations to the BizTalk team for another awesome release and for keeping on schedule. These guys are like clockwork man! It was just 2 years ago that we got BizTalk 2004 and now we get this great update with awesome feature updates to BAM, administration and the messaging engine. I’m psyched about being able to use this new release in our BizTalk course.