by community-syndication | Nov 30, 2006 | BizTalk Community Blogs via Syndication
For the last couple of months, I’ve had a lingering item sitting in my Outlook todo list. Finish that damn BizTalk 2006 exam. It may seem strange that a BizTalk Technology Specialist wouldn’t have the BizTalk certification and one may wonder why I didn’t knocked that exam off as soon as it came out. Participating in the beta was in fact what most of my fellow TS’s did and there are many blog posts out there to show for it. So why didn’t I get it done as well? It basically came down to an issue of timing. I started with Microsoft at the end of February of this year. BizTalk 2006 and the beta exam came out shortly after I joined. We’ll, it took me a number of months to get my feet under me with the new role and to adjust to the Microsoft culture, processes and lifestyle. I felt at the time that there were more pressing issues and jobs for me to undertake than fooling around with the beta exam, I left that to other more seasoned MS veterans. Once I settled into the role and really got my head focused around the MS sales processes and such, I was eager to get out into the field and really start extoling the virtues of BizTalk. After a few months of doing that, I began to remember about that pesky certification that I needed to do. So this week, I blocked off a couple of days for a refresher dive into the documentation and planned the exam for today. Glad to say I passed as it would have been a real kick in the ass to fail a certification on the very technology that I’m paid to specialize in.
On a side note, I met with a developer from one of our BizTalk partners about 3-4 weeks ago that had just taken the exam. This bright fellow managed to obtain the awesome score of 1000 out of 1000. While I’m humbled to say that I didn’t manage to match that lofty score, I am really happy to see that we’ve got some really bright and skilled BizTalk developers out there.
For those of you that are looking at going for the certification, I’ve got the same advice for you that you’ll see most everywhere else. There is lots of BAM, lots of BRE and lots of BAS. If you’ve got a firm handle on those technologies, then you’re looking good. Additionally, you’ll want to make sure that you’ve got a good understanding of the new messaging elements that shipped with 06. Recoverable interchange, failed message routing etc. Good luck if you’re planning on going for it.
Cheers and keep on BizTalking
by community-syndication | Nov 30, 2006 | BizTalk Community Blogs via Syndication
On a recent project, we had to deal with binary messages in BizTalk Server 2004. Specifically,
we needed to run a binary message through an orchestration, both receiving it and
sending it through messaging ports, as well as passing it to custom .NET components
that knew how to manipulate it.
In our case, this was basically a binary compressed (zipped message), which we decided
explicitly not to decompress during a pipeline, and instead always keep it in it’s
original raw form. Handling it wasn’t much of a problem for the most part, since you
can apply the same trick the RawString class does, but just keeping the contents as
a Byte[] instead of a string (implementing it from scratch left as an exercise for
the reader).
The only tricky part was getting BizTalk to receive the raw binary message and fire
off our orchestration to process it without jumping through too many
hoops. Scott
Colestock has a great entry on
how to receive a RawString on an orchestration, so we obviously turn to that for inspiration.
But, even with all the great advice from Scott and both alternatives presented there,
it’s still quite a bit of work.
Fortunately, working with my friend Gustavo we
found out an easier way that didn’t involve multipart message types or custom pipelines.
It goes like this:
The BizTalk orchestration is setup to receive a message of type XmlDocument, which
ensures that the activation subscription for the orchestration does not contain a
MessageType-based predicate. Then, we construct a new message declared using
our custom .NET class (let’s call it BinaryMessage), and initialize it in a Message
Assignment shape by calling the new operator to call a constructor, passing the original
XmlDocument message as an argument.
Here’s were we differ from the other proposed solutions: Our custom constructor takes
an XLANGMessage instance as an argument. We figured it wouldn’t work at first because
the original message in the orchestration was an XmlDocument and not a regular BizTalk
message, but it didn’t cause a problem. Anyway, since we have the XLANGMessage instance,
we could simply get the body part and call RetrieveAs() to get the contents of the
message as a Stream and read that into our internal Byte[] array, like this:
public BinaryMessage(XLANGMessage
doc) { Stream stream = (Stream)doc[0].RetrieveAs(typeof(Stream)); using (stream)
{ MemoryStream tempStream = new MemoryStream(); byte[]
buffer = new byte[64*1024]; int bytesRead
= 0; while ( (bytesRead = stream.Read(buffer, 0, buffer.Length))
> 0 ) tempStream.Write(buffer, 0, bytesRead); Contents = tempStream.ToArray();
} }
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
Granted, the solution isn’t terribly efficient (since it does involve doing a message
copy), but since we’re only going to be processing about a couple messages a day,
that wasn’t a significant concern.
Technorati
tags:
BizTalk,
BTS
by community-syndication | Nov 30, 2006 | BizTalk Community Blogs via Syndication
(he) Sometimes, we have to put 10,000 lines into a single scrollbox. Computer hangs, memory consumption goes through the roof, slider is too small, nothing works. What would you do?…(read more)
by community-syndication | Nov 30, 2006 | BizTalk Community Blogs via Syndication
I’m pleased to announce that all of the content for my book is now complete! As of two weeks ago we’ve now entered the reviewing/copyedit phase which should last until the end of the year keeping us on track for a March publication. Woo Hoo!
Words cannot express my relief! The last few months have been hell with pretty much every evening and weekend focused on churning the remaining chapters out. However, after now having time to reflect and review the content end-to-end it’s been well worth it, the content for the book is amazing and on the whole isn’t available anywhere else (books, internet or documentation).
The key focus of the book remains on best practice, distilling all of the experience and techniques used on some of the largest BizTalk deployments. Failure to follow much of this best practice leads typically leads to problems which this book should hopefully address.
Here’s a run-down of the chapter content to whet your appetites yet further!
Chapter 1: Technical Primer (Darren Jefford)
This chapter primes the reader on some key technologies which each developer should have a basic understanding of, otherwise they will run into problems during development and testing. I cover XML Schemas, Namespaces, XPath and Serializable classes.
Chapter 2: BizTalk Architecture (Darren Jefford)
This chapter is the only chapter that really deals with explaining the basic BizTalk principles, I position what value BizTalk brings to solutions and to counter the regular discussion I cover what it would take to write a solution without BizTalk and compare to a BizTalk solution. I then run through all the key architectural peices of BizTalk and explain what they do and how they work.
Chapter 3: Adapters (Darren Jefford)
In this chapter we cover the key Adapter concepts, Ports, Retries, Tracking, etc. and then drill into each Adapter supplied with BizTalk to explain what the adapter does, what context properties it promotes and where appropriate and relevant a walk-through to show how to use the Adapter.
We detail the new R2 adapters at a high-level and in-time for publication we’ll add downloadable content to demonstrate how to write an adapter using the new WCF Adapter framework shipped with BizTalk Server 2006 R2 (not available at the time of writing sadly).
Chapter 4: Pipelines (Kevin Smith)
In this chapter we cover the core principles of pipelines, IBaseMessage, Pipeline Architecture, Transactions, Stages, Failures, etc. We then provide a full example of how to write a pipeline highlighting the core tenants you must follow to write a great pipeline component, sadly many pipeline components don’t follow such best practice and can be the prime culprit in performance problems.
Chapter 5: Orchestrations (Darren Jefford)
In this chapter we cover what Orchestrations look like under the covers, how subscriptions work, Persistence Points (in great detail), Ports, Multi-part messages, Shapes (including how the Parallel shape REALLY works), Promotions, Correlation, Convoys, calling .NET components, Exceptions, etc.
Chapter 6: Business Activity Monitoring (Darren Jefford)
In this chapter we cover all of the BAM concepts (Activities, Continuation, Relationships) and detail how BAM can be used to instrument your solution and in fact your entire enterprise. We then demonstrate in great length how a fictional solution can be instrumented and how the BAM Portal or a custom portal can be created to view this data.
Chapter 7: Rules Engine (Darren Jefford)
In this chapter we cover the Rules Engine principles, explain the rete algorithm theory and show how the rules engine actually executes rules and generally demystify the rules engine. We then explain Policies, Rules, Facts, Data sources, Vocabularies, how to test and critical techniques such as long term fact retrievers.
Chapter 8: Testing (Kevin Smith)
In this chapter we cover what types of testing that you must apply to your BizTalk solution (Unit Testing, Functional Testing, Integration Testing, Performance Testing and tuning, Stress Testing, Overload Testing, User Acceptance Testing, DR Testing and Production Support testing.
We detail in great depth how BizUnit can be used to automate your testing, how you can perform code coverage of your BizTalk solution, how LoadGen can be used to generate load, how you can profile your solution using the Visual Studio profiler, etc.
Chapter 9: Performance and Scalability (Darren Jefford)
In this chapter we cover all of the things you must have in place before beginning any performance testing, such as suitable disk array, SQL Database growth disabled, MaxConnection settings ,etc. Forgetting these often lead to bad performance results.
We then move to explain how to monitor BizTalk, SQL and IIS, explaining all of the relevant performance counters and what they actually mean and also provide a complete reference on what BizTalk throttling is and how it works including explaining all of the various throttling states.
The final section of the chapter is where we cover a number of Common Symptoms, these are problems that I’ve seen customers run into regularly. Things like High CPU usage, Orchestrations completed/sec drops off suddenly, Low Throughput and explain what the common reasons for this are and things to check to identify the underlying problem.
Chapter 10: Low Latency (Kevin Smith)
In this chapter we cover what Low Latency means to BizTalk solutions and how you can measure it. We then drill into a variety of techniques that you can employ to reduce latency for your solution.
We cover how low latency can be reduced by adjusting configuration settings, calling Send Ports directly from Orchestrations and the implications that you must consider when doing this and how you can call Web Services directly from Orchestrations and how you manage thread issues with this approach.
Chapter 11: Administration (Ewan Fairweather)
In this chapter we cover everything required to administer BizTalk effectively, it’s a great resource for Administrators as well as developers. The chapter covers the key administration tools, tasks that you need to undertake and highlights areas that are often overlooked such as SSO maintenance, suspended instances subscriptions, etc.
The chapter also covers in great detail how the SQL Server used by BizTalk needs to be looked after including backup, log shipping and how you can monitor the performance of the SQL Server. The chapter also covers MOM and how it can in conjunction with the BizTalk Management pack greatly simplify management.
Chapter 12: End to End Scenarios (Darren Jefford)
In this chapter I highlight all of the End to End Scenarios that were shipped as part of BizTalk 2006, these are often overlooked but provide full working solutions of real-world customer scenarios that have been fully tested and demonstrate best practice.
Each scenario is supplied with complete source code and provides implementations that you can use within your own solution. I cover each scenario and highlight key deliverables in each scenario such as the code required to use the SSO store for configuration data, an adapter to enable messages to be sent for “manual repair”, in-line pipeline invocation, etc.
Chapter 13: BizTalk Best Practices (Darren Jefford)
In this chapter we cover a number of best practices and techniques that have been used in a number of projects, these include Orchestration Message Debatching, handling large messages, storing configuration data, subscribing to failure messages, message controller pattern, never have suspended messages or orchestrations, instrumenting your solution, etc.
Chapter 14: Windows Workflow and BizTalk (Darren Jefford)
In this chapter we position cover the key concepts of Windows Workflow and how it fits with BizTalk Server, demonstrate how the BAM Interceptor for Windows Workflow works which is delivered as part of BizTalk Server 2006 R2 and position how BizTalk, WF and Sharepoint can be used together to provide compelling solutions.
by community-syndication | Nov 30, 2006 | BizTalk Community Blogs via Syndication
Obligatory Rant
BAM is such a good name for this technology for a few reasons. The first is that it reminds me of the kid Bambam on the Flinstones (You know, the baby with the club that had a thing for Pebbles). Very powerful and also tough to handle at times. Also because “BAM” is the sound my forehead makes while it collides with the palm of my hand as I discover an all too simple solution to some problem that has plagued me for hours or even days.
The Basics
This article will be a fairly unstructured summary of my recent experiences with BAM under BTS 2006. If there is interest, I will create some more in-depth posts on the topic.
I’ll be brief here because most of you know this allready. The basic way BAM works is the following:
- Create a Business Activity showing the LOGICAL progression and points of interest in a process (using Excel, or Visio ODBA)
- Deploy that Activity (as .xls or .xml to the biztalk server) using BM.exe
- Associate low level technical events (i.e. hitting a shape in an orch or a method in a .net class) with items you defined in the activity (using the BAM API or the Tracking Profile Editor (TPE)
The end result of this is that you can take the mess of code, orchestrations etc and transform it back into a logical/business view that even a CEO could understand. All in all, very cool.
Some Pragmatic Advice
“DO” List
- Try both ODBA and Excel for creating activities (I like excel better, but hey, who cares?)
- Become familiar with the BM.exe utility. You really only need the “deploy-all” and “remove-all” commands to start with
- Keep your activity files versioned! If you deploy an activity, then change the activity file, then try to undeploy you may have problems. Then you have to undeploy by manually deleting tables and rows in the BAMPrimaryImport table.
- I recommend that you manipulate your activity definition in Excel. When you are ready to deploy it, export it to xml and then deploy the xml. That way if you fiddle with the Excel file, you can still undeploy the activity using the same xml you used to deploy it.
- Before you re-export your activity to XML, undeploy the activity. Or even better export your Activity to a new version of the xml (i.e. with a version number in the file name
“DON’T” List
- Don’t try to map milestones from Message Constructs or Loops (or other special cases). That is advanced and you may need to use the BAM API call to make that work as you would expect
- Don’t put spaces or special characters in your Activity Names. It can lead to problems later where the Event Log will say that “stored procedure does not exist” and your BAM Aggregation will not work.
- Don’t have a group shape as the last shape in the orchestration you are tracking in TPE. For some reason it won’t work. Just add any shape after the group to fix the issue.
Continuations & Relationships
I can’t do these justice in this entry so this will be a bit shallow. I will probably blog on this separately. The more comments i see below requesting this, the sooner I will get around to it.
- Continuations allow a single activity defintion to span muliple orchestrations or even other classes/systems seamlessly. It can also allow you to use different unique identifiers accross those systems to When using TPE to create a continuation. Create a continuation and name it, then create a continuationID and name it the exact same thing. Map an some kind of ID value (PO ID, RequestID, etc.) from a messaging shape (receive, construct, send as described in the above DO List) in the first orchestration to the continuation. Then in the second orchestration map that same ID value to the continuationId. The idea is that the engine will recognized that those to values match and will correlate them so that both instances can contribute data to the same Activity instance
- Use Continuations when the Relationship is 1:1
- Use Relationships when the Relationship is 1:Many
RealTime vs. Scheduled Aggregations
There is a little button in the Excel view (if you have installed the BAM addon) that will mark the orchestration as RealTime or Scheduled. Again, this is complex so I will summarize below.
Common Problems & Solutions
Problem: The BAM Activity and the Tracking Profile have been deployed, the process has been run, but there is no data displayed for the activity in the BAM portal or LiveData Excel sheet.
Solution: When you deploy a non-Real Time Aggregation (this is default) while using SQL 2005. It will create the <package>_DM and <package>_AN packages. These packages need to be run to populate the BAM tables and get you the BAM Data. To run these packages you should select “Integration Services” in the SQL Management Console (not database engine). Once you have found them, run them. Hopefully, you’re data will appear.
Problem: The BAM portal or LiveData Excel sheet shows “BLANK” for one column and one row in the Aggregation display. It seems like the measures, and the milestones are not being combined into the same activity. Also, the data shows up in two rows (or more) in the BAMPrimaryImport.dbo.bam_<ActivityName_Completed table.
Solution: This could be a number of things, but if it was the same problem that I had the solution is the following. When you are in the TPE and you are associating data from the message, be sure to use the orchestration schedule. Right click on your receive, Message Construct, or Send shape and select Message Payload and map the items from there. Many people go to the “Select Event Source” and use the “Messaging Payload” from there. YOU ONLY WANT TO DO THAT IF YOU ARE JUST USING SEND RECEIVE PORTS TO MOVE MESSAGES. If you are using an orchestration, map everything from the orchestration schedule.
Problem: Your BAM aggregation is not working and you see “Could not find stored procedure” in your event log.
Solution: This is probably caused becaues you have a “space” or some other bad character in your Activity Name. Undeploy your activity, rename it so it does not have the space. Redeploy it. Adjust your TPE (if required). This should fix the problem.
by community-syndication | Nov 29, 2006 | BizTalk Community Blogs via Syndication
Irena
Kennedy posted a lengthy entry on choosing
the right integration technology from the growing stack of frameworks and servers
Microsoft Offers. Included in the entry as candidates are: SQL Server Integration
Services (SSIS), SQL Server Service Broker (SSBS), SQL Server Replication, MSMQ, BizTalk
and WCF.
Plenty of good stuff to consider. Highly Recommended! (Found via Richard
Seroter).
by community-syndication | Nov 29, 2006 | BizTalk Community Blogs via Syndication
A quick note to say thanks to all those folks who attended the webcast i gave yesterday on “Introduction to Biztalk” for VBUG. There was quite a good turnout and i enjoyed giving the presentation Slides have been sent to VBUG and you can get copies from there. The WMV should also be available soon. Do email me if you have more questions on Biztalk, BizUnit and the stuff discussed in the webcast.
by community-syndication | Nov 29, 2006 | BizTalk Community Blogs via Syndication
Yesterday Irena Kennedy posted a great compilation of multiple sources which resulted in a big fat post outlining selection criteria for an integration solution. She summarizes choosing between SSIS, MSMQ, WCF, BizTalk, SQL Service Broker, and Host Integration Server. Check it out.
Also, if you’ve never undergone a code review by Irena, you haven’t truly lived. It’s a truly terrifying and rewarding experience from a brilliant MCS asset.
by community-syndication | Nov 29, 2006 | BizTalk Community Blogs via Syndication
I had some discussion today with Christof Claessens about the merits of using .NET classes as orchestration message types in BizTalk Server 2006. Here are some of the reasons I came up with, and also some of the issues.
http://geekswithblogs.net/cyoung/articles/99464.aspx
by community-syndication | Nov 29, 2006 | BizTalk Community Blogs via Syndication
There are two interesting BizTalk 2006 bugs to be aware of – it seems like they would
be encountered fairly often, so I’m a little surprised that I haven’t seen more newsgroup/forum
activity around these.
The first bug is encountered when you attempt to start a BizTalk 2006 application
definition with orchestrations that have been bound to different hosts. (Say,
OrchA in HostA, and OrchB in HostB.) After the application has been deployed,
the first time the application is started (and ‘orchestration start/enlist’
is included in the start options) you will receive an error that looks something like
this:
Microsoft.BizTalk.ExplorerOM.BtsException: Could not enlist orchestration 'BizTalkSample.Orchestrations.Echo,BizTalkSample.Orchestrations,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=05a4a8d4071035d9'. Could not enlist
orchestration 'BizTalkSample.Orchestrations.Echo,BizTalkSample.Orchestrations, Version=1.0.0.0,
Culture=neutral, PublicKeyToken=05a4a8d4071035d9'. Value does not fall within the
expected range.
You can work around this error by manually starting your orchestrations in the MMC,
either one-at-a-time or grouped by host. The error only occurs the first time
you attempt to start – all future attempts will succeed. This is because the
underlying bug has to do with the creation of subscriptions for the orchestrations.
This is actually a regression from BizTalk 2004 – where you could in fact change multiple
orchestrations (across differing hosts) to a started state via the BtsOrchestration
‘Explorer OM’ class in a single transaction (i.e. before calling SaveChanges.)
In addition to the behavior just described with BizTalk 2006, you can’t do this kind
of Explorer OM work once multiple hosts are involved in BizTalk 2006.
But hey, there is a hotfix. Contact PSS and ask about KB927052.
The second bug is encountered when you have a BizTalk 2006 application with multiple
MSMQ receive locations. You will find that restarting your host instances takes
an excessively long period of time. This is due to a bug in the adapter which
was not allowing the host service to stop correctly (the Service Control Manager would
eventually just terminate the service…) Contact PSS and ask about KB928222.
