- This topic has 11 replies, 1 voice, and was last updated 9 years, 2 months ago by
.
-
Posts
-
-
Is there any limitation for size of message in BizTalk? I have a message of size approximately 200 MB. It’s an XML file. I will have to insert the data into an SQL table. I have not yet implemented it. I just want to know if I have to take care or follow any specific steps while handling large messages.
Thanks in advance.
-
I don’t think there is any hard size limit, but you need to be very careful with large messages to avoid performance problems. I would go ahead and build out the solution using the 200 MB messages, and then performance test it. If the performance is bad, then one approach is to split the large messages into a bunch of smaller messages so the processing can be distributed across multiple servers or processors. Here is more info on how BizTalk processes large messages: http://msdn.microsoft.com/en-us/library/aa560481.aspx.
-
Hi ,
I have experience in handling large messages in BizTalk. It’s really a major performance bottleneck.The performance bottleneck comes into play when the entire file is received, parsed by the XMLReceive pipeline, and then stored into the Messagebox. You can use custom decode pipeline component which will help you to identify the Biztalk message size & if it exceeds the message limit ( say 50 MB) then store the file to disk in a uniquely named file and store the path to the file in the IBaseMessagePart.Data property. The end result will be a message that only contains the path.
Throughout the custome pipeline encoder you can take that large message back in to your process.
But you need to be careful if you load this large message in to XML DOM . You will get out of memory exception .
So better way to use XMLReader .
Let me know if you need more information on this.
Thanks & Best regards,
Nandika
-
If I remember correctly, depending on what you need to do to the message, BizTalk might never load the DOM. For example, if you used a messaging only solution with passthrough pipelines, the DOM wouldn’t get loaded. Better performance. However, if you need to map the message it’s definately getting loaded. Performance hit.
-
If we are going to use custom pipeline components & trying to get/set message properties then easy way to load to DOM.
for example “Set Namespace Pipeline component ” comes with “Microsoft BizTalk Adapters for Enterprise Applications” is very helpful but it’s raised “memory exception issue ” for large messages .
Nandika
-
Guys,
Thanks for all your responses. I just have to process the incoming xml using map to update more than one table in my SQL. Please Suggest me what precautions I need to take.-
You might want to debatch the document on the incoming pipeline, so that you are mapping a bunch of small messages instead of one huge message. If the performance is bad, you might want to look at using SSIS instead. Bulk loading data into SQL Server is not BizTalk’s strong point. Unless you are getting some other benefit out of using BizTalk here, I would lean toward SSIS.
-
-
-
-
Hi I am Vamshi and I would like to know more details about how to implement this. Can you please provide more details on this. My email id is [email protected].
-
-
-
You can try using SQLBulkInsert adapter. Please refer http://www.biztalkgurus.com/blogs/biztalksyn/archive/2005/10/23/Processing-a-Large-Flat-File-Message-with-BizTalk-and-the-SqlBulkInsert-Adapter.aspx
-
In my recent experience while using biztalk 2006, I have found that using very large messages causes a huge performance hit. If throughput speed is not an issue, and as long as you NEVER load the document into memory (through using xmldocument \ object serialization or anything that may require the entire document to be loaded ) you will find biztalk works, albeit very slowly. If this is not part of a process or ESB I would suggest you use SSIS instead, It is much quicker.
-
-
- The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.