Home Page › Forums › BizTalk 2004 – BizTalk 2010 › Processing 1.5MB file generates 940MB increase in BizTalkMsgBox
- This topic has 2 replies, 1 voice, and was last updated 9 years, 1 month ago by
community-content.
-
AuthorPosts
-
-
November 30, 2006 at 2:15 PM #16644
We have a file containing 10,000 records, each record is 155 characters and consist of fixed length fields. There are no header or trailers. I have created a process that converts each message into a XML message and puts it into a MQSeries queue.
1) Receive Location using the FILE transport type looks for a file in a specific directory (c:\jeff\LM*) and passes it to a custom pipeline
2) Custom pipeline uses the Flat file disassembler and references a schema created with the flat file extentions.
3) The flat file schema uses a Child Delimiter (0x0D 0x0A) and the order is postfix.In the schema editor it looks like
<Schema?
TheRequirement Delimited
Requirement Positional 0 155
<Sequence> Positional 0 155
Date Positional 0 7
BuyCode Positional 7 1
NSN Positional 8 15
and do on until all the fields are defined.
4) A Send Port using the MQSeries adapter is configured to write to a queue using the passthru default pipeline and a filter (BTS.ReceivePortName == Split_Receive_Port)
This works well and generates one XML message for each line in the file. The problem is that running 10,000 records through the process, the 1.5 MB file results in an increase of the BizTalkMsgBoxDb by 241 MB and the BizTalkMsgBoxDb_log by 669 MB. It appears that all this disk increase occurs during the Receive Port processing. It takes about 10:47 (10 minutes 47 seconds) for the file to disappear out of the directory in which no messages are being written to the MQSeries queue. Once the file disappears from the directory, messages start showing up in the queue. During this 10:47 all the increase in the DBs are taking place. During the time to put the 10,000 messages into the queue, the database does not increase by any noticable amount. It takes about 14:39 to put all 10,000 messages into the queue.
I would appreciate any thoughts you may have.
-
December 2, 2006 at 5:25 PM #16678
The receive pipeline debatching process is one large transaction. This is why you see nothing happening for the first 10 minutes. If something was to fail (not so much so in BizTalk 06 and you can sometimes bypass this) the whole process would roll back – thus getting no messages out.
There are some ways around this… Matt has some good stuff on this blog. One is: http://objectsharp.com/blogs/matt/archive/2005/08/31/3209.aspx
I don’t know much about the size of the databases.
Hope this helps.
-
December 4, 2006 at 10:09 AM #16695
I kinda expected the Receive process being one big transaction. Despite this all occuring under a single unit of work, it seems strange that the 1.5 MB file would generate an increase of 940 MB in the database tables. We are hoping there are some extensive logging facilities we accidently turned on to account for a large portion of this increase but we do not know enough about the BizTalk -> SQL Server configuation to know if this is the case.
-
-
-
AuthorPosts
- The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.