Processing 1.5MB file generates 940MB increase in BizTalkMsgBox

Home Page Forums BizTalk 2004 – BizTalk 2010 Processing 1.5MB file generates 940MB increase in BizTalkMsgBox

Viewing 1 reply thread
  • Author
    Posts
    • #16644

      We have a file containing 10,000 records, each record is 155 characters and consist of fixed length fields. There are no header or trailers. I have created a process that converts each message into a XML message and puts it into a MQSeries queue.

      1) Receive Location using the FILE transport type looks for a file in a specific directory (c:\jeff\LM*) and passes it to a custom pipeline

      2) Custom pipeline uses the Flat file disassembler and references a schema created with the flat file extentions.

      3) The flat file schema uses a Child Delimiter (0x0D 0x0A) and the order is postfix.In the schema editor it looks like

      <Schema?

           TheRequirement          Delimited

                Requirement          Positional        0       155

                <Sequence>          Positional        0       155

                     Date                 Positional        0       7

                     BuyCode          Positional        7        1

                     NSN                Positional        8        15

       and do on until all the fields are defined.

      4) A Send Port using the MQSeries adapter is configured to write to a queue using the passthru default pipeline and a filter (BTS.ReceivePortName == Split_Receive_Port)

      This works well and generates one XML message for each line in the file. The problem is that running 10,000 records through the process, the 1.5 MB file results in an increase of the BizTalkMsgBoxDb by 241 MB and the BizTalkMsgBoxDb_log by 669 MB. It appears that all this disk increase occurs during the Receive Port processing. It takes about 10:47 (10 minutes 47 seconds) for the file to disappear out of the directory in which no messages are being written to the MQSeries queue. Once the file disappears from the directory, messages start showing up in the queue. During this 10:47 all the increase in the DBs are taking place. During the time to put the 10,000 messages into the queue, the database does not increase by any noticable amount. It takes about 14:39 to put all 10,000 messages into the queue.

      I would appreciate any thoughts you may have.

    • #16678

      The receive pipeline debatching process is one large transaction.  This is why you see nothing happening for the first 10 minutes.  If something was to fail (not so much so in BizTalk 06 and you can sometimes bypass this) the whole process would roll back – thus getting no messages out. 

       

      There are some ways around this… Matt has some good stuff on this blog.  One is: http://objectsharp.com/blogs/matt/archive/2005/08/31/3209.aspx

       

      I don’t know much about the size of the databases. 

       

      Hope this helps.

      • #16695

        I kinda expected the Receive process being one big transaction. Despite this all occuring under a single unit of work, it seems strange that the 1.5 MB file would generate an increase of 940 MB in the database tables. We are hoping there are some extensive logging facilities we accidently turned on to account for a large portion of this increase but we do not know enough about the BizTalk -> SQL Server configuation to know if this is the case.

Viewing 1 reply thread
  • The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.