Flat File Messages Processing Slow in File Adapter

Home Page › Forums › BizTalk 2004 – BizTalk 2010 › Flat File Messages Processing Slow in File Adapter

Viewing 2 reply threads
  • Author
    Posts
    • #23761

      I have a very large flat file (15MB, 125,000 records in 500K lines).  My file receive adapter has a flat file disassembler pipeline to publish the messages from this file to the MsgBox.  I stopped the send port that subscribes to the messages, so I can get a read on how fast the messages are being created and sent to the MsgBox.

      This occurs on a test server that has BT 2006 r2 and SQL Server 05 installed.  The server is Win Svr Std 03, 64 bit  (SP2) and has 4 processors, Xeon 3.40 GHz with 12 GB of ram.

      I’ve worked very hard with the BT Group properties (Large File threshold & fragments) and the throttling params for the Receive Host, but cannot get the throughput on this file to get above 122 messages/sec (PerfMon’s Msg publishing incoming rate and outgoing rate counters).   Watching these two PerfMon counters, I will see both of these drop to zero frequently and stay there for several seconds.

      Can you suggest anything else I could be looking at?  I think with this kind of horsepower, there should be more throughput, despite having BT and SQL on the same server.

      Am I just wrong on this assumption?  And is 122 msgs/sec about all that I can expect?

       

    • #23762

      Hi JLingenfelder,

       

         Consult the following MSDN article on best practices to increase performance of BizTalk Server in processing large messages:

      http://technet.microsoft.com/en-us/library/aa560481(BTS.20).aspx

       

         You probably have done some of the things mentioned in the article, but it’s always good to have another look so as not to overlook any options.

       

         Daniel.

       

    • #23764

      One other piece of info.  The performance counter “In-process message count” stays at 0 (zero) and the “In-process message count threshold” is set to 1000.

      • #23781

        If you have followed on processing large messages:  http://technet.microsoft.com/en-us/library/aa560481(BTS.20).aspx

        then you need to fine tune your system. As you are using BizTalk server & SQL server on the same machine this can be a one performance hit.

        You can try with the following performance counters what I have successfully used to fine tune the system.

        • 1. Process Memory

         (Performance counter located inside the BizTalk:Message Agent performance object)

        The default Process memory usage setting is set to 25. Because your system has 12 GB RAM.

        So using the defaults, when a BizTalk host operating system process uses more than 3 GB of

        memory (25 percent of 12GB), process memory throttling is introduced, which will restrict the size of

        the thread pool and/or introduce a progressive delay to message publishing. So you can increase

        percentage & see the throughput. Apply only one threshold change at once & fine tune the system.

         

        • 2. Database Session Count –

         Database session count throttling indicates that too many concurrent database connections

         

        have been opened by the BizTalk host. The number of database sessions currently open is

         

        viewable via the Database session performance counter located in the BizTalk:Message Agent

        performance object.

         

        The default Database connections per CPU setting are set to 0, which means that this particular

         

        type of throttling is disabled. So can have infinite database connections. You can give 6,7,8 & see

         

        the performance . For our tune-up system it took “8” .

         

        Do test and see what happened.

Viewing 2 reply threads
  • The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.