Re: Exception of type ‘System.OutOfMemoryException’ using a Pipeline to split a large positional flat file

Home Page Forums BizTalk 2004 – BizTalk 2010 Exception of type ‘System.OutOfMemoryException’ using a Pipeline to split a large positional flat file Re: Exception of type ‘System.OutOfMemoryException’ using a Pipeline to split a large positional flat file

#18119

First of all, 2g memory is not much.  The os will use about half that.

You need to determine which process is taking up the memory.  The pipeline debatching or the per-line processing.  Personally I would pass the filename to a .net object that could extract each line in order.  Create a MoveNext property that would return an xml message holding the parsed content of a line optimally formatted for whatever processing BTS is required to do.  Further I would make sure I was minimizing the foot print by using a file stream rather than reading the entire file into memory at once.

It does not sound like the per-line content is large , but try to use inline or call orchestrations to complete the task to minimize the number of messages being processed.