- This topic has 3 replies, 1 voice, and was last updated 9 years, 3 months ago by
.
-
Posts
-
-
Hello,
I have a pipeline hosted in an Orchestration that is being used to split a large flat file:
recPipeOutput = Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(MyPipeline), MessageType);
Each file is approx 13MB in size and contains around 270,000 lines. Each line is a separate message that needs to be processed by a single Orchestration, a line at a time, hence hosting the pipeline and then using:
recPipeOutput.MoveNext()
recPipeOutput.MoveNext()
In a loop.
This works for small files, but with the files containing 270,000 lines, the orchestration dies with an 'System.OutOfMemoryException' error.
Any advice on how I can receive, split and process the lines of the flat file without running out of memory? The box has 2GB of RAM, and I imagined that would be enough for a file of this size.
Would splitting and processing the file in C# code in a custom pipeline component be a better way?
Thanks in advance.
-
First of all, 2g memory is not much. The os will use about half that.
You need to determine which process is taking up the memory. The pipeline debatching or the per-line processing. Personally I would pass the filename to a .net object that could extract each line in order. Create a MoveNext property that would return an xml message holding the parsed content of a line optimally formatted for whatever processing BTS is required to do. Further I would make sure I was minimizing the foot print by using a file stream rather than reading the entire file into memory at once.
It does not sound like the per-line content is large , but try to use inline or call orchestrations to complete the task to minimize the number of messages being processed.
-
-
mchallis,
I would recommend performing this kind of splitting on a receive location rather than in an orchestration – you should be able to use the same pipeline.
This will produce 270,000 messages into the MessageBox for your 13Mb file, but BizTalk will be able to handle the load much more easily.
Hope this helps,
Nick.
ps. you've just posted a good question for a blog entry! Thanks 🙂
-
- The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.