Very large FlatFile – best approach ?

Home Page › Forums › BizTalk 2004 – BizTalk 2010 › Very large FlatFile – best approach ?

Viewing 1 reply thread
  • Author
    Posts
    • #20128

      Hi,

           I have a positional flat file which is > 0.5GB in size.Using the FF wizard the FF was transformed into an XML schema.
      The FF has header info.
      From the original FF records (having 50 items) we need only 5 items per record. Hence a transform was done to collect and calculate only the neccessary items.
      This is then loaded into the DB using bulkinsert.

           All this works fine when the size of the file is small.

      When it is > 1.5MB  the system errors out.

      What is the best approach to process large flat files ?
      What are settings in Biztalk that i need to be aware of for processing these large files ?

      Any good articles for Biztalk 2006 ?

      -Jerry

    • #20137

      Jerry,

      Thats a good question and unfortunately there isn’t a great deal of material out there on best practice for (very) large flat-files, however a good starting point would be the following articles:

      When I’ve had to process ~200Mb flat-files I’ve written pre-processing scripts (external to BizTalk) to reduce the number of records per file (while preserving the header) and custom adapters that do a similar job. Its not an easy job, but it is possible to come up with workable strategies.

      Nick.

Viewing 1 reply thread
  • The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.