- This topic has 7 replies, 1 voice, and was last updated 9 years, 2 months ago by
.
-
Posts
-
-
Hello,
I have a flat file which is as follows Header, Date, Sender Data, John, 18 Data, Randy, 24 Trailer, 2 I would like to split the flat file into individual XML documents that contain only the Data lines so the above example would have only 2 XML documents each containing one data line. I would like to debatch it by calling the pipeline component in my orchestration because,
1) I will receive the whole file in my orchestration, so I validate my value in the trailer vs the number of data line Hence in this example the trailer line has a value of 2 which indicates there are 2 data lines. I have already done this in my orchestration using an Xpath query and now I would like to debatch this message using the pipelines in my orchestration.
2) There is a solution using Xpath but speed is really slow, If i want to process more than 10,0000 lines then it takes hours.
3) Splitting using Maps — I loose the flexibility and advantage of making it versatile So I would like to use the pipeline components in my orchestration, I have see articles in this blog that debatch an XML but not a flat file. I am not able to get it to work with a flat file. If any one could help me with this it would be really gr8 :)……………………………………………. -Kelly
-
Can you post an example of your flat file data, there are a few potential traps with trailer records and pipeline debatching
-
Thanks, for your reply
The trailer contains the number of data lines and the source system. For example
Trailer, Sys1, 25
As a matter of fact the above line is from my data file.
-
Any chance of getting a whole file. With trailer records, it is not so much the Trailer record itself, as the preceding records which determine how you would debatch
-
I am pasting an Instance of my Schema
Header,Date,SystemID,Location
Data,Name,Amount,Customer,Location,Rating,Year,POSID
Trailer,DataRec
The above is my data file with the the data line Max occurance set to unbounded, As a matter of fact Every line starts with Header, Data or Trailer so we know what we are looking at.
The specs are
Header (1,1)
Data (0,Unbounded)
Trailer(1,1)
-Thanks
-
It seems you want to define the parsing schema as
<Rootnode>
<Header> (0,1)
<Data> (0,1)
<Trailer>(0,1)
So your batch is the whole file and each individual XML instance will either be a header, a data or a trailer record.
Then ignore the header and trailer in your application. This obviously does not preserve your integrity check, but it parses the instances like you want.
Hope that is useful
– weak architect
-
I am doing the integrity check in the beginning of my orchestration and I would like to parse it as
Root
<Data>
</Data>
So if you could tell me how to use the pipeline in my orchestration it would be really great.
Thanks for your Help!!!!!!!!
-
If you are looking for speed, then you could consider debatching the file in the first pipeline.
Define a header schema and a document schema:
Header
–Record maxOccurs = 1 tag=HeaderDocument
–Data maxOccurs = 1 tag=Data
–Trailer maxOccur = 0 tag=TrailerAlso create a property schema and add a record count property. In the document schema promote the record count field from the trailer into this property.
Each record will be debatched into a separate message, with the last record also having the trailer record attached
Write a custom pipeline component that checks your custom property against BTS.InterchangeSequenceNumber. You will need to perform this logic after the entire stream has been read and all fields have been promoted (Check out the CustomPartyResolution pipeline component in the SDK on how to do this)
If your custom property does not equal the BTS.InterchangeSequenceNumber you can throw an exception, this will abort the entire batch if you have Recoverable Interchange Processing turned off. Or you could promote an error message into the message context that could be handled by you orchestration
-
-
-
-
-
-
-
- The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.