- This topic has 12 replies, 1 voice, and was last updated 9 years, 2 months ago by
.
-
Posts
-
-
-
You mean RcvPipeOutput.MoveNext() throws an error? or what?
You can always just extract the messages into a secondary collection on one loop and then loop over that in a second one.
-
-
Joel,
What you say makes sense, but that’s not why I said what I did. Basically what I was hinting at is that if the OP wanted to keep his processing code like he’s doing now, then I don’t see him as having much choice at all.
While the serialization hit would be big (you’re right about that), trying to call a webservice 200 times to process the batch of messages inside a single atomic transaction would probably be far worse.
It sounds like perhaps his best bet is to skip trying to call the pipeline inside the orchestration at all and instead just disassemble it inside a real receive port and then let BizTalk do its thing in the \”old way\”. It would still be pretty heavy (processing large message batches always is), but at least BizTalk already has better mechanics in place to handle it.
-
[quote:3ab64229f2=\”tomasr\”]You mean RcvPipeOutput.MoveNext() throws an error? or what?
You can always just extract the messages into a secondary collection on one loop and then loop over that in a second one.[/quote:3ab64229f2]
Wouldn’t the collection and all the types in it have to be serializable to exist outside the atomic scope?
-
[quote:a237317aac=\”tomasr\”]Sure they would. Why would that be a problem?[/quote:a237317aac]
I guess the issue is more one of performance. Thinking from a performance perspective, I wonder what kind of overhead you would be looking at. I guess it depends on how GetCurrent is implemented. For instance, if you have multiple messages (say like in an envelope), I believe each message contained in the envelope is wrapped in an XmlDasmStreamWrapper class and the stream is not consumed until the BizTalk engine reads it for serialization to the message box (assuming no further downstream components in the pipeline touch it.) At least this is how it works for FFDasmComponent. So does ReceivePipelineOutputMessages.GetCurrent() trigger reading of the XmlDasmStreamWrapper stream? If you had 200 messages loaded into memory, say in an ArrayList collection, depending on your persistence points you could incur a lot of database hits with a large amount of data in each \”write\”.
So if each message to be returned by GetCurrent isn’t \”read\” until GetCurrent is called, then it might be more efficient to find an alternative method of processing the messages. If the ReceivePipelineOutputMessages class internally loads all messages into memory immediately after completion of the pipeline, then the performance issue is moot, as you don’t have a choice. 🙂
-
[quote:4567494171=\”tomasr\”]Joel,
What you say makes sense, but that’s not why I said what I did. Basically what I was hinting at is that if the OP wanted to keep his processing code like he’s doing now, then I don’t see him as having much choice at all.
While the serialization hit would be big (you’re right about that), trying to call a webservice 200 times to process the batch of messages inside a single atomic transaction would probably be far worse.
It sounds like perhaps his best bet is to skip trying to call the pipeline inside the orchestration at all and instead just disassemble it inside a real receive port and then let BizTalk do its thing in the \”old way\”. It would still be pretty heavy (processing large message batches always is), but at least BizTalk already has better mechanics in place to handle it.[/quote:4567494171]
Agreed. 😉
We recently had to redesign our batch processing solution due to performance concerns with using callable receive pipelines. It’s too bad there isn’t a built-in mechanism for invoking a receive pipeline that acts *exactly* like a pipeline bound to a port; i.e. the messages are put into the message box. Implementing a correlation set to receive the resulting messages from the pipeline would hopefully be trivial, perhaps if there were a way to use the ReceivePipelineID for instance.
-
[quote:228df42617=\”Dave\”]The reason I would like other suggestions.. is I have found performance to be sub-par. I think its the web service being called 250K times.. vs the pipeline however.[/quote:228df42617]
Here is a suggestion, though it might not suit your needs.
It sounds like your Web service is returning a single field from a SQL table and that field contains, essentially, a flat-file. If you want each record to go into the message box instead of being held in memory – i.e. external port/pipeline vs. called pipeline – what you might want to do is this:
1. Have an orchestration that queries the Web service or SQL database and writes the contents of the field to a send port bound to the FILE adapter.
2. Create a receive port that picks up the file and splits it using the flat-file disassembler. Each record will now be put into the message box.
3. Create an orchestration that receives these types of record messages and does something with them. If you need a convoy, you can correlate on the BTS.InterchangeId, as all the records from that \”file\” should have the same BTS.InterchangeId.
While you are going in-and-out of BizTalk, you are at least taking advantage of the BizTalk messaging engine and i.e. SQL, clustering, multiple hosts etc. Needless to say, 250,000 records is a LOT of records and invoking a pipeline from an orchestration might not be the best solution.
Another alternative would be to use one-way ports. The Web service receives a one-way request and calls back to BizTalk using another one-way port. You can use WS-Addressing’s ReplyTo header or a custom SOAP header so that the Web service knows where to return the response. You can then create a receive port that uses the flat-file parser to split the \”file\”. Your orchestration can then use correlation, convoys, direct-bound ports, or whatever to receive these records and parse them.
[b:228df42617]Edit:[/b:228df42617]
I think I misunderstood your response/initial post. Are you invoking a Web service to get the single SQL field containing the flat-file or are you using the SQL Adapter?
-
I am using the following code:
[code:1:0377daf02a]RcvPipeOutput = Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(BizTalkFromWipToAC.SQLParseReceivePipeline), OutputInterchange);[/code:1:0377daf02a]
This recquires that it is run inside an atomic scope. So all my logic afterwards, the loop of the records and message assignments and the sending to the web service are occuring in the atomic scope.
The web service was changed to provide a response. Atomic scope does not allow receive message of web service inside the scope. How do I use the above code to receive a message through a pipeline and still allow me to send the results to a web service that sends a response.
Here is more detaile code:
1. Receive SQL Message
2. Transform to Flat File format
3. Issue Send pipeline code to put in FF form.—Atomic Scope Start —
4. Run Receive Pipeline
RcvPipeOutput = Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(BizTalkFromWipToAC.SQLParseReceivePipeline), OutputInterchange);5. Begin Loop with : RcvPipeOutput.MoveNext()
6. Build the message:
FFMessage = null;
RcvPipeOutput.GetCurrent(FFMessage);7.Transform message to format I want
8. Convert to message for Webservice call
xmlDoc = XMLPrepMessage;
WebServiceMessage.xml = xmlDoc.OuterXml;9. Send to Webservice
9a. Response from Web Service
— End Atomic scope
So 9a fails.. how do i get this to work?[/code]
-
I already attempted that and had issues with the loop not allowing it to be sequenced… as the pipeline dies when the atomic scope dies.
-
I appreciate all your comments.
I am doing it this way because of the data I am trying to process. I could not find a better way to do this, so my solution was the send/receive pipeline calls.
I get data from a single SQL field. Its comma delimited with nested fields and rows inside those comma fields. I could not figure out how to use the SQL Adapter and map a parse of the data into XML.
So I read the data from SQL. Map it and send it to a Flat File pipeline that add CR/LF to the structure. I then read in the message on a receive pipeline that parses the Flat File structure into all the XML nodes.
We are talking large amounts of records. It is about 200-250,000 records per file. Although.. the SQL call can return any subset of data as a count.
I was able to get around the issue by sending a message out of the atomic scope to another orchestration that then calls the web service. I used direct bind ports for that.
If you have suggestions on how to optimize this I would love to hear ideas. But as is, I could not find a better way to map SQL field (whcih is essentially a comma delimeted text file record) to XML without sending it on a sendpipeline to make it appear as FF, and then back in to parse to xml.
Thanks…
-
-
-
-
-
-
-
-
-
-
-
-
- The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.