- This topic has 2 replies, 1 voice, and was last updated 9 years, 2 months ago by
.
-
Posts
-
-
We have a process where a MQSeries queue contains thousands of XML messages. Each message is a single requirement. Our application needs these messages combined. They want each XML message to contain 500 requirements. So I created an orchestration that reads in a message, adds it to the larger message and loops 500 times, reading individual messages and adding to the single message. After adding 500 messages together, it writes the message to another MQSeries queue and starts over. It continues until it runs out of messages.
Due to limited resources, I could only do tests with small batches so everything seemed to be working fine. When we moved to the test environment with its additional resources, I have messages disappearing. There are two effects that I am noticing.
1) After sending 65,861 XML messages through the process, HAT showed 33,303 messages with the text "Completed with discarded messages". Basically I believe this says the Orchestration completed successfully but discarded messages. This I do not quite understand. I do not see how the Orchestration can complete unless the MessageBox database has a significant delay in delivering the messages since I use lack of messages arriving as my signal to end the Orchestration. Then again, I would expect the Orchestration to end and another Orchestration to start up due to the subscription signaling the start of the Orchestration. Could the MessageBox be discarding messages during this gap of one copy of the Orchestratio ending and another copy starting?
2) The Event log noted 5 times that the messaging engine needed to throttle inbound messages due to stress levels. This occured 5 times in the 1 hour and 26 minutes the processing took place. It is not clear what is the cause of the stress levels. Is the Receive location, responsible for reading the XML from the queue and putting into the MessageBox, trying to write messages to the database too fast? Is the Orchestration not able to process the messages from the MessageBox fast enough? Both? And more important, what can be done about this?
Unfortunatly I do not have a good view of the output. When the 500 requirement XML message is created, it is sent to a remote system via a MQSeries channel and immediatly consumed. The channel statistics imply enough data crossed the channel to account for all the XML messages but there is no evidence on the remote system they were processed. Still trying to track this down.
If anyone has any ideas, I would appreciate it.
-
I time or count based loop is very prone to Zombies. That’s that these messages are. Messages in the Completed with discarded messages state are very easy to, well, terminate. If you end the Orchestration, they just go away.
I think a better approach would be to write all the messages to a SQL table when they arrive. Then, use a Stored Procedure to extract messages out in groups of 500. I think that would be a safer way to go.
Hope this helps.
-
Thank you for the reply, yes it turns out this is the very problem. With further research it turns out in 2004 there was a large number of people who had this problem. While the Orchestration does not directly read from the MQSeries queue, I assumed the performance hit would not be significant so used a 12 second delay functoid to determine when there were no more messages.
Apparently a message is read from the queue by the MQSeries adapter (possible transformation), sent through the Receive Location (perhaps more transformations), written to the BizTalkMsgBoxDb, read by the Pub/Sub service and routed to my orchestration. Seems 12 seconds was way too low resulting in many zombie processes and discarding 33,303 messages out of 65,861. WOW!
I changed the delay to 20 minutes and the very first test had a single Orchestration go zombie on me due to a 23:16 minute gap in processing one message and the next. At least that time I only discarded 67 out of 65,861.
Final result is I changed the delay to 60 minutes. This will process all message, batching them into groups of 500 but the final batch of messages will be delayed that 60 minutes while the Orchestration takes that time to figure out there are no messsages. For our process this will not be a real issue.
We did look at some sample of going to a database instead but we have a very limited time to fix this and could not consider an alternative unless we could not get this working and 60 minute delay does seem to resolve the issue. For the next release I may be able to revisit this design.
Thank you for the help.
-
-
- The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.