I was finally able to resolve the issue .
I have identified the following in the system
1. Time was not sync among the machines in the production
2. Unexpected behaviour in the delay shape
3. System is throttling because of high DB (Message Publishing Throttling State is 6 in the system)
The system performance was increased considerable amount after doing the time synchronization. But still delay shape behaviour was there. So I removed & ran the application. I stopped with database deadlocks.
So I went for an alternative approach. The business process (orchestration activity) was divided in to two processes. The main orchestration invokes the nested orchestration synchronously. The enclosing orchestration waits for the nested orchestration to finish before continuing. As this is inside the loop, the next cycle has to wait till the sub orchestration finished. So this is kind of waiting or reducing the number of open transactions to the external database. I used “Delivery Notification = Transmitted” at the send port.This approach was really success one. It increased the system throughput tremendously.
I have noticed some required BizTalk Jobs are not running in the SQL from last year. No idea why it has been stopped .Once I ran it, system stopped throttling. But need to do further analysis. Because could not find old events records in the system event viewer.
After doing the above, the system is quite good. It is running smoothly without any issues so far.