[Source: http://geekswithblogs.net/EltonStoneman]
We had an issue with one of our BizTalk estates with incoming messages being suspended if they were bigger than the large message threshold:
A response message sent to adapter “WCF-BasicHttp” on Receive Location: “x” with URI:”y” is suspended. Error details: There was a failure executing the response(receive) pipeline: “z” Source: “Unknown ” Send Port: “x” URI: “y” Reason: 0x8004d027
The error message wasn’t particularly helpful but the reason code 0x8004d027 had a few threads and posts relating to DTC. This made sense as we’re in a distributed environment so DTC will be used to create message fragments under a transaction, if the whole message exceeds the threshold size. Using Web Service Studio I found the expected message size, and setting the large message threshold above that figure fixed the problem so that seemed to confirm it.
We finally tracked the issue down to permissions on the cluster where MSDTC was running. Buried in the Event Log for the BizTalk host suspending the messages was this error:
The application could not connect to MSDTC because of insufficient permissions. Please make sure that the identity under which the application is running has permission to access the cluster. Please refer to MSCS documentation on how to grant permissions. Error Specifics: d:\nt\com\complus\dtc\dtc\msdtcprx\src\dtcinit.cpp:652, Pid: 6772 No Callstack, CmdLine: “C:\Program Files\Microsoft BizTalk Server 2006\BTSNTSvc.exe” -group “BizTalk Group” -name “x” -btsapp “y”
This was logged when the BizTalk service started, which had been the day before we found the suspended messages, so I didn’t see the connection. This post from Ben Cops: http://bencops.blogspot.com/2008/12/error-on-installing-sso-on-clustered.html pointed me in the right direction, and the steps for adding permissions are simple – the only oddity is that you set permissions at the cluster level, not at the resource level:
- Open Cluster Administrator
- Right-click the cluster (the topmost node) and select Properties
- Under Security, add the BizTalk user group with Full Control
This and a host restart fixed the issue. Along the way we did notice a couple of other issues in our MSDTC configuration which we tracked down and fixed with DTCPing. Worth noting that in Windows Server 2003 the properties for MSDTC under dcomcnfg may not correctly reflect the registry settings, so while you think DTC security is configured like this:
– it may actually be configured like this:
The DTCPing log will tell you what registry settings are in use, and the BizTalk Best Practices Analyzer will warn you if the DTC security settings are incorrect, so worth double-checking this.