I’ve seen some posts on some discussion boards and blogs and support cases opened on this BTS2013 issue so I just wanted to let everyone know that Microsoft is aware of the problem and working on a permanent solution.
I also wanted to provide some general information about the problem and what your workaround options are in the short-term.
Periodically, your BizTalk host process crashes with the following errors in the eventlogs. Note the error code is 80131544. If you see the same error with a different code, you’re likely running into a different issue.
Also, notice none of the errors come from event source BizTalk Server. The ones in the Application event logs come from .NET Runtime and Application Error and the one in the System event logs comes from Service Control Manager.
Log Name: ApplicationSource: .NET RuntimeDate: 9/20/2013 3:47:42 PMEvent ID: 1023Task Category: NoneLevel: ErrorKeywords: ClassicUser: N/AComputer: servernameDescription:Application: BTSNTSvc64.exeFramework Version: v4.0.30319Description: The process was terminated due to an internal error in the .NET Runtime at IP 000007FDED170BC1 (000007FDECE00000) with exit code 80131544.
Log Name: ApplicationSource: Application ErrorDate: 9/20/2013 3:47:42 PMEvent ID: 1000Task Category: (100)Level: ErrorKeywords: ClassicUser: N/AComputer: servernameDescription:Faulting application name: BTSNTSvc64.exe, version: 126.96.36.199, time stamp: 0x50fe567aFaulting module name: clr.dll, version: 4.0.30319.19106, time stamp: 0x51a512d4Exception code: 0x80131544Fault offset: 0x0000000000370bc1Faulting process id: 0xca8Faulting application start time: 0x01ceb6394f1dd32aFaulting application path: C:\Program Files (x86)\Microsoft BizTalk Server 2013\BTSNTSvc64.exeFaulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dllReport Id: 830374f6-222d-11e3-93f8-00155d4683a2Faulting package full name: Faulting package-relative application ID:
Log Name: SystemSource: Service Control ManagerDate: 9/20/2013 3:47:43 PMEvent ID: 7031Task Category: NoneLevel: ErrorKeywords: ClassicUser: N/AComputer: servernameDescription:The BizTalk Service BizTalk Group : BTSOrchHost service terminated unexpectedly. It has done this 2 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.
There is a change in the .NET 4.5 CLR that results in the BizTalk process crashing during XLANG AppDomain shutdown. XLANG AppDomain Shutdown is when the .NET AppDomain that contains the Orchestration Engine tears itself down during periods of inactivity or idleness.
The Orchestration engine’s AppDomain shuts down by default after 20 minutes of idleness (all orchestration instances are dehydrateable) or 30 minutes of inactivity (no orchestration instances exist).
BTW, if you see this same issue in BizTalk 2010, it’s because you installed .NET 4.5 in your BizTalk 2010 environment – which is not tested or supported. You need to be on .NET 4.0 if you’re running BTS2010.
The first and easiest workaround is to do nothing. Note that this crash happens during periods of idleness/inactivity so your orchestrations aren’t doing anything anyway. Also, Service Control Manager is configured to bring a BTS process up a minute after a crash so the process will come back up just fine and continue to process fine. The crash won’t happen again until after there is orchestration activity and another period of idleness/inactivity occurs.
Note that if there are other things going on in the same host (like receive or send ports) then I wouldn’t be comfortable letting the host crash since non-orchestration work within the same host could be impacted. In that case, you can separate out non-orchestration work into other hosts (that’s generally recommended anyway) or you can go with the below workaround.
Another thing to consider is that, depending on the Windows Error Reporting (Dr. Watson) settings on the machine, these crashes can build up dump files on the drive. The default location for these would be C:\ProgramData\Microsoft\Windows\WER\ReportQueue. Check for any subfolders starting with "AppCrash_BTSNTSvc" - you can delete them if you don't need them. So if you have limited disk space on the system drive, that might also be a reason to go with the below workaround.
If allowing the host to crash during periods of orchestration idleness/inactivity is not an option, the workaround is to turn off XLANG AppDomain shutdown. This is generally safe to do. The only concern is if you have an orchestration that calls custom code that pins objects in memory to the appdomain (not good in general), then not tearing it down periodically could lead to excessive memory usage. Still, any 24x7 environment will never have appdomain shutdown happening anyway and I’ve seen a number of environments that have very low latency requirements turn it off (since reloading XLANG engine after periods of inactivity is a perf hit to that first request).
So, to prevent the crash from happening altogether, here’s what you do:
Microsoft is working on a permanent solution and I’ll update this page once it is available.