Not really sure about your problem. But I’ve seen custom pipeline components actually crash BizTalk due to exception handling. I’m guessing there is something not being done correctly in terms of how the failed message is handled.
Just a guess though. I’d probably check the help guide and Charles Young’s blogs for some hints.