I’ve seen this before too, but only when the Biztalk SQL Server database is on a server other than where you are running the Server Configuration tool.
I don’t remember all of the possible ways to cause it, but it boils down to
(1) permissions of the signed in user running the Server Configuration tool.
(2) DTC configuration of the machine where the Server Configuration tool is running
(3) DTC configuration of the machine hosting the instance of SQL Server.
In really ugly cases, like where one is Win2k and the other is Win2k3, there are some really obscure registry settings to be made.
See KB article 899191 if you think you may have DTC issues (you’ll know by watching the SQL trace when you add a send port)
Stephen covered some similar issues in a blog some time ago… check it out at http://geekswithblogs.net/sthomas/archive/2005/06/21/44361.aspx