First you have to install your new database server an leave it as it is. No installation of other databases are required so far.
Unconfigure on the BizTalk server SSO, Groups an whatever you need.
Just configure BizTalk new – also the parts you have unconfigured before.
The first thing you have to configure is the SSO and you will see on the database server, that there will be a new table automatically installed.
If this isn’t working properly, please check the log.
And one more thing. Check that DTC is enabled that means on the database side is Network DTC Access enabled with Allow Inbound and Allow Outbound and no Authentification Required. So it works for me.
I hope this helps.
Wolfgang