Is the SQL Agent running on the BizTalk database server. There are a number of jobs that cleanup references etc. If these jobs are not running or are failing then you are unable to terminate some instances.
If the SQL Agent is running and all jobs are successfully completing then you may need to get the MessageBoxViewer out of its holster. http://blogs.technet.com/jpierauc/pages/msgboxviewer.aspx