A long running transaction is not controlled by the DTC and it's not a 2-phase commit one. Hence, there is no inherent rollback capabilities and you need to use the Compensation facilities in BizTalk to implement it by "undoing" your actions manually.
Because of this, usually a long running transaction will be composed of a set of discrete steps, some of which can be "independent" atomic transactions.
As for the serialization stuff, that's because an orchestration cannot be persisted during an atomic transaction (it can be persisted before the transaction begins or right after it commits, but not right in the middle). Because of this, you can safely declare and use non-serializable types within an atomic scope.