Inbound trading partner and document security

by | Jan 12, 2007 | BizTalk Community Blogs via Syndication

Hello all!


Please allow me to introduce myself, my name is Tony Bernard and I am a Senior Program Manager on the BizTalk Server team. I am taking over for Suren Machiraju who, as relayed in his final post, has moved on to new and exciting opportunities elsewhere within Microsoft.


I want to take this opportunity to thank Suren for his many years of dedication and leadership on the B2B team!


In this posting I thought I would share a little trick for implementing trading partner and document security on inbound transmissions. By default, the EDI implementation in BizTalk 2006 R2 (which I will simply refer to as R2 for the remainder of this posting) will accept and validate all inbound interchanges as long as the proper schema is deployed and there is a receive location with either the EdiReceive or AS2EdiReceive pipeline specified. I’ll call this approach an ‘implicit accept’ approach since all transmissions are by default processed assuming the appropriate artifacts are deployed.


So what do you do if you want to take an ‘implicit deny’ approach to processing interchanges? And how would you do that at both a trading partner level (i.e. only allow interchanges from specific partners) and at a document level (i.e. only allow certain documents from approved trading partners)?


Receive Port Authentication


One way to implement partner level security is to leverage the Authentication settings on the Receive Port. In R2 the EDI receive pipeline will compare security and authorization fields in the incoming interchange envelope with those specified in the party. If valid, the interchange will be accepted and processed. If not, the interchange will be handled according to the settings in the Authentication section on the General page of the Receive Port properties.


Partner Level Security


But what if you aren’t using the security or authentication settings specified in your EDI standard (which many, arguably most, don’t) or for some other reason don’t want to turn on Receive Port Authentication? Well that’s where our little trick comes in. There is a property on the EDIFACT and X12 Interchange Processing Properties pages of the EDI Global Properties called Target namespace. The primary intent of the property is to facilitate the ‘implicit accept’ behavior mentioned above by ensuring that all inbound interchanges would get assigned a valid schema namespace. However, the same field can be used to implement ‘implicit deny’ behavior as well. By setting this field to a dummy or ‘dead letter’ namespace (i.e. one in which there are no schemas) by default all inbound interchanges will be assigned that namespace unless they are successfully associated with a specific party agreement by the EDI receive pipeline. They will then be suspended by BizTalk due to a missing or invalid schema. Therefore, the only documents that will make it through will be ones which have a party agreement implemented.


Document Level Security


Now that we have discussed partner level security, let’s talk about how to implement document level security. The scenario here is that you’ve implemented a partner for certain document types (say purchase orders), but don’t want to accept other documents (say advanced ship notices or invoices) that may still be in testing. The approach is very similar to that for partner level security in that it leverages namespaces. However, for this trick we will work within the party-specific settings for that trading partner. As in the EDI Global Properties, there is a property where a default namespace for the party can be specified. This property is labeled Default Target namespace and can be found on the EDIFACT and X12 Interchange Processing Properties pages of the EDI Properties for a given party. Again, the primary intent of this setting is to ensure that all documents for this party get assigned a valid namespace. However, by leverage this property in conjunction with the Enable custom transaction set definitions grid just below it you can implement the same behavior with individual document types as you did with partners as described above. Simply set the Default Target namespace to a dummy or ‘dead letter’ namespace and then create entries in the grid below for the documents that you want to allow to be processed. This is analogous to creating a ‘white list’ for spam filtering. Documents that don’t match the ones on that list will be assigned the default (i.e. dummy) namespace and will be suspended.


So there you have it. Two techniques to enhance the security of your R2-based EDI solution. That’s all for now, but as always, feel free to keep the questions, comments and suggestions coming.


Cheers,


Tony

Disaster Recovery Part 2

by | Jan 12, 2007 | BizTalk Community Blogs via Syndication

Slight disclaimer. The team in charge of putting our docs online has managed to break the content so the links below are accurate, but all of the “steps” on how to do the stuff has been excluded from the online content. 🙁    We are working on fixing this so hopefully it will be fixed by the time you read this and then I will remove this comment 🙂 


Okay, so some people noticed that I deleted pretty much all of the content from an earlier post on DR because, well, it was no longer valid. Most of it was sample scripts that were now replaced by supported shipping versions of those scripts. However, I guess there was some information that was still useful so I will put back a few comments here and address a question that was raised about the correctness of changing the Recovery Mode for our database / when can you do this.


 So first, let me say that the online documentation should be extremely good for our Disaster Recovery at this point. Michael McConnell, Shu Zhang and I spent a good bit of time making sure we covered every detail we needed and tried to order it so that it was more readable. If you have any complaints about the docs, please send feedback via this blog posting. I will get it to the appropriate person (Michael) to update the docs if necessary. You can find these docs at http://msdn2.microsoft.com/en-us/library/aa562140.aspx.


I will not repeat everything which is in the docs here, but most of the docs are spent explaining how to setup our backup jobs, how to configure log shipping, and how to restore the system in the event of a disaster. A number of companies have had their DBAs complain about having to use our backup process versus using their standard process which is to backup a database at perhaps 1am every day and then take periodic log backups. This link (http://msdn2.microsoft.com/en-us/library/aa577848.aspx), a sub topic of the above link, gives a good explanation of what we are doing … transactional log marking. The problem with what the DBAs are doing is that they are treating each db as a completely separate entity. BizTalk, though, will at times use Distributed Transactions to ensure reliability of actions across databases. If you treat each db as a separated entity, then if DB1 and DB2 participate in a distributed transaction together and I do a log backup of DB1 before the transaction commits and a log backup of DB2 right after the transaction commits, then when I restore the two of them my environment will be in an inconsistent state. For that purpose we use transaction log marking with “restore to mark” for all the databases in our environment (as described in the previous link and in many online sql docs).


There is no magic here. This is all standard TSql. You can read through our code and figure out everything that we are doing. We never install custom components on database servers for doing external procedure calls and we never use xp_cmdshell so this is all code that any database developer could have written, it is just that most never seem to care about DTC (even if maybe they should).


The question has come up as to are there any alternatives. The quick answer is that if you want to use log backups, then no, there are no other alternatives. You would have to write all the same code that we do so you need to use the code that we wrote. However, if you don’t really care so much about the data in your system and are okay with, say, 24 hour data loss windows (your data loss window with log backups is the interval between log backups), then you could optionally just have a period of time every day where you shut the entire system down and then just do full backups. Since there is no activity going on in the system when it is totally down, then you don’t need to worry about distributed transactions. To shut the entire system down, you need to shutdown all BizTalk Host Instances, make sure no traffic is coming in via Isolated Hosts (for example http / soap receive adapters), shutdown the sql agent processes on the various databases, and make sure that no other administrative tasks are on going (like changing configuration, redeploying, adding a new sendport, …). You should be able to verify that nothing is happening on the system via SQL profiler or some other device monitoring the databases. Once the whole system is down, you could take a backup of all of the databases in the system. To get a list of all of the databases which the BizTalk backup process was going to backup, you just need to “SELECT * FROM admv_BackupDatabases” from the BizTalkMgmtDb database. That is a view which all of our backup code uses during our DR process. If you add custom databases to our backup process (http://msdn2.microsoft.com/en-us/library/aa561198.aspx) then it will also show up in this list of databases.


At this point, if you are chosing to have a 24 hour data loss window and use scheduled, daily downtime to create a complete set of full backups for recovery, then you should also change the recovery mode on the databases to SIMPLE. This will automatically manage the size of the transaction log by essentially doing “truncate log on checkpoint”. By default, our databases are all set to FULL recovery mode which will only truncate the log when the user performs a backup of log or when the user manually forces the log to truncate (ie BACKUP LOG WITH TRUNCATE ONLY type command). There are lots of docs out there on this, just search for SQL Server Recovery Models. The key is that if you leave it in FULL recovery mode and then don’t do log backups (with our job), then your log will grow unbounded and you will eventually run out of disk space.


Hope that helps to clear up some confusion.


 Thx


Lee

Some items in the removed assembly are still being used by items not defined in the same assembly

by | Jan 12, 2007 | BizTalk Community Blogs via Syndication

When trying to redeploy a BizTalk 2004 solution I received the following error message:

“Some items in the removed assembly are still being used by items not defined in the same assembly, thus removal of the assembly failed. Make sure that items in the assembly you are trying to remove fulfill the following conditions:

1. Pipelines, maps, and schemas are not being used by Send Ports or Receive Locations

2. Roles have no enlisted parties.”

When you execute the following query against the configuration database, the result gives you the name(s) of the BizTalk artifact(s) which block(s) the deployment of your solution:

select ‘RcvPort’ PortType, r.nvcName Port, item.name MapName,assem.nvcName Assembly, nSequence, indoc_docspec_name, outdoc_docspec_name from bts_receiveport_transform rtinner join bts_receiveport r on rt.nReceivePortID = r.nIDinner join bt_mapspec ms on ms.id = rt.uidTransformGUIDinner join bts_assembly assem on ms.assemblyid = assem.nID inner join bts_item item on ms.itemid = item.id
union
select ‘SendPort’ PortType, r.nvcName Port, item.name MapName, assem.nvcName Assembly, nSequence, indoc_docspec_name, outdoc_docspec_name from bts_sendport_transform rtinner join bts_sendport r on rt.nSendPortID = r.nIDinner join bt_mapspec ms on ms.id = rt.uidTransformGUIDinner join bts_assembly assem on ms.assemblyid = assem.nID inner join bts_item item on ms.itemid = item.id
order by PortType, Port, nSequence

In my case a Send Port with a filter (which contained a reference to a schema) blocked redeployment. Removing the filter from the Send port was enough to be able to redeploy.

Service Broker Adapter v1.5

by | Jan 11, 2007 | BizTalk Community Blogs via Syndication

After compiling a lot of feedback from Microsoft and customers our team started working in the next version of the Service Broker adapter for BizTalk Server 2006. Yesterday we finally made available the new version in AdapterWorx (Read Joel Post). This is not a major release in terms of new features but includes significant enhancements to existing features like dynamic ports and transactional receiving.

I am still looking for testers for the R2 version of the Service Broker Adapters so if you are interested drop me line at [email protected]

Advanced Message Queuing Protocol (AMQP) and Microsoft BizTalk Server

by | Jan 11, 2007 | BizTalk Community Blogs via Syndication

Recently I’ve read the AMQP (Advanced Message Queuing Protocol specification http://www.amqp.org/). Obviously this is the different protocol level then BizTalk messaging system but…


The AMQP specification let me get some thoughts.


And take in mind this topic is not a critic of AMQP or BizTalk, not at all. And this is not a comparing them, maybe sometimes 😉


After discussion with Tomas Restrepo http://www.winterdom.com/weblog/ I’ve made several changes in the text. Thanks Tomas!


AMQP:





Thoughts:

 


  •  AMQP does not include the “Transformation” of the messages. Otherway the paradigm of the BizTalk is “All outer formats are transformed to/from the unified (Xml) internal format.” I think the “adapter” and “pipeline” and “map” paradigms in the BizTalk are quite productive and helpful. They can be implied in the Exchange but anyway AMQP does not have the transformation API. All implementations of transformation API will be out of standard.
     

  • AMQP distinguish the different queues. The queues deliver the messages to the consumers. In BizTalk this is responsibility of the send ports. Inside BizTalk all queues are unified and acts as one queue. We cannot distinguish different queues by API or UI.

  • In AMQP the “consumer-cardinality” is the feature of the Queue or it seems so. I mean the Queue can have one consumer or many. In BizTalk the queue (MessageBox + agents) doesn’t care about consumers, it is the feature of Subscription agent (which works like the Exchange in AMQP).

  • In AMQP the paradigm is “routing –> many queues”.
    In BizTalk the paradigm is “transformation –> one queue –> routing –> transformation”.
    (Or is it “transformation –> routing –> many queues –> transformation”? We don’t know, because the BizTalk try to hide the queue implementation details.)




What is interesting in AMQP:

 


  •  The File Content class. It works as a message type to perform bulk file transfer. It is the case when we do not transform the message only route it and all routing data are outside the message (in binding parameters, for example). In BizTalk we have to do a lot of work only to transfer the bulk file through the system. This is the continual case too. This is the problem of the BizTalk: it always pretend we always have to “route + transform” the messages, no such case as “route, do not transform” messages.

  •  The Stream Content class. It implements the order delivering.

  •  The “Topics” Subscription. It uses the wild-cards for routing, like “STOK.USD.*” against routing parameter values as “STOK.USD.NYSE”.
    Can we use the wild-cards in the Filters in BizTalk? No. But I think there is a simple implementation of the Topic subscription and much more if we could use the custom filters in BizTalk. Why do not add the new feature to BizTalk the “Users predicate” as it implemented for Functoids with Script functoids, for example?

  •  The message priority in the Queue.

  •  The processing of the messages with zero size body.

  •  The Standard for Error handling. In BizTalk we have many places where we got the errors and there are no strict rules. For example different adapters have different error handling when we sometimes get exceptions, sometimes the suspended messages. It is a “gray area” in BizTalk.


Please, feel free to critique and comments.