Architectural Issue in Designing Biztalk Services (Urgent Help Required)

http://www.biztalkgurus.com/Samples/Xml-Documents-via-SOAP.html

Here is my as-yet untested thinking about one solution your question. 

In a nutshell, a service consumer might be well advised to use a config file to identify the response schema it would expect from a service, rather than hardcoding the expected response schema. The consumer's config file could be easily changed.  We're not done yet though:  The code in the consumer would need to be written in such a way, that minor additions to the schema would not invalidate the code that the consumer uses to read the response message, such as xpath queries or generated wrapper classes, some how.  It may be possible, and I'm given hope, because by comparison with SQL SELECT database coding, it certainly is possible to add output columns to a SELECT statement, and I have seen instances where the older consumer program can be written to blissfully ignore the additional columns it does not care about.

Below, I am also going to share with you my latest, best but untested proposal regarding the process of service development which led me to that suggestion.

A View of SOD Process, from Space

1. Get a use-case.
2. Make its messages schemas, fewest and biggest, around the schemas you already have.
3. Make its services, around the services you already have.

A View of SOD Process, at 20,000 Feet

1. Get a new use-case, just one. There is not much time.
2. Check the existing services and messages, for something to offer
3. Create messages first, new-ish*, to match the use-case, minimizing # of exchanges.
4. Design services, new-ish, to exchange the (new-ish) messages.
5. Stub the services, just to exchange the messages.
6. Build the service-consumer app to exchange messages with (stubbed) services.
7. Internal to the services, stub any new services they likewise need to use.
8. Internal to the services, orchestrate the existing real and stubbed services.
9. Fill in stubs with actual code.

A View of SOD Process, Hands-On

1. Get a new use-case, just one. There is not much time.
2. Review and identify existing relevant services which provide the messages which contain the data the use-case is going to need.
3. To match the new use-case, compose new “outer” (or “container”) message schemas, by attaching and containing existing messages together where possible, and adding new schemas where necessary.  Satisfy the use-case in the fewest number of back and forth message exchanges with the service consumer.  Message exchanges move across the network and require at least a pair of processes to react to, for each exchange, which slows overall throughput down.  By design we are trying to send fewer, but larger bundles of information.  This will tend to result in large-size outer messages, which is fine.   The outer messages will likely directly contain some inner messages which get re-used across different use-cases, such as customer-info, or order-info, or package-info, which is good.  By design we don’t pare schemas down to the minimum needed, custom-making new special purpose schemas; rather, we prefer to reuse schemas.  Some of the information in the inner messages will tend to be ignored in any given use-case, which is fine.
4. Identify new “public” service(s) to exchange these outer message schemas of this use-case, and stub them out.
5. Inside the public services, orchestrate (call or establish adapters to) the existing contributing services, and stub out any new internal services, or applications, or programs, or components.
6. Swap out the simple stubs to good stubs, using internal interface contracts to make the switch.  You can easily switch back to the stubs if later, with this technique.  You don’t throw away the stubs, but you keep them aside, and configurationally able to be reinstated at any time.  Unit test new code as you make it.

Evaluation: Why is This Process a Good Process for Service Development?

1. It results in chunky not chatty services, for faster consumer application responsiveness.  Each request-response cycle is a large overhead, so making each trip count is desirable.  School-buses get the kids to school faster than school-motorcycles would, even though motorcycles are faster than school-buses.
2. It results in thinner, easier to understand consumers, since service orchestration logic has mostly moved out and now lives in the services.  This is good because the consumer is the part that business people can see, and they will want to change the parts they can see. Changing a UI that is simpler to understand, is faster and will have less bugs.  It results in service interfaces that are more like the business, so it’s easier to understand by people outside the IT department, and it’s easier to map to the business user interface.
3. It results in a service interface that has less of the enterprise-paralyzing technical bondage to the particular databases, telephony equipment, CRM, payment processors that underpin the service interface.  You can change the underpinnings, with reduced disruption of the service consumers.
4. Interface-first development process, not implementation first, assures workable service contracts that meet the needs of business, on the most important programmatic interface, the enterprise’s SOA (or Service Oriented Interface).
5. Consumer development begins and ends soonest for two reasons.  One, consumer development does not wait for service implementation to be completed.  There are working stubs behind the service interface at first.  Simple static but schematically correct messages are sent out by the stubs.  Two, just one business use-case is the recommended granularity of an effort.  A single use-case is the smallest, fastest-to-live software artifact that a business user can directly use and still extract a business benefit.  Big bang development is more risky and slower to deliver.  The sooner business gets their hands on an iteration, the sooner it helps the business.  One use-case is approximately the first available but still-usable chunk of IT output which a business can use.

On Schema and Service Changes

There will be times when business drives an addition of elements to a UI.  This change could ripple, necessitating the addition of elements to the schemas which match the use-cases.

A practical precaution is to externalize the schema identifiers used during schema validations by the service consumers which do validation.  If a consumer already does schema validation, then the setting can be updated in the external configuration setting, so the consumers don’t have to be recompiled and redeployed when elements are added to schemas.

If the consumer does not already do any kind of schema validation, then adding new elements might simply not hurt existing code.  However, avoiding schema validation is a risky plan because new and different data could appear and be undetected in places in a message that were assumed to have something else before, resulting in bugs.

Another precaution is that virtual service and schema versioning systems such as AmberPoint can help isolate the change from other consumers that still depend on the older schema permitting different versions of the same schema to run concurrently.

Finally, it appears possible to avoid virtual versioning, just by treating the changed schema as a completely new schema rather than a new version of an existing schema, whenever a new element or some other change is needed.  Leave the other schema alone, don’t change it at all, and the other consumers will still work if they depend on the older schema.  A new web service method for the new schema may be required as well, to return the new schema without breaking callers of the previous schema.  You will accumulate a lot of schemas and service methods this way, as opposed to a lot of versions of them, and the difference between the two approaches might be small enough to be inconsequential depending on the tools you use.