- This topic has 6 replies, 1 voice, and was last updated 9 years, 2 months ago by
.
-
Posts
-
-
Hi all,
Forgive me on this, but I would appreciate some feedback on how others would handle and I also need to know (for my own well-being) that I'm doing this up to par.
I've been developing Biztalk solutions for about 3 years now, and to this day, I question myself on if I've done it right. As for the overall definition of EAI, I am a strong believe that my integration layer should NOT be responsible for data issues with subscribed applications that may receive a transaction that may not quite adhere to the applications business rules. Right now, this is the case and my support load is devastating as I am the one being notified if a feed fails on an application. I do not believe that I should be responsible for the customer's data and I push "garbage in, garbage out" but I have received push back from the administrators of the application that are subscribed to my integration layer, main because they feel that I AM responsible for the data since it's my integration that feeds it to them.
I have a single logging database within the integration layer, dubbed as the 'Blackbox" as to simply trap the life-cycle of every transaction and to monitor the internal workings of the integration. I have a web app built on top of the database for our support team to simply look over what happened, but I do NOT want to put in so much logic in my integration that I need to know intimately, what a subscriber expects for business rules as I'm only responsible for delivering the data, not the outcome of the data…isn't this what the definition and goal of 'loose-coupling' EAI is?
Also, I have several subscribers that have given me web service portals as to deliver data, but again, it is my responsibility to assess the response of the adapter and report on the application's exception if there is one, again, this places the responsibility of the customers issues in my pocket and I have enough support issues.
Can anybody share some ideas or if needed, slap me in the face with some reality if I am incorrect in my beliefs of how my integration role should play in the field?
Thanks
-Rich
-
-
Who is the originator of the data here? If it is you then messages getting pushed back to you have some reasoning. If you are not, then push back all of the rejected messages (or subscriber exceptions) back to the originator (If it is a web application, then some how to that app) and keep quiet. Just like an honest post master.
In your scenario, "loose coupling" is what the cause of the problem is. You said, you are using Posting for subscribers. In such cases, if I were you, I'll hold the client (customer) until I post the data to subscriber, get a sync response from subscriber and post the same response back to customer so that he knows what the problem is and can take corrective action or re-send the data with correct information. Note that, customer is still waiting until you do all this in between. Meanwhile, you can simply note down which message came in and which message went out, just for the sake of reporting purposes.
data data
Customer ——–> You ———–> Subscriber
<——– You <———-
Sync Response Sync Response
You don’t need loosely coupled Integration, but a simple point-to-point synchronous integration. This will work only if the size of the data is not very big since you'll have to hold the customer's connection open until subscriber sends you exception response.
Technically, I'll create an orchestration (and publish it as a web-service) which does nothing but to receive data and send it to subscriber, receives response from subscriber and sends this response back to customer without bothering about what data is getting exchanged and there by leaving the onus of answering the exception to customer and not you.
-
Who is the originator of the data here? If it is you then messages getting pushed back to you have some reasoning. If you are not, then push back all of the rejected messages (or subscriber exceptions) back to the originator (If it is a web application, then some how to that app) and keep quiet. Just like an honest post master.
In your scenario, "loose coupling" is what the cause of the problem is. You said, you are using Posting for subscribers. In such cases, if I were you, I'll hold the client (customer) until I post the data to subscriber, get a sync response from subscriber and post the same response back to customer so that he knows what the problem is and can take corrective action or re-send the data with correct information. Note that, customer is still waiting until you do all this in between. Meanwhile, you can simply note down which message came in and which message went out, just for the sake of reporting purposes.
data data
Customer ——–> You ———–> Subscriber
<——– You <———-
Sync Response Sync Response
You don’t need loosely coupled Integration, but a simple point-to-point synchronous integration. This will work only if the size of the data is not very big since you'll have to hold the customer's connection open until subscriber sends you exception response.
Technically, I'll create an orchestration (and publish it as a web-service) which does nothing but to receive data and send it to subscriber, receives response from subscriber and sends this response back to customer without bothering about what data is getting exchanged and there by leaving the onus of answering the exception to customer and not you.
-
The Sales application, and the development team/sales team owns the data in question from the subscriber. When the finalize a sale, a contract and wrapped up into an XML feed and fired off to a remote private MSMQ that I monitor. I have had too many issues with web services themselves so I only enforce MSMQ as the pick up/delivery standard.
The problem isn't necessarily with the data from the publisher being 'bad', but something not working 100% at the subscriber. Even with the situation you offered, at the end of the day, don't I end up taking on MORE responsibility of being the God of support for the actual application and then extending that and being responsible for reporting back to the original publisher just because a subscriber had a problem that really has nothing to do with the data being sent to it? It would be an issue with the application that failed on the data feed, not the publisher as all other subscribers handled the data just fine and as expected.
Here's an idea of what I'm doing
Sales Application ——–> BizTalk —————> FPO Application
|
———————–> ERP
|
———————–> ODSIn the above situation, the ERP and ODS consumption feeds into their databases are successful, but the FPO app, although my message delivery is successful, fails on the data for any reason. If it's an error in the FPO application, and NOT the integration, who's responsibility is it to be 'aware' of the issue with the app? Should I proactively be responsible for letting the app admin know, or should they have their own hooks in place to monitor for issues? How am I responsible if I simply play the post master role and I have successfully delivered the message? Hence, "Don't shoot the messenger." 🙂
I can see where the situation you offer can be helpful, but in the world of EAI, at least from what I've read and done in the past few years, and forgive me, I'm not knocking your idea at all, but all this does as I see it, is extend the connection of the publisher by using Biztalk as a glorified adapter with direct point-to-point between two applications. Isn't the 'technical' definition of EAI encapsulated by the whole loosely coupled solution by utilizing a message broker to distribute published data to multiple subscribers? The application involved shouldn't know anything about each other, nor should the integration know anything about the applications other than the format of the data it needs to accept and send out.
Again, I'm only putting out what I got out of the whole deal of EAI/SOA. I'm not challenging anybody's knowledge, just putting mine out to be tested.
-Rich
-
-
-
I have only been working with BizTalk for 6 months or so but fall into your school of thought. I find BizTalk can do validation and such but not sure if it should. It may be my inexperience but there are several problems with BizTalk doing too much work to validate data.
1) The biggest can be seen by looking at who is complaining the most. In your example it is the receiver that yells the loudest when the data is bad. But the problem is, if BizTalk determine the data is bad, the sender will likely not accept responsibility for the data. They will complain just as loud if not louder if they have to support a design where BizTalk will reject their data and return it back to the sender. If the sender washes their hands of the data, what can BizTalk do if validation fails? Throw away the data? Create exception processing that the receiver must then design to, allowing it to accept error codes and descriptions. They will yell just as much if you expect them to process bad data via thier regular routines or if you expect them to code exception routines to handle special exception processing.
2) I find that many of the errors that BizTalk can detect, throws the transactions into HAT and the Orchestrations never see them. Thus you need to handle exception processing at the port level, orchestration level, and still need to keep and eye on HAT to get the rest of the error messages. I try to keep all my schema with string elements and do no validation. I pass the data I was sent. So far that has been acceptable. I expect the sender to use the schema to validate the XML before sending, I use a special version of the schema that treats everything as string, and then expect the receiver to use the real schema to validate the data. So far I have not had anyone complain like you have.
The real answer depends on who is your customer. if your customer is the company that creates the EAI solution and everyone else is anonymous publish and subscibe entities, they accept what is provided and you are responsible only for the EAI solution as defined by the "customer". If on the other hand, the people publishing and subscribing are the real "customer" you have to at least take their input into account.
-
Thanks for the response. Here's my argument, I should have been clearer on this to begin with now that I reread my original post.
My customers are all internal…different development teams, so one application (sales system) feeds an ERP, an ODS, a Data Warehouse and a Field Purchase Ordering system, for one example and all of these solutions are owned by their development teams, not by me. My EAI solution sits in the middle of it all as the message broker to distribute published messages from the sales system to the others I mentioned. Any exceptions that I run into are managed within Biztalk but that only covers my arena of message delivery (i.e.: msmq on remote server not available, web service is down, etc…).
The data, as far as schema and usability is good, so it's not a matter of the XML or schema being bad, more a matter of a subscribed application not being able to handle a feed for the following scenario:
1) A sales feed is published to the framework and Biztalk is responsible for picking up the message, formatting the message if need be, and pushing it along to the subscribed solutions. I have internal logging to a database that taps data from the running orchestrations simply to log what I've done and if I run into any issues. The feed is distributed to the downstream systems and as it sits now, I receive a notification from each subscriber after they process the data feed, ONLY as a logging solution…or that was my goal.
2) All other subscribed applications accept the feed and process the data correctly, except for one, we'll say the Field Purchasing Order app. For whatever reason, again this is not my application, the feed fails due to a business rule violation on the side of the subscribed application, not the integration. I receive an exception message and I log it on my end just so I am aware there was an issue and can reference the health of my integration layer.
3) The message from the published application is resent, and of course, the feed is again dispersed to all subscribed applications as I do not have any hooks in place to say, "this has been sent already and last time it was, only feed the application that failed last time.
Here's my argument on the scenario.
My responsibility is to deliver the messages correctly, and in sequential order, which I do now with a sequential convoy, once the message is essentially delivered, I should be the one washing my hands of the transaction. If step two occurs, although I receive notification of an issue, today, I am currently the one 'assumed' responsible for notifying the administrators of the application that failed, where an issue arose, so as the EAI developer, my role is now stretched into a babysitting role for the other developers and their applications. Again, my understanding is that, my role should end strictly at the point of data delivery to the subscribed solution's adapters and that's it, and I should not be accountable if their applications run into a data exception solely based on their processing logic. Sure, if the physical data is bad, I take responsibility, but only at the level of a message broker, after all…garbage in, garbage out…I should not build the EAI to be smarter than the business process and if there is bad data, then the external user community needs to realize there is a broken process and fix the root of the failed process, but not rely on the integration layer to just 'know' how to handle a user's boo-boo. Lastly, if a feed is sent through the integration more than once, again, the integration layer should be not responsible for tracking every single feed and contain logic to know, "Oh, someone sent this before, I need to dynamically adjust my output to support the fault of one application". Again, that puts the responsibility of support into the integration and a fully disconnected, "loosely coupled" integration layer should NEVER be that intimate to the subscribed applications.Whew….hopefully you're not asleep by the end of this 🙂
Please, anybody who may have an opinion as well, join in. I'm really struggling with this as I've read in several articles the exact opposite that ALL EAI solutions should be fully aware of the subscribed solutions and assume full responsibility of them by extending the umbrella of exception handling beyond the delivery port.
-Rich
-
-
- The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.