by stephen-w-thomas | May 23, 2016 | BizTalk Community Blogs via Syndication
I have a scenario at work where we need to provide some simple syncronization between a SQL Azure table and a SharePoint Online Custom List. As a pre-requisite each morning before business users get into the office, we need to purge the contents from the SharePoint list and update it with today’s data + a 6 day forecast of future data.
I have integrated BizTalk with custom SharePoint Lists in the past, and even wrote about it in one of my books. It wasn’t a particularly good experience so I was interested in evaluating how Logic Apps would deal with custom lists.
One difference between BizTalk and Logic Apps, in this case, is that BIzTalk has a SharePoint Adapter but it will only interface with SharePoint Document Libraries. If you wanted to integrate BizTalk with SharePoint Custom lists you are likely going to do so with the Lists.asmx web service. While it is completely possible to use this custom web service approach, be prepared to spend a few hours (if you are lucky) getting everything working.
With Logic Apps, it is a very different experience. From the Logic Apps canvas you need to add a trigger to kick off your workflow. In my case, I used a Recurrence trigger that will run every day at 7:45 am. I can also kick this trigger off manually in the Azure Portal if I wish.
Next, I want to add an action and then search for SharePoint from the Microsoft managed APIs dropdown. After we do that, all available SharePoint Online operations will be displayed.
In my case, I want to purge all items, but there is no Delete List operation. Instead, I need to get all items in my list first, so that I can use the ID from each record to delete that item. In my scenario I expect 7 days * 24 hourly records (168 items) to be in my list at any given time so this load is not a concern.
With this situation in-mind, I will select the Get Items operation.
Once I have selected my Get Items operation, I need to establish my connection to my Office 365 subscription. (I will spare you from the password prompts)
With my connection established, I now need to provide the URL to my SharePoint List. I can also specify optional parameters that control the number of items returned.
I must say the experience in this dialog is a good one. I can click on the Site URL dropdown and all of the sites that I have access to will be in that list. Once I have selected my URL and then click on the List Name dropdown, I then see all the lists that I have access to on that site.
Next, I need to add another activity and this time I will select the Delete Item operation.
I have a similar experience in the Delete Item dialog that I had in the Get Items dialog. With my connection already established, I need to provide the same Site URL and the same List Name. What is different this time is I need to provide an ID for the list item that I would like to delete. In this case it will be an ID that is coming from my Get Items response.
You might be asking yourself – will how is that going to work for all of my items in my list? Don’t you need a for loop to iterate through the Get Items collection? The answer is Yes, but the Logic Apps team has made this very simple – they have done the heavy lifting for you. If you go into Code View you can see it there:
“foreach”: “@body(‘Get_items’)[‘value’]”,
“inputs”: {
“host”: {
“api”: {
“runtimeUrl”: “https://logic-apis-westus.azure-apim.net/apim/sharepointonline”
},
“connection”: {
“name”: “@parameters(‘$connections’)[‘sharepointonline’][‘connectionId’]”
}
},
“method”: “delete”,
“path”: “/datasets/@{encodeURIComponent(encodeURIComponent(string(‘https://SharePointsharepoint.com/sites/HOPOC/Bighorn/SitePages/Home.aspx’)))}/tables/@{encodeURIComponent(encodeURIComponent(string(‘c28b1ea2-e2a0-4faf-b7c2-3eerec21a8b’)))}/items/@{encodeURIComponent(string(item()[‘ID’]))}”
},
Awesome!
The end result is a Logic App that looks like this:
I can now my Logic App from the Azure Portal by clicking on Select Trigger and then recurrence.
I can follow my execution and dive into my Get Items and Delete Items calls. By inspecting my inbound and outbound traces I can see the exact payloads and HTTP Status codes from the underlying operations. In this case I was able to delete 300 items in 23 seconds. For each item in my collection, a Delete Item call is made.
Conclusion
It honestly took me about 10 minutes to figure this out. Part of the reason why I am writing about this is I know how long this would take with BizTalk. I anticipate it would take at least 3 hours to do this in BizTalk if it was your first time. So this isn’t a knock against BizTalk, as Logic Apps has been built, IMO, for these lightweight scenarios with little friction.
In the short term I think Microsoft Integration architects and developers will have many opportunities like this one where you can choose one tool or the other. For me, developer productivity needs to be part of that equation. Where the systems that you are integrating are hosted will also play a role. In this case, I am connecting to SharePoint Online and SQL Azure so it also doesn’t make sense, IMO, to route all of this info back on-premises only to go back up to the cloud.
We may also see a Hybrid approach where a BizTalk process can call out to a Logic App where you can take advantage of these new Logic App connectors. This was something that Microsoft discussed at 2016 Integrate Event in London.
In a future blog post we will talk about the other half of this solution which is how to create items in SharePoint Online from SQL Azure.
by Mick Badran | May 23, 2016 | BizTalk Community Blogs via Syndication
After a few customer engagements recently this topic has come up several times, some lean in closer and let’s have a chat on it.
How application infrastructure is evolving to support digital business?
The great thing in this modern era is that businesses are placing great pressure on traditional IT, integrators and solution architects to innovate and look for that next disruptive edge.
With a pea sized idea, access to the rapidly evolving cloud technologies businesses can disrupt even the most cemented industries. In fact many of my customer meetings are around disruption, “What/How can we disrupt today?”. The business climate has never been better & hence the birth of the phrase ‘digital business’.
Many companies that ‘push tin’ are scrambling to offer a range of other services as they also see the future is not in tin, not so much in technical expertise (while I do love being part of a great team, cloud templates can put short work to our previous ‘cluster’ expert), but in being agile! Taking an idea whether it’s IoT or anything else and realising the solution. In my opinion, this is the skill that will be much sought after in the market.
Integration is key here.
What about the application landscape? How has it changed?
Great question!
Figure 2 – Depicting the role of middleware
From what I’ve experienced is that over the past 20 odd years, application integration ‘layers’ (or middleware) were large, monolithic and usually cost a fair bit of $$. Associated with a software platform purchase was a good 9-18 month evaluation of the platform to see if it was ‘fit for purpose’.
(There were a whole series of VAN’s in this space as well – Value Added Networks, which made life simple to move data/messages through a particular industry vertical. This was partly because it was made complex by proprietary interfaces, and also the fact that the VAN providers wanted customer lock-in. So naturally everything to do with these environments was ‘hard’ and we let the ‘experts’ deal with it)
In these times many of the systems and applications all had custom ways to communicate. Software vendors reluctantly opened up ways of getting data in/out of their system. Communications standards were lacking, as well as message formats and protocols.
The fact that the middleware platforms communicated with a large number of systems from DB2, SAP, OracleEB, JD Edwards through to Pronto (ERPs) made attractive choices for businesses that saw the value in getting ‘end-to-end integration’ and a full 360 degree customer view.
Fast forward to present day…..
Software vendors are exploding at a rate of knots all over the web. Applications are more about functionality than specifically where they run – on-premises or cloud or…phone…or where ever. Applications today are modular and connected by well known interfaces – although the resiliency and interface workflows are lagging behind a little in this realm.
The software industry realises that mobile devices are king and they build applications accordingly. For e.g. the pseudo modern accepted message/data exchange JSON/HTTP (aka REST) based APIs. I had one of my team members complain when an interface they were talking to was SOAP. I felt like I was talking about tape backups 🙂
Given that Mobile is taking charge and software is providing better foundations for ease of mobile development and operation, you could call it the mobile evolution….
REST based APIs are accepted as the norm with OpenId/OAuth being token based authentication standards, allowing the beauty of delegated authentication (something that plagued many previous integration implementations in the quest for the elusive single sign-on capability).
Figure 3 – Illustrates the business need to maximise and provide a comprehensive set of APIs in which to monetise.
Businesses today are realising they don’t need mobile, they don’t need a website….they need an API!!! An API:
- to expose their business to public consumers
- to expose their business to down stream consumers
- to expose their business to upstream consumers
- to commercialise their offerings.
They now realise they can get away from building ‘yet another mobile app’ and focus squarely on turning data into information.
Software today needs to produce analytics cause gone are the days when we got excited at just being ‘connected’.
by Mick Badran | May 23, 2016 | BizTalk Community Blogs via Syndication
After a few customer engagements recently this topic has come up several times, some lean in closer and let’s have a chat on it.
How application infrastructure is evolving to support digital business?
The great thing in this modern era is that businesses are placing great pressure on traditional IT, integrators and solution architects to innovate and look for that next disruptive edge.
With a pea sized idea, access to the rapidly evolving cloud technologies businesses can disrupt even the most cemented industries. In fact many of my customer meetings are around disruption, “What/How can we disrupt today?”. The business climate has never been better & hence the birth of the phrase ‘digital business’.
Many companies that ‘push tin’ are scrambling to offer a range of other services as they also see the future is not in tin, not so much in technical expertise (while I do love being part of a great team, cloud templates can put short work to our previous ‘cluster’ expert), but in being agile! Taking an idea whether it’s IoT or anything else and realising the solution. In my opinion, this is the skill that will be much sought after in the market.
Integration is key here.
What about the application landscape? How has it changed?
Great question!
Figure 2 – Depicting the role of middleware
From what I’ve experienced is that over the past 20 odd years, application integration ‘layers’ (or middleware) were large, monolithic and usually cost a fair bit of $$. Associated with a software platform purchase was a good 9-18 month evaluation of the platform to see if it was ‘fit for purpose’.
(There were a whole series of VAN’s in this space as well – Value Added Networks, which made life simple to move data/messages through a particular industry vertical. This was partly because it was made complex by proprietary interfaces, and also the fact that the VAN providers wanted customer lock-in. So naturally everything to do with these environments was ‘hard’ and we let the ‘experts’ deal with it)
In these times many of the systems and applications all had custom ways to communicate. Software vendors reluctantly opened up ways of getting data in/out of their system. Communications standards were lacking, as well as message formats and protocols.
The fact that the middleware platforms communicated with a large number of systems from DB2, SAP, OracleEB, JD Edwards through to Pronto (ERPs) made attractive choices for businesses that saw the value in getting ‘end-to-end integration’ and a full 360 degree customer view.
Fast forward to present day…..
Software vendors are exploding at a rate of knots all over the web. Applications are more about functionality than specifically where they run – on-premises or cloud or…phone…or where ever. Applications today are modular and connected by well known interfaces – although the resiliency and interface workflows are lagging behind a little in this realm.
The software industry realises that mobile devices are king and they build applications accordingly. For e.g. the pseudo modern accepted message/data exchange JSON/HTTP (aka REST) based APIs. I had one of my team members complain when an interface they were talking to was SOAP. I felt like I was talking about tape backups 🙂
Given that Mobile is taking charge and software is providing better foundations for ease of mobile development and operation, you could call it the mobile evolution….
REST based APIs are accepted as the norm with OpenId/OAuth being token based authentication standards, allowing the beauty of delegated authentication (something that plagued many previous integration implementations in the quest for the elusive single sign-on capability).
Figure 3 – Illustrates the business need to maximise and provide a comprehensive set of APIs in which to monetise.
Businesses today are realising they don’t need mobile, they don’t need a website….they need an API!!! An API:
- to expose their business to public consumers
- to expose their business to down stream consumers
- to expose their business to upstream consumers
- to commercialise their offerings.
They now realise they can get away from building ‘yet another mobile app’ and focus squarely on turning data into information.
Software today needs to produce analytics cause gone are the days when we got excited at just being ‘connected’.
by Nick Hauenstein | May 22, 2016 | BizTalk Community Blogs via Syndication
In this post, I am going to try to capture in text the presentation that I gave at the Integrate 2016 conference over in London. Text is likely the worst medium in which to capture such a session, but, alas, I do realize that sometimes it is the best medium for proper digestion of such content. If you’d rather see it in video form, click here.
So with that, let’s pretend that you are sitting among fellow professionals in a beautiful room on the 3rd floor of ExCeL London – complete with bright colored lights to set the mood. A wild American then appears, flailing his arms and babbling about how it’s actually 3 AM, and we’ve all been deceived. Then he starts talking about food.
Getting Our Priorities Straight
The world of software development might be a better place if we approached our tasks in that world the way that we approach each meal. We don’t really start each meal by a trip to the racks or shelves that hold our appliances – thinking, “Well, I have a vegetable peeler, and a fondue pot, I guess that means we’re eating some melted Gruyère and Emmentaler mixed with white wine, and carrot strings for every meal.”
Usually, the way it actually works out is that I’m thinking about what I’m craving, the ingredients that I have on hand and their flavor/nutritional value relative to my needs. From there, I look to proven recipes that satisfy those things, and finally, reach for the specific tools needed to do the job. If I don’t have them, I acquire them, or fashion a workable approximation.
We have to be really careful that we, when approaching software development and integration, take the same approach as we would when crafting an excellent meal. An approach that looks first to the needs and constraints, then to proven patterns/recipes, and allows the tools used to flow from the rest – even to the point of crafting/buying new tools that we haven’t used before if necessary.
Business Challenges / Constraints
So, let’s imagine that we all work together now. We want to take the approach outlined above – one in which we have to consider the business need and the constraints that we may very well simply be stuck with. From there, we can consider proven patterns that might help us overcome, and then finally identify/acquire/create the tools required to get the job done.
Our company makes custom bobbleheads.
The way that it works is that a customer uploads a 3D model of their face, and then selects a pre-built body from the gallery. The 3D print of their face starts immediately so that the order can be shipped as soon as possible. The customer is permitted to take as long as they need to select the body from the gallery of pre-built bodies. Once they select a body, we attach the printed face to the chosen body and ship the assembled dolls to the customer.
So what happens behind the scenes that we can’t escape?
Well, sadly we don’t do greenfield development here. It’s not really brownfield development either. It’s more like a house haunted by ghost IT – shadow IT that has left.
Whenever a customer uploads their 3D face model, an XML notification message is created that contains the order id and a reference to their face model. At the time it was built, our developers emulated the BizTalk Server demos of the day and built distributed, fault-tolerant XML file copy operations whilst applying the wisdom of Chris Maden who has been quoted saying, “XML is like violence. If it doesn’t solve your problem, you’re not using enough of it.”
These same developers learned while attending conferences over the past few years that Dropbox is the next big thing in enterprise integration. They may have been wrong, but it’s now up to us to Make Dropbox Great Again™.
Once the customer selects a body for their doll, another XML message is created and dropped into the same folder in Dropbox with the same file naming conventions. Ultimately, we need to pair up the two components of the order – the head and body – in order to complete the processing.
Proven Patterns
It’s at this point that we would consult the great oracles of all wisdom and knowledge in the world of integration, Gregor Hohpe and Bobby Woolf. We will search through the patterns to find pieces that help solve each piece of the puzzle.
Which patterns might we find? Well, to handle the communication with Dropbox, we might utilize the adapter pattern in hopes that one day the data will be sourced from a different system. We could apply the pipes and filters pattern and build a reusable translation/transformation pipeline made up of reusable independent steps organized in the proper order to provide the required translation/transformation/message enrichment for each interface.
From there, we could apply the publish and subscribe pattern to enable loosely coupled communication between the source system and any number of downstream subscribers – maybe routing the message in a content-based fashion. We could also layer on top of this a process manager to enable content-based correlation.
Tools
How would we use these patterns in concert with the tools we have/don’t have? BizTalk Server might seem like a natural fit.
It already provides for us the concept of a port that begins with an adapter which delivers a byte stream to a pipes and filters style pipeline responsible for translation of the message and promotion of context properties used for routing, this is followed by a transform before publishing to the message box. From there, we have process managers in the form of BizTalk Orchestrations that understand the concept of content-based correlation of published messages, allowing them to be reunited by the messaging engine. You get adapters, pipelines, maps, orchestration out of the box – and publish subscribe whether you want it or not!
It’s already bringing to the table everything I need. Everything but a handy Dropbox adapter. Now, I know that we could always build our own, or use out of the box adapters with ungodly amounts of WCF extensions to make some magic happen, but maybe that won’t be our best bet here.
So, let’s set that aside for a moment, and consider what might become possible if we started using Logic Apps like this? It’s really the same question I posed before about MABS.
In this case, we’re marrying Logic Apps and Service Bus. We have some Logic Apps that act in a similar fashion as BizTalk Ports. They provide adaptation and message enrichment and transformation through the use of relevant API apps for those concerns. Others act more like BizTalk Server orchestrations, coordinating the sends and receives of messages and operating on the content.
The messages are routed to the “orchestration” style Logic Apps through Service Bus. Each flow is triggered by subscribing to messages that arrive on a given Service Bus topic subscription (pre-created). Correlation can then be enabled by subscriptions dynamically created mid-flow.
At this point, you may have the following thought (which I humbly share indeed):
Demo Walkthrough
This isn’t all just a pipe dream – it’s real. I’ve built it. So, let’s see how it can fit together. The flow kicks off with an XML message. For this message, I have created a BizTalk Server 2016 schema (i.e., a regular XML schema with special notations about properties that should be promoted to the message context for routing purposes). The message looks like this:
The message contains a promoted OrderId property that we should be able to correlate on. In other words, the second message that will show up in Dropbox for the order should also contain the same OrderId value – which allows us to determine that they are indeed related messages. The first message also contains a reference to the head for the bobblehead doll that we will be printing.
When this message is uploaded to Dropbox, it will be picked up by our “Port” style Logic App that looks like this:
The first API App after the Dropbox receive is a custom API app that essentially builds a context property bag when it is passed an XML payload. It does this by comparing the document to a BizTalk schema, and using the instructions in the schema to “promote” properties by extracting the relevant content. It takes two inputs to operate.
The first input is a URL to the root of an Azure Blob Storage container that contains BizTalk schemas. It will use these schemas to perform message type resolution and property promotion. The second input is a string containing the entirety of an XML message. Not exactly the screaming performance of a forward-only streaming pipeline component, but it gets the job done, considering we’re already taking on latency to get to the cloud in the first place.
The output of that API app looks like this (click to enlarge):
The next API app takes the payload, along with the property bag (which it treats as a set of brokered message properties) and publishes the message to an Azure Service Bus Topic. This is just the out of the box connector using the outputs of our custom API app. The call out to that app looks like this:
This published message is picked up by our Logic App that is acting like an Orchestration. That Logic App has a pre-defined subscription on the same service bus topic for any message with a Message Type of Print Job.
After the message is received, the Logic App must quickly setup a subscription for any related messages that come in for that order. Unfortunately, the out of the box connector for Service Bus doesn’t yet have a way to create a new subscription – only ways to subscribe for messages on an existing subscription.
Thus, we will have to use a custom API app to create a subscription unique to this running instance of the Logic App – one that is based on the OrderId property of the received message. To provide this capability, we have a custom API app called CreateInstanceSubscription.
It requires quite a few inputs to function since we don’t yet have the capability of reading details from a stored API connection in a custom API app.
The API takes in a Correlation Property property, which contains the name of the property that is shared in common between the message that triggered the Logic App instance and the message that will be correlated with this running instance.
It also takes in the Message Type (in the namespace + # + root node name format) of the next expected message. Both of these properties will be used to create a new subscription on the service bus topic referenced by the last two configuration properties (Service Bus Topic, and Service Bus Connection String).
After it executes, we might expected to see a subscription like the following (click to enlarge):
Now that we have the subscription created, we can take our time with the rest of the process until we absolutely require the second message. In this case, we’re calling another custom API which provides a visualization of the received messages. In order to read the content, we can either use the xpath() function of Logic Apps to read the XML directly, or we can covert it to JSON first using the json() function, and then simply dot into it. I decided to use the JSON function since I hadn’t attempted to use it in a situation like this yet. It was okay (it was pretty darn verbose). xpath() would have been a better choice here – and the more natural choice given an XML payload.
This yields the following visualization (a body-less bobblehead awaiting its correlated message containing information about which body to use):
And we would expect that there is both an instance subscription in service bus and a Logic App that is still actively processing – waiting for Service Bus to re-activate it with a new message.
At this point, the new message can arrive at any moment in time. It will land in the same Dropbox folder, and process through the same Logic App serving as a “Port” – with the same XML property promotion, and Service Bus publishing action. It will land in the same Service Bus topic as before, and with a matching order id to the originally submitted message.
This second message carries some new information, however. In this case, it contains the body that the customer selected for their custom bobblehead doll.
Once published to the topic, the instance subscription previously created by the second Logic App in the process will be matched by a listening Service Bus connector.
The connector uses the topic name and instance subscription name passed to it from the Create Instance Subscription API App. The name of the subscription will be a randomly generated id for that running instance of the Logic App.
Now that we have the message, it’s time to ensure that we don’t bring the problem of zombies into the world of Logic Apps. There is a step that follows which will clean up the instance subscription for the Logic App before continuing with the final bits of the process.
Again, since the OOTB Service Bus connector does not contain any operations for managing subscriptions, the custom API App is called to delete the instance subscription using the details returned from the original call to the API which created it.
After that, it’s back off to the bobblehead visualizer with the details from the correlated message received.
Call to Action
So that’s pretty cool. We can now stand with confidence and proclaim that content-based correlation is possible with Logic Apps! However, it was built out of necessity, and required custom crafted components – as is often the case with anything worth doing.
You may be wondering why this talk was titled API Apps 101 for BizTalk Developers. I didn’t really tell you how to create API apps. Instead, I showed that API apps behave in a fashion similar to different components within BizTalk Server (adapters, pipeline components, orchestration shapes, etc…). I don’t want to leave you hanging though, because we are at a point in time where there is a golden opportunity to make your mark in the foundations of this new world.
This is the ground floor of Logic Apps and API Apps. As BizTalk Server developers, we know the required ingredients of enterprise integrations. We know the recipes for success. It’s just a matter of crafting some additional tools for use in the world of Logic Apps, and for the first time we have a unified marketplace to share and even sell these components.
From working on BizTalk Server integrations, we know that we will need custom API apps that can serve as adapters, pipeline components, and pattern utility apps (e.g., content-based correlators). In fact, you may have built such things before. It’s honestly not that difficult to port those things over into this new world of integration (where it makes sense) and reap the rewards. If you need inspiration, check out the listings of such components that have already been created for BizTalk Server. Each component represents a solution to a specific integration challenge – many of which are timeless challenges.
We write BizTalk components and API Apps in the same languages, though with different techniques, and targeting a different runtime.
How do we make that all happen? Well, today we are providing the world with “the goods”. All of the slides from this talk, a sample module from the February 2016 version of our Cloud-based Integration Using Azure App Service course, and all of the code involved in the demo. With those combined resources, you should be set on the right track to start building custom API Apps for use in Logic Apps – leveraging skills and work you’ve already accomplished.
If you’re ready to get started, click the image below to download the resources:
Until Next Time
That’s all for now! Again, go forth and create API apps and come visit us in our Cloud-based Integration Using Azure App Service course if you’d like to learn more.
I’ll leave Simon Young with the final word – and dining tip!
