2nd Annual Global Integration Bootcamp a Success!

2nd Annual Global Integration Bootcamp a Success!

DZLx-R1UQAEhxvOLast Saturday I had the great privilege of organising and hosting the 2nd annual Global Integration Bootcamp in Brisbane. This was a free event hosted by 15 communities around the globe, including four in Australia and one in New Zealand!

It’s a lot of work to put on these events, but it’s worth it when you see a whole bunch of dedicated professionals give up part of their weekend because they are enthusiastic to learn about Microsoft’s awesome integration capabilities.

The day’s agenda concentrated on Integration Platform as a Service (iPaaS) offerings in Microsoft Azure. It was a packed schedule with both presentations and hands-on labs:

It wasn’t all work… we had some delicious morning tea, lunch and afternoon tea catered by Artisan’s Café & Catering, and there was a bit of swag to give away as well thanks to Microsoft and also Mexia (who generously sponsored the event).

0a4f5bc5-e350-401a-b98d-a26c7cd6a502-originalOverall, feedback was good and most attendees were appreciative of what they learned. The slide decks for most of the presentations are available online and linked above, and the labs are available here if you would like to have a go.

I’d like to thank my colleagues Susie, Lee and Adam for stepping up into the speaker slots and giving me a couple of much needed breaks! I’d also like to thank Joern Staby for helping out with the lab proctoring and also writing an excellent post-event article. 

Finally, I be remiss in not mentioning the global sponsors who were responsible for getting this world-wide event off of the ground and providing the lab materials:

  • Martin Abbott
  • Glenn Colpaert
  • Steef-Jan Wiggers
  • Tomasso Groenendijk
  • Eldert Grootenboer
  • Sven Van den brande
  • Gijs in ‘t Veld
  • Rob Fox

Really looking forward to next year’s event!

The Current State of Microsoft Integration Related Content

The Current State of Microsoft Integration Related Content

With technology changing fast and services in the cloud evolve more rapidly than their on-premise counterparts creating and updating content around those services becomes challenging. Microsoft Integration has expanded over the years from Grid their the on-premise offering BizTalk Server to multiple cloud services in Azure like Service Bus, Logic Apps, API Management, Azure Functions, Event Hubs, and Event.

Introduction

The server product BizTalk has numerous available content types like Microsoft Docs, Blog posts, online recordings, and presentations. Does this also apply to the mentioned Azure Services? Yes and no, because of the rapid change content is out-of-date fast and people creating the material have a hard time keeping up. At least for me, it’s a challenge to keep up and produce content.

The Questions

Do Integration minded people in the Microsoft ecosystem feel the same way as I feel? Or what’s there view about content? To find out I created  in Google Docs. Furthermore, I sent out a few tweets and a LinkedIn post to encourage people to answer some Integration Content related questions. These questions are:

  • What type of content do you value the most?
  • What Integration Event has your preference?
  • What online content in the integration space do you consume the most?
  • What type integration focused content do you think is valuable for your work as integration professional?
  • Have you attended Integrate London, a local user group meeting or the Global Integration Bootcamp?
  • Does the Global Integration Bootcamp, Integrate London or the local integration focused user group provides value for you?
  • Do have any comments or feedback on Microsoft Integration content?

With the questions above I hope to get a little glimpse into the expectations and thoughts people have with regards to integration content. That is what do they think about the existing content, what is do they appreciate, what content types and through what preferred channel.

The Outcome

The number of responses exceeded 50, which can be the representation of either one up to ten percent of the general population of people working in the integration space. At least that my assumption. However, assessing the actual representation, in the end, is hard. Anyways, let’s review the results of the questionnaire.

The first question was around what specific content type people value the most. And it appears that the majority of respondents still favors blogs, one of the older content types, before vlogs, webcasts, and video became more mainstream. Almost 60% favors blogs over any other content type.

In line with the previous question is what content is consumed the most. The response correlates with what is valued. Moreover, static content is preferred over let’s say dynamic content like vlogs or on-line recordings like Integration Mondays or Middleware Fridays. I left out live Events and Channel 9 intentionally, to see how community content would be consumed. Note that Microsoft Docs is open for changes via GitHub, where the community contributes too. Thus this content type is partially maintained by the community.

With another question, I tried to see which event was preferred the most of the three we have available from an integration perspective. A global, centralized one like Integrate, a local user group, or a Global Integration Bootcamp on one day in various venues. Close to 50% favor Integrate London, while local user groups and the boot camp are around 25%.

As a follow-up, I asked who attend any of these events or not. And most (>75%) respondents attended either a local user group, a Global Integration Boot camp or Integrate.

The other questions were open ones. Here, people could more specifically provide feedback on what content they value apart from the channel it is delivered through, and how much value an event is providing (if attended), and one more where people could provide more general feedback people about integration content.

Conclusions

Respondents have strong preferences for content around examples, use-cases (real-world), up-to-date content, architecture, design, and patterns. This feedback was expressed by many in the question “What type integration focused content do you think is valuable for your work as integration professional?”. Furthermore, the answers are reflected in the general feedback they could give about integration content. An example is in the following comments (feedback):

“I would like to see more of how companies are adopting the Azure platform. For instance, a medium to large enterprise integration employing Logic apps and service bus and they came up with the solution architecture, challenges faced, lessons learned.”

Or

“Docs are getting better and better, but finding the right content and keeping up with the release speed of Microsoft appears to be a challenge sometimes.”

With people attending events, the value lies in the opportunity for networking, see (new) content, and have interactions with peers in the fields, MVPs, and Microsoft. Generally, a local event, a boot camp, or a bigger event tend to be the right places to socialize, learn about new tech, and get a perspective on the integration ecosystem. This perceived view is reflected in the answers about the value of attending an event.

To conclude people have an overall satisfaction in content and how it is delivered. However, a clear demand for more up-to-date content online and practical guidance is requested by people for their day to day jobs as integrators.

Finally, I like to thank everyone for taking time to answer the questions.
Cheers,

Steef-Jan

Author: Steef-Jan Wiggers

Steef-Jan Wiggers is all in on Microsoft Azure, Integration, and Data Science. He has over 15 years’ experience in a wide variety of scenarios such as custom .NET solution development, overseeing large enterprise integrations, building web services, managing projects, designing web services, experimenting with data, SQL Server database administration, and consulting. Steef-Jan loves challenges in the Microsoft playing field combining it with his domain knowledge in energy, utility, banking, insurance, healthcare, agriculture, (local) government, bio-sciences, retail, travel, and logistics. He is very active in the community as a blogger, TechNet Wiki author, book author, and global public speaker. For these efforts, Microsoft has recognized him a Microsoft MVP for the past 8 years. View all posts by Steef-Jan Wiggers

Building reactive, event driven solutions with the new Azure Event Grid Service

Building reactive, event driven solutions with the new Azure Event Grid Service

Microsoft has released yet another service in its Azure Platform named Event Grid. This enables you to build reactive, event driven applications around this service routing capabilities. You can receive events from multiple source or have events pushed (fan out) to multiple destinations as the picture below shows.

New possible solutions with Event Grid

With this new service there are some nifty serverless solution architectures possible, where this service has its role and value. For instance you can run image analysis on let’s say a picture of someone is being added to blob storage. The event, a new picture to blob storage can be pushed as an event to Event Grid, where a function or Logic App can handle the event by picking up the image from the blob storage and sent it to a Cognitive Service API face API. See the diagram below.

Another solution could involve creating an Event Topic for which you can push a workload to and an Azure function, or Logic App or both can process it. See the diagram below.

And finally the Event Grid offers professional working on operation side of Azure to make their work more efficient when automating deployments of Azure services. For instance a notification is send once one of the Azure services is ready. Let’s say once a Cosmos DB instance is ready a notification needs to be sent.

The last sample solution is something we will build using Event Grid, based on the only walkthrough provided in the documentation.

Sent notification when Cosmos DB is provisioned

To have a notification send to you by email once an Azure Service is created a Logic App is triggered by an event (raised once the service is created in a certain resource group). The Logic App triggered by the event will act upon it by sending an email. The trigger and action are the Logic and it’s easy to implement this. And the Logic App is subscribing to the event within the resource group when a new Azure Service is ready.

Building a Logic App is straight forward and once provisioned you can choose a blank template. Subsequently, you add a trigger, for our solution it’s the event grid once a resource is created (the only available action trigger currently).

The second step is adding a condition to check the event in the body. In this condition in advanced mode I created : @equals(triggerBody()?[‘data’][‘operationName’], ‘Microsoft.DocumentDB/databaseAccounts’)

This expression checks the event body for a data object whose operationName property is the Microsoft.DocumentDB/databaseAccounts operation. See also Event Grid event schema.

The final step is to add an action in the true branch. And this is an action to sent an email to an address with a subject and body.

To test this create a Cosmos DB instance, wait until its provisioned and the email notification.

Note: Azure Resource Manager, Event Hubs Capture, and Storage blob service are launch publishers. Hence, this sample is just an illustration and will not actually work!

Call to action

Getting acquainted with this new service was a good experience. My feeling is that this service will be a gamechanger with regards to building serverless event driven solution. This service in conjunction with services like Logic Apps, Azure Functions, Storage and other services bring a whole lot of new set of capabilities not matched by any other Cloud vendor. I am looking forward to the evolution of this service, which is in preview currently.

If you work in the integration/IoT space than this is definitely a service you need to be aware and research. A good starting point is : Introducing Azure Event Grid – an event service for modern applications and this infoq article.

Author: Steef-Jan Wiggers

Steef-Jan Wiggers is all in on Microsoft Azure, Integration, and Data Science. He has over 15 years’ experience in a wide variety of scenarios such as custom .NET solution development, overseeing large enterprise integrations, building web services, managing projects, designing web services, experimenting with data, SQL Server database administration, and consulting. Steef-Jan loves challenges in the Microsoft playing field combining it with his domain knowledge in energy, utility, banking, insurance, health care, agriculture, (local) government, bio-sciences, retail, travel and logistics. He is very active in the community as a blogger, TechNet Wiki author, book author, and global public speaker. For these efforts, Microsoft has recognized him a Microsoft MVP for the past 7 years. View all posts by Steef-Jan Wiggers

BizTalk Server 2016 Developer edition is now available on Visual Studio Dev Essentials

With the recent release of BizTalk Server 2016 we are happy to announce that the developer image of BizTalk Server 2016 is licensed for free to use in your test and development scenarios. BizTalk Server 2016 is Microsoft’s latest release of the new hybrid integration software.

With BizTalk Server 2016, customers can automate mission critical business processes, leverage support for latest first party platforms and gain newer capabilities within the BizTalk Administration console. With this release, customers also have the flexibility to adopt a hybrid approach by choosing to connect to SaaS applications through Azure Logic Apps, or running BizTalk Server on Azure leveraging full support in production environments.

Azure Logic Apps is another core part of Microsoft’s Hybrid Integration Platform, that allows you to rapidly connect to numerous SaaS and enterprise applications, enable business to business communication and leverage the power of other Azure services like Functions, Cognitive Services, Machine Learning and many more. BizTalk Server 2016 is the first version to take full advantage of this hybrid integration story, with an out of the box adapter to connect directly to Azure Logic Apps. This opens up numerous hybrid and cloud integration scenarios – all from the comfort of using your on-premises BizTalk Server 2016.

BizTalk Server 2016 also provides support to many of the first party Microsoft platforms such as Windows Server 2016, Office 2016, Visual Studio 2015 and SQL Server 2016. In addition, it provides the ability to run your Enterprise Integration Solution in the cloud leveraging the AlwaysOn and Availability groups through SQL Server 2016 to achieve high available environments in Azure on IaaS.

Please follow this link to sign up for Visual Studio Dev Essentials and download BizTalk Server 2016 Developer edition for free today.

Building a composite service using Logic Apps

Building a composite service using Logic Apps

Microsoft has made the Logic Apps General Available (GA) a few months ago (end of July 2016) and this Azure service has evolved ever since. Each month the product group responsible for the service, Jeff Hollan and Kevin Lam together with Jon Fancey, present updates through a YouTube channel. The latest update (26 January 2017) included for instance, browsing enhancements for connectors, switch case condition (similar to C# switch), JSON Parse to tokenize properties, new SaaS connectors, operations and upcoming new features.

Besides the evolution and thus an increase in maturity, the adoption of Logic Apps rises. I myself and a colleague at Macaw have built and deployed a Logic App solution for a customer in production that runs for over a month now without any issues. And we are not the only ones as there are many more customer cases. Businesses start seeing the added value brought by Logic Apps with its connectors, short lead times and no requirements for servers. Logic Apps run in Azure and are Server less, i.e. you can have an enterprise ready integration solution running in the cloud for low cost and maintenance (TCO).

This blog post will provide some additional insight on the power and value of Logic Apps. It can be seen as a game changer concerning integration in the cloud. Logic Apps fall under the moniker integration Platform as a Service (iPaaS), which is what actually is, and I will be quoting Wikipedia here to emphasize this:

“The most common cloud-based integration service model is iPaaS (Integration Platform as a Service), which is a suite of cloud services enabling customers to develop, execute and govern integration flows between disparate applications. Under the cloud-based iPaaS integration model, customers drive the development and deployment of integrations without installing or managing any hardware or middleware.”

You provision a service/host, define the logic and execute in Azure. And iPaaS service is characterized as (according to Wikipedia):

Characteristic Azure
Deployed on a multi-tenant, elastic cloud infrastructure. The Logic App service
Subscription model pricing (OPEX, not CAPEX) Consumption Plan
No software development (required connectors should already be available) Logic App Definition, the available connectors (i.e. managed and/or custom) and actions
Users do not perform deployment or manage the platform itself ARM Template and the Azure Portal for management
Presence of integration management & monitoring features Logic App overview (blades), integration with OMS

Scenario

In this blog post I would like to share another use case or scenario of a Logic App to demonstrate its value and some of the new features. We will build a scenario to demonstrate API integration, where the focus is on connecting different API’s. One common reason for doing this is to automate more of a business process. And Logic Apps can facilitate the integration of one or multiple API’s.

A Logic App will be a composite service that, based on a set of parameters, will provide a result containing weather and event details. The parameters will be city for which you like to know the details about the weather the coming 7 days and events in that time frame. The Logic App will call two API’s hosted in API Management, and combine the result of each call into one single response to the client. The complexity of calling the API’s is abstracted away in API Management, and the composition of the response of both API’s is done in the Logic App.

Building the API

The API’s are generally available API’s, which will be abstracted away from the Logic App by creating a proxy in API Management. In API Management, we will create an API to add operations to them and tie them to the actual API and its operations combined with policies to manage security, and other quality aspects. We provide a name for the API, set the Web service URL (endpoint of the API) and can observe the actual WebAPI URL i.e. API Management instance name with a standard DNS addition .azure-api.net + suffix.

The WebAPI will have an operation daily that connects to the daily forecast operation of the WeatherAPI (api.openweathermap.org) and its parameters.

A policy will be applied to the operation, to add an APPID key in the query parameters.

<policies>
        <inbound>
                <set-query-parameter name="APPID" exists-action="append">
                        <value>67b0fe39a73fdf8c5045f9538270cee4</value>
                        <!-- for multiple parameters with the same name add additional value elements -->
                </set-query-parameter>
                <rewrite-uri template="daily" />
        </inbound>
        <backend>
                <forward-request />
        </backend>
        <outbound />
        <on-error />
</policies>

The policy ensures that the operation can be executed, as the APPID is mandatory in the query parameters. However, the consumer of the Azure API is unaware of this as it is abstracted away through the policy. The same process will be done in the event API (api.eventful.com).

Build the function

A HTTP function will merge the two bodies i.e. JSON strings into one and currently there is no action for it. Hence, we will build a function that will accept a string with two JSON bodies in them.

In general, when it comes to certain operations on your data within a Logic App, you will notice that a function can help facilitate in having that specific capability like merging JSON strings or perhaps date time manipulation. In our scenario, we’ll merge the two JSON responses to one using a function that can be called through HTTP. It is one big string that will be sent to a function, and with a Web Hook type of function it expects a JSON! A HTTP Function can accept a string, manipulate it and return a JSON as shown below.

The two JSON response bodies are concatenated together when calling the function, which means the braces }{ need to be replaced by a comma ‘,’ to form a JSON again.

Building the Logic App Definition

The Logic App itself is a service, i.e. the infrastructure is in Azure and once provisioned you can define your logic. Logic in this context is a definition with a trigger and one or more actions. To define the trigger and action we will have a designer available to add the trigger followed by actions and/or workflow specific actions i.e. for instance a condition, case or scope.
The switch case is one of the new additions.

The Logic App definition of our solution will have a HTTP request trigger followed by an action to consume the weather API and later the event API. It contains a switch to support the possible responses from the API i.e. HTTP status like 200, 400 or other. And based on those types of responses you have a switch in place to control the various outcomes of your Logic App. A 200 will mean proceed to call the next API or create some sort of response to the caller of your Logic App. In status 200 branch, a JSON parse will be used to tokenize the JSON response from the weather API. The JSON parse is another recently added action in Logic Apps as discussed in the introduction. Subsequently, the event API operation search will be called and if the HTTP status is 200, the left branch will call the function through HTTP presenting both response (bodies) to create a JSON response, which will be returned to the client as a response. If the call to search fails, the right branch will return a custom response.

Test the Logic App

To test the Logic App, we will use Postman to send a request (POST) with a payload. This payload will contain a city, start- and end date.

In the Azure Portal we can observe the detail of the Logic App run and drill down into each of the actions. And this is a great feature of Logic Apps, you can see what happens and even resubmit, which is very useful to investigate where failures happen or change the flow in the designer and then resubmit. You do have to use a client like Postman to resubmit!

The Logic App provides monitoring capabilities in its blade like metrics. You can monitor your runs and select metrics to see how your Logic App definition i.e. flow performs.

Metrics are not the only feature within the monitoring section of Logic App. You have diagnostics, and logs you can examine. Thus, its adheres to the characteristic of “Presence of integration management & monitoring features” of iPaaS.

Considerations

Logic App can add value to the business with automating processes, which are agile by the nature of this Azure Service. A process can be created and operational in hours instead of days, can be changed on the fly or replaced by a new process i.e. Logic App. Logic Apps will bring IT closer to the business than ever before. The tremendous number of connectors lowers the connectivity barrier or API’s can be tied together as shown in this blog post as a composite service or to create a business flow/process (orchestration).
However, the power and agility of Logic Apps will also bring the risk of losing grip on the processes if everyone can easily build these flows and run them, i.e. you will need some form of governance to prevent the business completely go wild with this service. With a great deal of flexibility or agility and lower barrier to build integrations some form of governance will necessary to prevent overlap, over consumption, and conflicts.

Logic App can be viewed as a counterpart for BizTalk Server, a server product from Microsoft suitable for application integration. In BizTalk business processes can be automated by connecting two or more applications or services. In this blog post we have seen that two API calls are combined to create one response. However, Logic Apps with its various connectors can do the same thing as we did with BizTalk on premise. With BizTalk, we can create a composite application similar to what we have shown in this post by providing a common front end to a group of existing applications. Hence the value BizTalk provided on premise is now available through Logic Apps. The most important difference is hosting and pricing i.e. cloud versus on premise and license fees versus pay as you go.

Logic Apps can be called directly through HTTP and thus are exposed to world wide web i.e. everyone, which means it will need some sort of security. Out of the box is that a signature (SAS) is bound to the endpoint and is required for clients to provide in the request. Some see this as a form of security by obscurity and to further enhance security you can use API Management. API Management abstracts away the security and you can enforce other means of security to access the endpoint i.e. OpenID or OAuth 2.0.

Supporting the solution such as the one described in this post can be a challenge since you need to monitor or keep an eye out for the API keys at API supplier i.e. the weather API and event API, key(s) for the function and SAS signature in the Logic App endpoint, and the consumption of the Logic App. Each action in Logic App is an Azure consumption and has a price, which can be monitored easily through observing the resource costs in a resource group; it is operational costs of executing Logic Apps. You have to factor in notifications and alerting in your solution, which is easy to set up using alert rules on your resource.

Call to Action

You can learn to work the service yourself, learn more about or express your wishes:

Author: Steef-Jan Wiggers

Steef-Jan Wiggers has over 15 years’ experience as a technical lead developer, application architect and consultant, specializing in custom applications, enterprise application integration (BizTalk), Web services and Windows Azure. Steef-Jan is very active in the BizTalk community as a blogger, Wiki author/editor, forum moderator, writer and public speaker in the Netherlands and Europe. For these efforts, Microsoft has recognized him a Microsoft MVP for the past 5 years. View all posts by Steef-Jan Wiggers