INTEGRATE 2018: Highlights from Another Great Event

This post was originally published here

Photo by Tariq Sheikh

Two weeks ago I had not only the privilege to attend the sixth INTEGRATE event in London, but also the great honour of speaking for the second time. These events always provide a wealth of information and insight as well as opportunities to meet face-to-face with the greatest minds in the enterprise integration space. This year was no exception, with at least 24 sessions featuring as many speakers from both the Microsoft and the MVP community.

As usual, the first half of the 3-day conference was devoted to the Microsoft product team, with presentations from Jon Fancy (who also gave the keynote), Kevin Lam, Derek Li, Jeff Hollan, Paul Larsen, Valerie Robb, Vladimir Vinogradsky, Miao Jiang, Clemens Vasters, Dan Rosanova, Divya Swarnkar, Kent Weare, Amit Kumar Dua, and Matt Farmer. For me, the highlights of these sessions were:

• Jon Fancey’s keynote address.  Jon talked about the inevitability of change, underpinning this with a collection of images showing how much technology has progressed in the last 30-40 years alone. Innovation often causes disruption, but this isn’t always a bad thing; the phases of denial and questioning eventually lead to enlightenment. Jon included several notable quotes including one by the historian Lewis Mumford (“Continuities inevitably represent inertia, the dead past; and only mutations are likely to prove durable.”) and a final one by Microsoft CEO Satya Nadella (“What are you going to change to create the future?”).
• Kevin Lam and Derek Li taking us through the basics of Logic Apps. It’s easy to become an “Integration Hero” using Logic Apps to solve integration problems quickly. Derek built two impressive demos from scratch during this session, including one that performed OCR on an image of an order and then conditionally sent an approval email based on the amount of the order.
• Jeff Hollan and his explanation of Azure Durable Functions. An extension of Azure functions, Durable Functions enable long-running, stateful, serverless operations which can execute complex orchestrations. Built on the Durable Task framework, they manage state, checkpoints and restarts. Jeff also gave some practical advice about scalable patterns for sharing code resources across multiple instances.
• Paul Larsen and Valerie Robb with a preview of Feature Pack 3 for BizTalk Server 2016. This upcoming feature pack will include GDPR support, SQL Server 2016 SP2 support (including the long-awaited ability to deploy multiple databases per instance of an Availability Group), new Office365 adapters, and advanced scheduling support for receive locations.
• Miao Jiang and his preview of end-to-end tracing via Application Insights in API Management. This includes the ability to map an application, view metrics, detect issues, diagnose errors, and much more.
• Clemens Vasters. (Need I say more?) The genius architect behind Azure Service Bus gave a clear, in-depth discussion about the difference between messaging (which is about intents) and eventing (which is about facts). He also introduced Event Grid’s support for CNCF Cloud Events, and open standard for events.
• Dan Rosanova and his clear explanation of the different cloud messaging options available in Azure. His slide showing the segmentation of services into Serverless, Big Data and Enterprise provided a lot of clarity as to when to use what. He also covered Event Grid, Event Hubs, and Event Hub’s support for Kafka.
• Jon Fancey & Divya Swarnkar talking about Logic Apps for enterprise integration. It was especially nice to see support for XSLT 3.0 and Liquid for mapping, as well as the new SAP connector. Also there was an enthusiastic response from the audience when revealed there were plans to introduce a consumption based model for Integration Accounts!
• Kent Weare talking about Microsoft Flow and how it is the automation tool for Office365. Together with PowerApps. Power BI, and Common Data Services you can can build a Business Application Platform (BAP) that supports powerful enterprise-grade applications. Best of all, Flow makes it possible for citizen integrators to automate non-critical processes without the need for a professional development team. A notable quote in Kent’s presentation was “Transformation does not occur while waiting in line!”
• Divya Swarnkar and Amit Kumar Dua talking about how Microsoft IT use Logic Apps for Enterprise Integration. It is always interesting to see how Microsoft “dog foods” its own technology to support in-house processes. In this case they built a Trading Partner Migration tool to lift more than 1000 profiles from BizTalk into Logic Apps. Highlights from the lessons learnt were understanding the published limits of Logic Apps and also that there is no “exactly once” delivery option.
• Vladimir Vinogradsky with a deep-dive into API Management. Covering the topics of developer authentication, data plane security, and deployment automation, Vladimir gave some great tips including how to use Azure AD and Azure AD B2C for different scenarios, identifying the difference between key vs. JWT vs. client certificate for authorisation (as well as when to use which), and the use of granular service and API templates to improve the devops story. It was also fun watching him launch his cool demos using Visual Studio Code and httpbin.org.
• Kevin Lam and Matt Farmer’s deep-dive into Logic Apps. This session yielded an enthusiastic applause from the audience with Kevin’s announcement of the upcoming preview of Integration Service Environments. This is a true game-changer bringing dedicated compute, isolated VNET connectivity, custom inbound domain names, static outbound IPs, and flat cost to Logic Apps. There was also a demonstration by Matt of building a custom connector, and how an ISV can turn this into a “real” connector providing it is for a SaaS service you own.
• Derek Li and Kevin Lam with Logic App patterns and best practices. This session sported some great performance improving and cost saving advice by limiting the number of actions with clever loop processing logic (hint: check out the Filter Array action as per this blog post). There was also a tip on how to easily distinguish built-ins, connectors and enterprise connectors: you can filter for these when choosing a connector in the dialog.
• Jon Fancey and Matt Farmer talking about the Microsoft integration roadmap. Key messages here were Jon confirming that Logic App will be coming to Azure Stack, and that Integration Service Environments (ISE) will also eventually come to on-premises. Matt also talked about Azure Integration Services, which on the surface appears to be a new marketing name to the collection of existing services (API Management, Logic Apps, Service Bus and Event Grid). However it promises to include reference architectures, templates and other assets to provide better guidance on using these services.

This concluded the Microsoft presentations on the agenda. After a brief introduction to the BizTalk360 Partnership Program by Business Development Manager Duncan Barker, the MVP sessions began, featuring speakers Saravana Kumar, Steef-Jan Wiggers, Sandro Pereira, Stephen W. Thomas, Richard Seroter, Michael Stephenson, Johan Hedberg, Wagner Silveira, Dan Toomey, Toon Vanhoutte and Mattias Lögdberg. Highlights for me included:

• Saravana Kumar giving a BizTalk360 product update. Saravana revealed that ServiceBus360 is being renamed to Serverless360, which supports not only Service Bus queues, topics and relays, but also Event Hubs, Logic Apps and API Management. And there is an option to either let BizTalk360 host it or host on-prem in your own datacentre.
• Steef-Jan Wiggers talking about Serverless Messaging in Azure. This was a great roundup of all the messaging capabilities in Azure including Service Bus, Storage Queues, Event Hubs and Event Grid. Steef-Jan explained the difference between each very clearly before launching into some very compelling demos.
• Saravana Kumar giving an Atomic Scope update. A product that provides end-to-end tracking for hybrid solutions featuring BizTalk Server, Logic Apps, Azure Functions, and App Services, Atomic Scope can reduce the time spent building this capability from 20-30% right down to 5%. Saravana affirmed that it is designed for business users, not technical users. Included in this session was a real-life testimonial by Bart Scheurweghs from Integration.Team where Atomic Scope solved their tracking issues across a hybrid application built for Van Moer Logistics.
• Sandro Pereira with his BizTalk Server lessons from the road. Sandro’s colourful presentations are always full of good advice, and this was no exception. Whilst talking about a variety of best practices from a security perspective, Sandro reminded us that the BizTalk Server platform is inherently GPDR compliant – but not necessarily the applications we build on it! He also talked about the benefits of feature packs (which may contain breaking code) and cumulative updates (which shouldn’t break code), being wary of JSON schema element names with spaces in them, REST support in BizTalk (how it doesn’t handle optional parameters OOTB), and cautioning against use of some popular patterns like singletons and sequence convoys which are notorious for introducing performance issues and zombies.
• Stephen W. Thomas and using BizTalk Server as your foundation to the cloud. Stephen talked about a bunch of scenarios that justify migration to or use of Logic Apps as opposed to BizTalk Server, including use of connectors not in BizTalk, reducing the load on on-prem infrastructure, saving costs, improved batching capability, and planning for the future. He also identified a number of typical “blockers” to cloud migration and discussed various strategies to address them.
• Richard Seroter on architecting highly available cloud solutions. One of the highlights of this event for me was Richard’s extremely useful advice on ensuring your apps are architected correctly for high-availability. Taking us through a number of Azure services from storage to databases to Service Bus to Logic Apps and more, Richard pointed out what is provided OOTB by Azure and what you as the architect need to cater for. His summary pointers included 1) only integrate with highly available endpoints, 2) clearly understand what services failover together, and 3) regularly perform chaos testing.
• Michael Stephenson and using Microsoft Flow to empower DevOps. It was great to see Michael return to the stage this year, and his Minecraft hosted demo of an automated process for onboarding users was a real crowd-pleaser. Adhoc tasks such as these can waste time and money, but automating the process with Flow not only mitigates this but is also relatively easy to achieve if the process can be broken down to well-understood tasks.
• Unfortunately I had to miss Johan Hedberg’s session on VSTS and BizTalk Server, but by all accounts it was a demo-heavy presentation that provided lots of useful tips on establishing a CI/CD process with BizTalk Server. I look forward to watching the video when it’s out.
• Wagner Silveira on exposing BizTalk Server to the world. This session continued the hybrid application theme by discussing the various ways of exposing HTTP endpoints from BizTalk Server for consumption by external clients. Options included Azure Relay, Logic Apps, Function Proxies and API Management – each with their own strengths and limitations. Wagner performed multiple demos before summarising the need to identify your choices and needs, and “find the balance”.
• Dan Toomey and the anatomy of an enterprise integration architecture. Well… I didn’t take any notes on this one, seeing as I was onstage speaking! But you can download the slides to get an overview of how Microsoft integration technologies can be leveraged to reduce friction across layers of applications that move at different speeds.
• Toon Vanhoutte’s presentation on using webhooks with BizTalk Server. In his first time appearance at INTEGRATE, Toon gave a very compelling talk about the efficiency of webhooks over a polling architecture. He then walked through the three responsibilities of the publisher (reliability, security, and endpoint validation) as well as the five responsibilities of the consumer (high availability, scalability, reliability, security, and sequencing) and performed multiple demos to illustrate these.
• Wrapping up the presentations was Mattias Lögdberg on refining BizTalk Server implementations. Also a first-time speaker at INTEGRATE, Mattias wowed us by doing his entire presentation as a live drawing on his tablet! He used this medium to walk us through a real-life project where a webshop was migrated to Azure whilst retaining connectivity to the on-prem ERP system. A transition to a microservices-based approach and addition of Azure services enhanced the capabilities of BizTalk which still handles the on-prem integration needs.

The conference finished with a Q&A session with the principal PMs from the product group, including Jon Fancey, Paul Larsen, Kevin Lam, Vladimir Vinogradsky, Kent Weare, and Dan Rosanova as panellists. The inevitable question of BizTalk vNext came up, and there was some obvious frustration from the audience as the panellists were unable to provide a clear and definitive statement about whether BizTalk Server 2016 would be the final release or not. Considering that we are halfway through the main support period for this product, the concern is very understandable as we need to know what message to convey to our clients. Reading between the lines, many have concluded for themselves that there will be no vNext and that migration to the cloud is now becoming a critical urgency. Posts such as this one from Michael Stephenson help to put things in context and show that compared to competing products we’re not as bad off as some alarmists may suggest, but it was still disappointing to end the conference with such a vague view of the horizon. I hope that Microsoft sends out a clear message soon about what level of support can be expected for the significant investment that many of our clients have made in BizTalk recently.

One especially cool thing that the organisers did was to hire Visual Scribing to come and draw a mural of all the presentations, capturing the key messages throughout the conference:

Day 1

Day 2 & 3

For more in-depth coverage of the sessions and announcements from INTEGRATE 2018, I encourage you to check out the following:

Also, keep an eye on the INTEGRATE 2018 website, as eventually the video and the slide decks will be uploaded there.

I really want to thank Saravana and his whole team at BizTalk360 for organising such a mammoth event where everything ran so smoothly. I also want to thank Microsoft and my MVP colleagues for their contributions to the event, as well as my generous employer Mexia for sending me to London to enjoy this experience!