A few weeks ago I had the great privilege of presenting a 60 minute breakout session at Microsoft Ignite | The Tour in Sydney. It was thrilling to have over 200 people registered to see my topic “Seamless Deployments with Azure Service Fabric”, especially in the massive Convention Centre.
In the session I demonstrated the self-healing capabilities of Service Fabric by introducing a bug in the code and then attempting a rolling upgrade. It was impressive to see how Service Fabric detected the bug after the first node was upgraded and then immediately started rolling it back.
As you can imagine, it took a fair amount of practice to get the demo smooth and functioning within the tight time limits of the average audience attention span. (In fact, I had to learn how to tweak both the cluster and the application health check settings to shorten the interval – perhaps the subject of another blog post!) Naturally this also entailed frequently “resetting” the environment so that I could start over when things didn’t go quite as planned, or if I wanted to reset the version number. If you’ve ever worked with Service Fabric before you would know that deployments from Visual Studio (or Azure DevOps) can take a while; and undeploying an application from Service Fabric manually in the portal is painful!
For example, if I want to undeploy an application from a Service Fabric cluster in the web-based Service Fabric Explorer, I have to do the following in this order:
- Remove the service
- Remove the application
- Unregister the application type
- Remove the application package
What becomes really annoying is that each step elicits a confirmation prompt where you need to type the name of the artefact you want to remove! That gets old pretty fast.
Thankfully, there is an alternative. Service Fabric offers a number of different ways to deploy applications, including Visual Studio, Azure CLI, and PowerShell. Underneath the covers I expect these all make use of the REST API. But in my case I found the simplest and most efficient choice was PowerShell. Using the documented commands, it is easy to create a script that will deploy or undeploy your application package in seconds. And I mean seconds! It was astounding to see how quickly the undeploy script could tear down the application!
The script I created is available in my demo code on GitHub. I’ll walk through some of it here.
First, it is necessary to have the Azure PowerShell installed. This is normally included when you install the Service Fabric SDK, but you must enable execution of the scripts first.
Second, in order for the Deploy_SFApplication.ps1 script to work, you must have already packaged the application. You do this by right-clicking the Service Fabric project in Visual Studio (not the solution file!) and selecting “Package”. The path to this package is a mandatory parameter for this script. The Undeploy_SFApplication.ps1 script does not require this.
The make the scripts reusable with the minimum amount of changes, I’ve parameterized all of the potentially variable settings:
|Parameter Name||Description||Example/Default Value|
|path||This is the path to your packaged application. (This parameter is not required for the Undeploy_SFApplication.ps1 script)||C:ReposDemosVoting_v3VotingpkgDebug|
|imageStorePath||Where you want the package stored when uploaded in Service Fabric. Typically this can be the application name, perhaps with a version.||Voting|
|appTypeName||Usually the app name with “Type” appended||VotingType|
|appName||Must be prepended with “fabric:/”||fabric:/Voting|
|appVersion||IMPORTANT! Cannot deploy the same version already existing, it will fail||1.0.0|
|ServerCommonName||If using your local development cluster, just “localhost”.|
Otherwise, if in Azure, “CLUSTER_NAME.REGION.cloudapp.azure.com”
|clusterAddress||Append the port number to the $ServerCommonName variable, usually 19000||$(ServerCommonName):19000|
|thumb||The thumbprint of the certificate used for a secured cluster (not generally required for a local cluster)|
NOTE: The script currently sets the location of the certificate in the current user’s personal store. However this could be easily parameterized.
Script Execution Steps
First thing we do is import the appropriate module:
Import-Module "$ENV:ProgramFilesMicrosoft SDKsService FabricToolsPSModuleServiceFabricSDKServiceFabricSDK.psm1"
Then it’s simply a matter of following using the documented commands, substituting the variables as appropriate in order to:
- Connect to the cluster
- Upload the package to the package store
- Register the application type
- Create the application instance
My Deploy_SFApplication.ps1 script also prints out the application instance details as well as the associated service instance details:
The Undeploy_SFApplication.ps1 script does much the same, except in reverse of course:
- Connect to the cluster
- Remove the application instance
- Unregister the application type
- Remove the application package
The use of the –force flag means that when you run this script you will NOT be prompted for confirmation like this:
Whilst the deployment script takes about 20 seconds for this Voting application, the undeploy script takes less than five seconds!
As mentioned previously, the scripts are freely downloadable along with the rest of the demo code on GitHub. I’m no PowerShell guru, so I’m sure there’s plenty of room for improvement. Send me a pull request if you have any suggestions! And feel free to get in touch if you have any questions.
This post was originally published on Deloitte’s Platform Engineering blog.
First of all, I’d like to apologise to all grandmothers out there… I mean you no disrespect. It’s just meant to be a catchy title, really. I know grandmothers who are smarter than most of us.
A couple of months ago I had the privilege of speaking at the API Days event in Melbourne. My topic was on Building Event-Driven Integration Architectures, and within that talk I felt a need to compare events to messages, as Clement Vasters did so eloquently in his presentation at INTEGRATE 2018. In a slight divergence within that talk I highlighted three common messaging patterns using a pizza based analogy. Given the time constraint that segment was compressed into less than a minute, but I thought it might be valuable enough to put in a blog post.
Photo courtesy of mypizzachoice.com
1) Synchronous Messaging
So before you can order a pizza, you need to know a couple of things. First of all, whether the pizza shop is open, and then of course what pizzas they have on offer. You really can’t do anything else without this knowledge, and those facts should be readily available – either by browsing a website, or by picking up the phone and dialling the shop. Essentially you make a request for information and that information is delivered to you straight away.
That’s what we expect with synchronous messaging – a request and response within the same channel, session and connection. And we shouldn’t have time to go get a coffee before the answer comes back. From an application perspective, that is very simple to implement as the service provider doesn’t have to initiate or establish a connection to the client; the client does all of that and service simply responds. However care must be take to ensure the response is swift, lest you risk incurring a timeout exception. Then you create ambiguity for the client who doesn’t really know whether the request was processed or not (especially troublesome if it were a transactional command that requires idempotency).
2) Asynchronous Messaging
So now you know the store is open and what pizzas to choose from. Great. So you settle on that wonderful ham & pineapple pizza on a traditional crust with a garnish of basil, and you place your order. Now if the shopkeeper hands you a pizza straight away, you’re probably not terribly inclined to accept it. Clearly you expect your pizza to be cooked fresh to order, not just pulled ready-made off a shelf. More likely you’ll be given an order number or a ticket and told your pizza will be ready in 20 minutes or so.
Now comes the interesting part – the delivery. Typically you will have two choices. You can either ask them to deliver the pizza to you in your home. This frees you up to do other things while you wait, and you don’t have to worry about chasing after your purchase. They bring it to you. But there’s a slight catch: you have to give them a valid address.
In the asynchronous messaging world, we would call this address a “callback” endpoint. In this scenario, the service provider has the burden of delivering the response to the client when it’s ready. This also means catering for a scenario where the callback endpoint it invalid or unavailable. Handling this could be as blunt as dropping the response and forgetting about it, or as robust as storing it and sending an out-of-band message through some alternate route to the client to come pick it up. Either way, in most cases this is an easier solution for the client than for the service provider.
But what if you don’t want to give out your address? In this case, you might say to the pizza maker, “I’ll come in and pick it up.” So they say fine, it’ll be ready in 20 minutes. Only you get there in 10 minutes and ask if it’s ready; they say not yet. You wait a few more minutes and ask again, and get the same response. Eventually it is ready and they hand you your nice fresh piping hot pizza.
This is an example of a polling pattern. The only burden on the service provider is to produce the response and then store it somewhere temporarily. The client has the job of continually asking if it is ready, and needs to cater for a series of negative responses before finally retrieving the result it is after. You might see this as a less favourable approach for the client – but sometimes this is driven by constraints on the client side, such as difficulties opening up a firewall rule to allow incoming traffic.
3) Publish & Subscribe
Now let’s say that the pizza was so good (even if a little on the pricey side) that you decide to compliment the manager. He asks if you’d like to be notified when there are special discounts or when new pizzas are introduced. You say “Sure!” and sign up on his mailing list.
Now the beauty of this arrangement is that it costs the manager no extra effort to have you join his mailing list. He still produces the same newsletter and publishes it through his mailing agent. The number of subscribers on the list can grow or shrink, it makes no difference. And the manager doesn’t even have to be aware of who is on that list or how many (although he/she may care if the list becomes very very short!) You as the subscriber have the flexibility to opt in or opt out.
It is precisely this flexibility and scalability that makes this pattern so attractive in the messaging world. An application can easily be extended by creating new subscribers to a message, and this is unlikely to have an impact on the existing processes that consume the same message. It is also the most decoupled solution, as the publisher and subscriber need not know anything about each other in terms of protocols, language, endpoints, etc. (except of course for the publishing endpoint which it typically distinct and isolated from either the publisher or consumer systems). This is the whole concept behind a message bus, and is the fundamental principle behind many integration and eventing platforms such as BizTalk Server and Azure Event Grid.
The challenge comes when the publisher needs to make a change, as it can be difficult sometimes to determine the impact on the subscribers, particularly when the details of those subscribers are sketchy or unknown. Most platforms come with tooling that helps with this, but if you’re designing complex applications where many different services are glued together using publish / subscribe, you will need some very good documentation and some maintenance skills to look after it.
So I hope this analogy is useful – not just for explaining to your grandmothers, but to anyone who needs to grasp the concept of messaging patterns. And now… I think I’m going to go order a pizza.
Photo by Tariq Sheikh
Two weeks ago I had not only the privilege to attend the sixth INTEGRATE event in London, but also the great honour of speaking for the second time. These events always provide a wealth of information and insight as well as opportunities to meet face-to-face with the greatest minds in the enterprise integration space. This year was no exception, with at least 24 sessions featuring as many speakers from both the Microsoft and the MVP community.
As usual, the first half of the 3-day conference was devoted to the Microsoft product team, with presentations from Jon Fancy (who also gave the keynote), Kevin Lam, Derek Li, Jeff Hollan, Paul Larsen, Valerie Robb, Vladimir Vinogradsky, Miao Jiang, Clemens Vasters, Dan Rosanova, Divya Swarnkar, Kent Weare, Amit Kumar Dua, and Matt Farmer. For me, the highlights of these sessions were:
- Jon Fancey’s keynote address. Jon talked about the inevitability of change, underpinning this with a collection of images showing how much technology has progressed in the last 30-40 years alone. Innovation often causes disruption, but this isn’t always a bad thing; the phases of denial and questioning eventually lead to enlightenment. Jon included several notable quotes including one by the historian Lewis Mumford (“Continuities inevitably represent inertia, the dead past; and only mutations are likely to prove durable.”) and a final one by Microsoft CEO Satya Nadella (“What are you going to change to create the future?”).
- Kevin Lam and Derek Li taking us through the basics of Logic Apps. It’s easy to become an “Integration Hero” using Logic Apps to solve integration problems quickly. Derek built two impressive demos from scratch during this session, including one that performed OCR on an image of an order and then conditionally sent an approval email based on the amount of the order.
- Jeff Hollan and his explanation of Azure Durable Functions. An extension of Azure functions, Durable Functions enable long-running, stateful, serverless operations which can execute complex orchestrations. Built on the Durable Task framework, they manage state, checkpoints and restarts. Jeff also gave some practical advice about scalable patterns for sharing code resources across multiple instances.
- Paul Larsen and Valerie Robb with a preview of Feature Pack 3 for BizTalk Server 2016. This upcoming feature pack will include GDPR support, SQL Server 2016 SP2 support (including the long-awaited ability to deploy multiple databases per instance of an Availability Group), new Office365 adapters, and advanced scheduling support for receive locations.
- Miao Jiang and his preview of end-to-end tracing via Application Insights in API Management. This includes the ability to map an application, view metrics, detect issues, diagnose errors, and much more.
- Clemens Vasters. (Need I say more?) The genius architect behind Azure Service Bus gave a clear, in-depth discussion about the difference between messaging (which is about intents) and eventing (which is about facts). He also introduced Event Grid’s support for CNCF Cloud Events, and open standard for events.
- Dan Rosanova and his clear explanation of the different cloud messaging options available in Azure. His slide showing the segmentation of services into Serverless, Big Data and Enterprise provided a lot of clarity as to when to use what. He also covered Event Grid, Event Hubs, and Event Hub’s support for Kafka.
- Jon Fancey & Divya Swarnkar talking about Logic Apps for enterprise integration. It was especially nice to see support for XSLT 3.0 and Liquid for mapping, as well as the new SAP connector. Also there was an enthusiastic response from the audience when revealed there were plans to introduce a consumption based model for Integration Accounts!
- Kent Weare talking about Microsoft Flow and how it is the automation tool for Office365. Together with PowerApps. Power BI, and Common Data Services you can can build a Business Application Platform (BAP) that supports powerful enterprise-grade applications. Best of all, Flow makes it possible for citizen integrators to automate non-critical processes without the need for a professional development team. A notable quote in Kent’s presentation was “Transformation does not occur while waiting in line!”
- Divya Swarnkar and Amit Kumar Dua talking about how Microsoft IT use Logic Apps for Enterprise Integration. It is always interesting to see how Microsoft “dog foods” its own technology to support in-house processes. In this case they built a Trading Partner Migration tool to lift more than 1000 profiles from BizTalk into Logic Apps. Highlights from the lessons learnt were understanding the published limits of Logic Apps and also that there is no “exactly once” delivery option.
- Vladimir Vinogradsky with a deep-dive into API Management. Covering the topics of developer authentication, data plane security, and deployment automation, Vladimir gave some great tips including how to use Azure AD and Azure AD B2C for different scenarios, identifying the difference between key vs. JWT vs. client certificate for authorisation (as well as when to use which), and the use of granular service and API templates to improve the devops story. It was also fun watching him launch his cool demos using Visual Studio Code and httpbin.org.
- Kevin Lam and Matt Farmer’s deep-dive into Logic Apps. This session yielded an enthusiastic applause from the audience with Kevin’s announcement of the upcoming preview of Integration Service Environments. This is a true game-changer bringing dedicated compute, isolated VNET connectivity, custom inbound domain names, static outbound IPs, and flat cost to Logic Apps. There was also a demonstration by Matt of building a custom connector, and how an ISV can turn this into a “real” connector providing it is for a SaaS service you own.
- Derek Li and Kevin Lam with Logic App patterns and best practices. This session sported some great performance improving and cost saving advice by limiting the number of actions with clever loop processing logic (hint: check out the Filter Array action as per this blog post). There was also a tip on how to easily distinguish built-ins, connectors and enterprise connectors: you can filter for these when choosing a connector in the dialog.
- Jon Fancey and Matt Farmer talking about the Microsoft integration roadmap. Key messages here were Jon confirming that Logic App will be coming to Azure Stack, and that Integration Service Environments (ISE) will also eventually come to on-premises. Matt also talked about Azure Integration Services, which on the surface appears to be a new marketing name to the collection of existing services (API Management, Logic Apps, Service Bus and Event Grid). However it promises to include reference architectures, templates and other assets to provide better guidance on using these services.
This concluded the Microsoft presentations on the agenda. After a brief introduction to the BizTalk360 Partnership Program by Business Development Manager Duncan Barker, the MVP sessions began, featuring speakers Saravana Kumar, Steef-Jan Wiggers, Sandro Pereira, Stephen W. Thomas, Richard Seroter, Michael Stephenson, Johan Hedberg, Wagner Silveira, Dan Toomey, Toon Vanhoutte and Mattias Lögdberg. Highlights for me included:
- Saravana Kumar giving a BizTalk360 product update. Saravana revealed that ServiceBus360 is being renamed to Serverless360, which supports not only Service Bus queues, topics and relays, but also Event Hubs, Logic Apps and API Management. And there is an option to either let BizTalk360 host it or host on-prem in your own datacentre.
- Steef-Jan Wiggers talking about Serverless Messaging in Azure. This was a great roundup of all the messaging capabilities in Azure including Service Bus, Storage Queues, Event Hubs and Event Grid. Steef-Jan explained the difference between each very clearly before launching into some very compelling demos.
- Saravana Kumar giving an Atomic Scope update. A product that provides end-to-end tracking for hybrid solutions featuring BizTalk Server, Logic Apps, Azure Functions, and App Services, Atomic Scope can reduce the time spent building this capability from 20-30% right down to 5%. Saravana affirmed that it is designed for business users, not technical users. Included in this session was a real-life testimonial by Bart Scheurweghs from Integration.Team where Atomic Scope solved their tracking issues across a hybrid application built for Van Moer Logistics.
- Sandro Pereira with his BizTalk Server lessons from the road. Sandro’s colourful presentations are always full of good advice, and this was no exception. Whilst talking about a variety of best practices from a security perspective, Sandro reminded us that the BizTalk Server platform is inherently GPDR compliant – but not necessarily the applications we build on it! He also talked about the benefits of feature packs (which may contain breaking code) and cumulative updates (which shouldn’t break code), being wary of JSON schema element names with spaces in them, REST support in BizTalk (how it doesn’t handle optional parameters OOTB), and cautioning against use of some popular patterns like singletons and sequence convoys which are notorious for introducing performance issues and zombies.
- Stephen W. Thomas and using BizTalk Server as your foundation to the cloud. Stephen talked about a bunch of scenarios that justify migration to or use of Logic Apps as opposed to BizTalk Server, including use of connectors not in BizTalk, reducing the load on on-prem infrastructure, saving costs, improved batching capability, and planning for the future. He also identified a number of typical “blockers” to cloud migration and discussed various strategies to address them.
- Richard Seroter on architecting highly available cloud solutions. One of the highlights of this event for me was Richard’s extremely useful advice on ensuring your apps are architected correctly for high-availability. Taking us through a number of Azure services from storage to databases to Service Bus to Logic Apps and more, Richard pointed out what is provided OOTB by Azure and what you as the architect need to cater for. His summary pointers included 1) only integrate with highly available endpoints, 2) clearly understand what services failover together, and 3) regularly perform chaos testing.
- Michael Stephenson and using Microsoft Flow to empower DevOps. It was great to see Michael return to the stage this year, and his Minecraft hosted demo of an automated process for onboarding users was a real crowd-pleaser. Adhoc tasks such as these can waste time and money, but automating the process with Flow not only mitigates this but is also relatively easy to achieve if the process can be broken down to well-understood tasks.
- Unfortunately I had to miss Johan Hedberg’s session on VSTS and BizTalk Server, but by all accounts it was a demo-heavy presentation that provided lots of useful tips on establishing a CI/CD process with BizTalk Server. I look forward to watching the video when it’s out.
- Wagner Silveira on exposing BizTalk Server to the world. This session continued the hybrid application theme by discussing the various ways of exposing HTTP endpoints from BizTalk Server for consumption by external clients. Options included Azure Relay, Logic Apps, Function Proxies and API Management – each with their own strengths and limitations. Wagner performed multiple demos before summarising the need to identify your choices and needs, and “find the balance”.
- Dan Toomey and the anatomy of an enterprise integration architecture. Well… I didn’t take any notes on this one, seeing as I was onstage speaking! But you can download the slides to get an overview of how Microsoft integration technologies can be leveraged to reduce friction across layers of applications that move at different speeds.
- Toon Vanhoutte’s presentation on using webhooks with BizTalk Server. In his first time appearance at INTEGRATE, Toon gave a very compelling talk about the efficiency of webhooks over a polling architecture. He then walked through the three responsibilities of the publisher (reliability, security, and endpoint validation) as well as the five responsibilities of the consumer (high availability, scalability, reliability, security, and sequencing) and performed multiple demos to illustrate these.
- Wrapping up the presentations was Mattias Lögdberg on refining BizTalk Server implementations. Also a first-time speaker at INTEGRATE, Mattias wowed us by doing his entire presentation as a live drawing on his tablet! He used this medium to walk us through a real-life project where a webshop was migrated to Azure whilst retaining connectivity to the on-prem ERP system. A transition to a microservices-based approach and addition of Azure services enhanced the capabilities of BizTalk which still handles the on-prem integration needs.
The conference finished with a Q&A session with the principal PMs from the product group, including Jon Fancey, Paul Larsen, Kevin Lam, Vladimir Vinogradsky, Kent Weare, and Dan Rosanova as panellists. The inevitable question of BizTalk vNext came up, and there was some obvious frustration from the audience as the panellists were unable to provide a clear and definitive statement about whether BizTalk Server 2016 would be the final release or not. Considering that we are halfway through the main support period for this product, the concern is very understandable as we need to know what message to convey to our clients. Reading between the lines, many have concluded for themselves that there will be no vNext and that migration to the cloud is now becoming a critical urgency. Posts such as this one from Michael Stephenson help to put things in context and show that compared to competing products we’re not as bad off as some alarmists may suggest, but it was still disappointing to end the conference with such a vague view of the horizon. I hope that Microsoft sends out a clear message soon about what level of support can be expected for the significant investment that many of our clients have made in BizTalk recently.
One especially cool thing that the organisers did was to hire Visual Scribing to come and draw a mural of all the presentations, capturing the key messages throughout the conference:
Day 2 & 3
For more in-depth coverage of the sessions and announcements from INTEGRATE 2018, I encourage you to check out the following:
Also, keep an eye on the INTEGRATE 2018 website, as eventually the video and the slide decks will be uploaded there.
I really want to thank Saravana and his whole team at BizTalk360 for organising such a mammoth event where everything ran so smoothly. I also want to thank Microsoft and my MVP colleagues for their contributions to the event, as well as my generous employer Mexia for sending me to London to enjoy this experience!
Last Saturday I had the great privilege of organising and hosting the 2nd annual Global Integration Bootcamp in Brisbane. This was a free event hosted by 15 communities around the globe, including four in Australia and one in New Zealand!
It’s a lot of work to put on these events, but it’s worth it when you see a whole bunch of dedicated professionals give up part of their weekend because they are enthusiastic to learn about Microsoft’s awesome integration capabilities.
The day’s agenda concentrated on Integration Platform as a Service (iPaaS) offerings in Microsoft Azure. It was a packed schedule with both presentations and hands-on labs:
It wasn’t all work… we had some delicious morning tea, lunch and afternoon tea catered by Artisan’s Café & Catering, and there was a bit of swag to give away as well thanks to Microsoft and also Mexia (who generously sponsored the event).
Overall, feedback was good and most attendees were appreciative of what they learned. The slide decks for most of the presentations are available online and linked above, and the labs are available here if you would like to have a go.
I’d like to thank my colleagues Susie, Lee and Adam for stepping up into the speaker slots and giving me a couple of much needed breaks! I’d also like to thank Joern Staby for helping out with the lab proctoring and also writing an excellent post-event article.
Finally, I be remiss in not mentioning the global sponsors who were responsible for getting this world-wide event off of the ground and providing the lab materials:
- Martin Abbott
- Glenn Colpaert
- Steef-Jan Wiggers
- Tomasso Groenendijk
- Eldert Grootenboer
- Sven Van den brande
- Gijs in ‘t Veld
- Rob Fox
Really looking forward to next year’s event!
For years now Integration Monday has been faithfully giving us webinars almost every week. There have been some outstanding sessions from international leaders in the integration space including MVPs, members of the Microsoft product team, and other community members. For the Asia Pacific community, however, it has always been a challenge to participate in the live sessions due to the unfriendly time zone. (I certainly know what a struggle it was to present my own session last October at 4:30am!) Even from the listener’s perspective, it is usually nicer to be able to join a live webinar and ask questions rather than to consume the recordings afterwards.
Thanks to the initiative of veteran MVP Bill Chesnut (aka “BizTalk Bill”) and the sponsorship of his employer SixPivot, we now have a brand new webinar series starting up in a friendlier time slot for our APAC community! Integration Down Under is launching its inaugural webinar session on Thursday, 8th February at 7:00pm AEST. You can register for this free event here.
This initial session will introduce the leaders and allow each of us to present as very short talk on a chosen topic:
There are already more than twenty registrations even though the link has been live for only a few days. I hope that this is a good sign of the interest within the community!
Feeling really fortunate to be part of this initiative, and looking forward to delivering my intro to Event Grid talk! It will be a slightly scaled down version of what I presented at the Sydney Tech Summit back in November. Hope to see you there!
(This post was originally published on Mexia’s blog on 1st September 2017)
Microsoft recently released the public preview of Azure Event Grid – a hyper-scalable serverless platform for routing events with intelligent filtering. No more polling for events – Event Grid is a reactive programming platform for pushing events out to interested subscribers. This is an extremely significant innovation, for as veteran MVP Steef-Jan Wiggers points out in his blog post, it completes the existing serverless messaging capability in Azure:
- Azure Functions – Serverless compute
- Logic Apps – Serverless connectivity and workflows
- Service Bus – Serverless messaging
- Event Grid – Serverless Events
And as Tord Glad Nordahl says in his post From chaos to control in Azure, “With dynamic scale and consistent performance Azure Event grid lets you focus on your app logic rather than the infrastructure around it.”
The preview version not only comes with several supported publishers and subscribers out of the box, but also supports customer publishers and (via WebHooks) custom subscribers:
In this blog post, I’ll describe the experience in building a sample logging mechanism for a service hosted in Azure Service Fabric. The solution not only logs all events to table storage, but also sends alert emails for any error events:
Creating the Event Grid Topic
This was an extremely simple process executed in the Azure Portal. Create a new item by searching for “Event Grid Topic”, and then supply the requested basic information:
Once created, the key items you will need once the topic is created is the Topic Endpoint and the associated key:
Creating the Event Publisher
As mentioned previously, there are a number of existing Azure services that can publish events to Event Grid including Event Hubs, resource groups, subscriptions, etc. – and there will be more coming as the service moves toward general availability. However, in this case we create a custom publisher which is a service hosted in Azure Service Fabric. For this sample, I used an existing Voting App demo which I’ve written about in a previous blog post, modifying it slightly by adding code to publish logging events to Event Grid.
The first requirement was storing the topic endpoint and key in the parameter files, and of course creating the associated configuration items in the ServiceManifest.xml and ApplicationManifest.xml files (this article provides information about application configuration in Service Fabric):
Note that in a production situation the TopicKey should be encrypted within this file – but for the purposes of this example we will keep it simple.
Next step was creating a small class library in the solution to house the following items:
- The Event class which represents the Event Grid events schema
- A LogEvent class which represents the “Data” element in the Event schema
- A utility class which includes the static SendLogEvent method
- A LogEventType enum to define logging severity levels (ERROR|WARNING|INFO|VERBOSE)
To see an example of how to create the Event class, refer to fellow Azure MVP Eldert Grootenboer’s excellent post. The only changes I made were to assign the properties for my custom LogEvent, and to add a static method for sending a collection of Event objects to Event Grid (notice how the Event.Subject field is a concatenation of the Application Name and the LogEventType – this will be important later on):
The utility method that creates the collection and invokes this static method is pretty straight forward:
This all makes it simple to embed logging calls into the application code:
Creating the Event Subscribers
Capturing All Events
The first topic subscription will be an Azure Function that will write all events to Azure table storage. Provided you’ve created your Function App in a region that supports the Event Grid preview (I’ve just created everything aside from the Service Fabric solution within the same resource group and location), you will see that there is already an Event Grid Trigger available to choose. Here is my configured trigger:
As you can see, I’ve also configured a Table Storage output. The code within this function creates a record in the table using the Event.Subject as a partition and the Event.Id as the row key:
Using the free Azure Storage Explorer tool, we can see the output of our testing:
Alerting on ERROR Events
Now that we’ve completed one of the two subscriptions for our solution, we can create the other subscription which will use a filter on ERROR events and raise an alert via sending an email notification.
The first step is to create the Logic App (in the same region as the Event Grid) and add the Event Grid Trigger. There are a few things to watch out for here:
- When you are prompted to sign in, the account that your subscription belongs to may or may not work. If it doesn’t, try creating a Service Principal with contributor rights for the Event Grid topic (here is an excellent article on how to create a service principal)
- The Resource Type should be Microsoft.EventGrid.topics
- The Suffix field contains “ERROR” which will serve as the filter for our events
- If the Resource Name drop-down list does not display your Event Grid topic at first, type something in, save it and then click the “x”; the list should hopefully appear. It is important to select from the list as just typing the display name will not create the necessary resource ID in the topic field and the subscription will not be created.
You can then follow this with an Office365 Email action (or any other type of notification action you prefer). There are four dynamic properties that are available from the Event Grid Trigger action (Subject, ID, Event Type and Event Time):
After saving the Logic App, check for any errors in the Overview blade, and then check the Overview blade for the Event Grid Topic – you should see the new subscription created there:
Finally, we can test the application. My Voting demo service generates an exception (and a ERROR logging event) when a vote is cast for a null/empty candidate (see the ERROR entry in the table screenshot above). This event now triggers an email notification:
So this example may not be the niftiest logging application on the market (especially considering all of the excellent logging tools that are available today), but it does demonstrate how easy it is to get up and running with Event Grid. You’ve seen an example of using a custom publisher and two built-in subscribers, including one with intelligent filtering. To see how to write a custom subscriber, have a look at Eldert’s post “Custom Subscribers in Event Grid” where he uses an API App subscriber to write shipping orders to table storage.
Event Grid is enormously scalable and its consumption pricing model is extremely competitive. I doubt there is anything else quite like this on offer today. Moreover, there will be additional connectors coming in the near future, including Azure AD, Service Bus, Azure Data Factory, API Management, Cosmos DB, and more.
For a broader overview of Event Grid’s features and the capabilities it brings to Azure, have a read of Tom Kerkhove’s post “Exploring Event Grid”. And to understand the differences between Event Hub, Service Bus and Event Grid, Saravana Kumar’s recent post sums it up quite nicely. Finally, if you want to get your hands dirty and have a play, Microsoft has provided a quickstart page to get you up and running.
Last week I had the privilege not only of attending the INTEGRATE 2017 conference in London, but presenting as well. A huge thanks to Saravana Kumar and BizTalk360 for inviting me as a speaker – what a tremendous honour and thrill to stand in front of nearly 400 integration enthusiasts from around the world and talk about Hybrid Connectivity! Also, a big thanks to Mexia for generously funding my trip.
With 380+ attendees from 52 countries around the globe, this is by far the biggest Microsoft integration event of the year. Of those 380, only four of us that I know of came from APAC: fellow MVP speakers Martin Abbott from Perth and Wagner Silveira from Auckland NZ, as well as Cameron Shackell from Brisbane who manned his ActiveADAPTER sponsor stand. Wagner would have to take the prize for the furthest travelled with his 30+ hour journey!
I’ve already published one blog post summarising my take on the messages delivered by Microsoft (which accounted for half of the sessions at the event). This will soon be followed by a similar post with highlights of the MVP community presentations, which in addition to BizTalk Server, Logic Apps, and other traditional integration topics also spanned into the new areas of Bots, IoT and PowerApps.
Aside from the main event, Saravana and his team also arranged for a few social events as well, including networking drinks after the first day, a dinner at Nando’s for the speakers, and another social evening for the BizTalk360 partners. They also presented each of the BizTalk360 product specialists with a beautiful award – an unexpected treat!
You have to hand it to Saravana and his team – everything went like clockwork, even keeping the speakers on schedule. And I thought it was a really nice touch that each speaker was introduced by a BizTalk360 team member. Not only did it make the speakers feel special, but it provided an opportunity to highlight the people behind the scenes who not only work to make BizTalk360 a great product but also ensure events like these come off. I hope all of them had a good rest this week!
As with all of these events, one of the things I treasure the most is the opportunity to catch up with my friends from around the globe who share my passion for integration, as well as meeting new friends. In my talk, I commented about how strong our community is, and that we not only integrate as professionals but integrate well as people too.
Arriving a day and a half before the three day event, I had hoped to conquer most of the jet-lag early on. But alas, the proximity to the solstice in a country so far North meant the sun didn’t set until past 10:30pm while rising just before 4:30am – which is the time I would involuntarily wake up each day no matter how late I stayed up the night before! Still, adrenalin kept me going and the engaging content kept me awake for every session.
And no matter what.., there was always time for a beer or two!
I look forward to the next time I get to meet up with my integration friends! If you missed the event in London, you’ll have a second chance at INTEGRATE 2017 USA which will be held in Redmond on October 25-27. And of course, if you keep your eyes on the website, the videos and slides should be published soon.
(Photos by Nick Hauenstein, Dan Toomey, Mikael Sand, and Tom Canter)
(This post was originally published on Mexia’s blog on 19th June 2017)
Microsoft recently announced that Azure BizTalk Services (MABS) is officially being retired. This was no great surprise, as those who actually used this service and its VETER pipelines to build integrations were well aware that the tooling was cumbersome, the DevOps story was terrible, scalability was severely limited, and the management capabilities left much to be desired. Logic Apps and the Enterprise Integration Pack already have already far surpassed the capabilities of MABS for cloud-based integration and B2B (EDI) scenarios. However, the one really useful feature of MABS was the free Hybrid Connections capability – free because this feature never made it out of preview mode.
Hybrid Connections allowed you to easily connect your Web App or Mobile service to an on-premises resource without making any changes to your corporate network, traversing NATS, routers, firewalls etc. with a purely codeless solution. In fact, you could literally “lift & shift” your existing on-prem website to Azure and not even have to alter the connection string to your database. Moreover, it worked at the transport layer so there was no dependency on WCF or .NET. I was so intrigued by the capabilities of this service that I authored a Pluralsight course on it, as well as creating a webcast and writing several blog posts.
With the obvious signs over the past year or so that MABS was on its way out, this had us wondering what would happen to Hybrid Connections? Other non-network related technologies like Service Bus Relay and the newer On-Premises Data Gateway certainly offer some viable alternatives, but nothing that permitted the same flexibility as Hybrid Connections. Fortunately, late last year we got our answer – the new Azure Relay.
A New Offering
Azure Relay became generally available on 27 March 2017, less than five months after the preview was announced. This service actually is comprised of two capabilities: the WCF Relay (which is the new name of the existing Service Bus Relay), and the new version of Hybrid Connections. This version of the latter is everything that the former version was, but much more:
- It is no longer hosted in a sunsetted technology (lives in Azure Service Bus)
- A published API means that the capability is no longer confined to Azure Web Apps and Mobile Services
- Reliance on web sockets means it is truly a cross-platform solution
In my Pluralsight course and in my previous webcast, I proved how easy it was to enable a single Azure hosted web site to talk to two separate on-premises resources (a web service and a SQL Server database). That capability exists in the new Hybrid Connections and can be set up in exactly the same way; a convenient downloadable manager agent can be installed in seconds which will complete the listener setup and allow you to flow messages into your network. I was easily able to recreate the same demo scenario in my webcast with the new version.
But even more compelling was the experience at using the API to build a more flexible solution, for example connecting an Azure hosted VM to an on-premises resource. Here, the supplied samples on GitHub (conveniently for both .NET and Node) really prove the extensive capabilities of this service.
The simple sample introduces you to the Hybrid Connections API by building a simple relay that bi-directionally exchanges blocks of text over a connection. I had this up & running in minutes just by deploying the server app on my local Hyper-V hosted VM and deploying the client application on an Azure-hosted VM. There were a few minor glitches to get over, for example how to correctly format the SQL Server connection string to talk to the local SQLEXPRESS instance I had running. This post helped me realise that I needed to explicitly include the port number in the connection string, as well as ensure that the TCP/IP settings for SQLEXPRESS used a static IP address (which is does not by default).
Of course you need to first create the Hybrid Connection in Azure, which you do in the portal by first creating an Azure Relay:
This sets up the Service Bus namespace that is associated with the relay artefacts. Then you can create the Hybrid Connection:
By going into the Shared Access Policies menu item for the Relay, you can copy the Primary Key for the “RootManageSharedAccessKey” – a necessary item for configuring the applications that will use the relay. (Of course you can and should configure your own policies for Manage, Send and Listen – but I didn’t for this example).
When you open any of the solutions from the GitHub samples, you will need to update the Microsoft.Azure.Relay Nuget package before compiling the solution:
Then it’s a matter of copying the solution to your servers, running the Client app on the Azure hosted machine and the Server app on the on-prem machine. I was impressed by the speed of the connection. Messages typed into the client appeared instantly in the server console window with absolutely no perceptible delay! Given the fact that under the covers, the relay uses Service Bus queues to transfer messages, I would have expected and happily tolerated a slight latency here – but there was none! I even opened multiple instances of the client to see how that would work:
As expected, all of the messages were visible in the server-hosted console, along with logging of sessions opening and closing:
Although the application in this format isn’t particularly useful for any production scenarios I can think of, it certainly proves that hybrid connectivity is no longer limited to Web Apps.
The next step was to try out the Port Bridge sample, which allows you to set up a hybrid connection that allows you to setup multiplexed socket tunnels that can bridge TCP and Named Pipe socket connections across the relay. This is an amazingly flexible demonstration as once you deploy the code, you can add SSH-like connections through configuration only, including imposing firewall-like restrictions – all without any involvement of your physical network configuration.
Again, I was keen to prove that this would allow connectivity across multiple nodes and ports, so I resorted again to my previous demo example and set up connections to both an locally hosted WCF service (on port 80) and my SQL Server database (on port 1433). The WCF service was really nothing more than a simple echo function that also outputs the name of the local server (same as used in my Guestbook demo in the webcast):
And for the database connectivity, all I needed to do was prove that a simple query would return the appropriate number of rows in a table:
I created two console test apps:
- One that called the WCF service
- One that executed a scalar query of the number of rows in the Refreshments table above
After testing both console apps locally on the Hyper-V VM, I then deployed these to my Azure VM, along with the portbridge solution (which was also deployed locally). The next step was to create the Hybrid Connection in Azure, which I did by adding to the existing Relay. The trick here was to ensure that the name of the relay (normally anything you like it to be) had to be the same name as your target host name (in this case, it was “devdan10”, the name of the VM hosted in my local Hyper-V).
Getting the configuration right took me a few goes, but eventually got there. Here is the configuration for the on-prem Server Agent console:
Notice on Line 10 that I’ve referenced the name of my target host (“devdan10”) and listed the port numbers which I will allow to be opened to the relay.
And here is the configuration for the Client Agent service running in Azure:
The local and remote ports happen to be the same here (lines 10 & 16), but they could be different – allowing you to map different ports between the remote client and the target local machine. The firewall rules in this case ensure that the requests come from the client machine – but these can also be adjusted to be as lenient or strict as desired.
There were no changes at all required for the WCF test client, in fact I was able to use “localhost” as the hostname in the client endpoint definition (line 53)!
One thing I did need to do was ensure that the connection string in the SQL test client used a TCP protocol and (again) a reference to the local machine rather than the target host name (this is presumably because the portbridge client service maps everything from the local machine on that port to the target remote server):
Note that you do need to use SQL Authentication rather than Windows Authentication, as there is no support for Active Directory authentication here.
With that in place I was able to run tests with both clients to prove connectivity over the ports I had configured for forwarding. The first test screenshot show the WCF client test app successfully invoking the service remotely. The window in the background is the console for the Port Bridge Client Agent that runs on the Azure machine, forwarding requests to the remote on-prem machine by way of the Hybrid Connection relay:
The Port Bridge Service Agent console windows runs on the on-prem machine, and logs the connections:
By the way, it is worth mentioning that both the Port Bridge Service Agent and the Client Agent can be installed and run as a Windows service.
The second test is the same thing but with the SQL Server database test client, correctly retrieving the row count from the on-prem machine:
How good is that?? You now have the capability to open of a port-based bridge between cloud and on-premises machines by adding configuration to the client and server agent services, meaning you can successfully connect your IaaS VMs and cloud services [almost] as easy as you connect your Web Apps and Mobile Apps.
In a week’s time I will be speaking about Hybrid Connectivity at the INTEGRATE 2017 conference in London. Unfortunately there won’t be time for demos, but this blog post might make for a handy follow-up reference.
(This post was originally published on Mexia’s blog on 07 April 2017)
Last weekend I had the great privilege of attending my first Microsoft MVP event – the MVP Community Connection held in Sydney. This was an invitation to all Australian MVPs to come together over two days to network, receive central communication from the program managers, and learn some best practices in how we can better serve the community.
First I should explain what an MVP is. The Microsoft Most Valuable Professional award is presented to recognised experts in the technical community who regularly and voluntarily share their passion and knowledge with others, including Microsoft itself. The award celebrates an individual’s deep commitment to serving the community and promoting awareness of Microsoft’s great products through a variety of channels – be it running user groups, speaking at events, blogging, mentoring, answering questions on forums, creating and sharing free software, etc. Aside from recognition, the award includes a number of benefits and privileges, including direct access to the Microsoft product teams via email distribution lists and Yammer, invitations to early previews and Product Group Interaction (PGI) meetings, exam vouchers, free software and subscriptions, and more. In return, Microsoft expects us to use these benefits and privileges to facilitate our continued involvement in the community and also to provide valuable feedback to Microsoft to help them design and build better products. The award lasts for one year, after which it may be renewed if the candidate has continued to demonstrate exemplary commitment and service.
The award is always granted in a particular Microsoft technological area or product for which the candidate has demonstrated expertise. My award was presented in February this year in the category of Azure – one of the broadest areas which happens to include enterprise integration. This was presumably in recognition of my efforts in running two user groups, speaking at multiple events (including Ignite Australia), writing posts on both Mexia’s and my personal blog, and promoting Microsoft events and announcements on social media. Yet, I always feel that I should be doing much more to serve an integration community that is always so willing to share and so appreciative of the efforts of those who do share!
This latest MVP event in Sydney was designed to allow us to come together and share ideas about how to improve our community leadership. It began with some fun networking activities on Friday afternoon & evening, followed by a more formal schedule of sessions on Saturday. It was a nice touch to arrive at the venue on Saturday morning and be presented with a personalised appreciation card complete with a custom Lego figurine. It quickly became evident that each Lego figurine was carefully selected to match the physical characteristics of the recipient in terms of hair colour, gender, etc!
After a warm welcome by Community Program Manager Lana Montgomery and a program update by Chris Olson from the DX team, there was a keynote about Digital Transformation by Technical Evangelist Vaughan Knight. This was followed by a series of enlightening “Skills Sessions” presented by prominent members of the MVP community including Troy Hunt, Orin Thomas, Marc Kean, Robert Crane, Leon Tribe and David Gardiner, spanning topics from developing your personal brand to blogging like a pro to recharging your user group. These were followed by some round table discussions around various topics relating to community involvement such as growing technical communities, embracing emerging technologies, supporting entrepreneurs and start-ups, and assisting young student technologists. The day wrapped up with some closing remarks by Lana and then a trip to the Helm Bar in Darling Harbour for some refreshing drinks before heading to the airport.
The event was not only well organised, informative and enjoyable, but I also found it truly inspiring to meet up with so many other enthusiastic technologists! While you might expect it to be intimidating to be a newbie in the company of so many extraordinarily brilliant and accomplished individuals, I actually found everyone to be extremely friendly and welcoming. I made many new friends that day as well as catching up with some old friends like Bill Chesnut, Mick Badran, Shane Hoey and Martin Abbott. Kudos to Lana for her exceptional organisational skills and her contagious positive energy! Really looking forward to more occasions like these, as well as the continuing support from Microsoft which helps us MVPs better support the wider community.
Photos courtesy of Lana Montgomery
Last week I had the opportunity to attend Microsoft Ignite on the Gold Coast, Australia. Even better – I had a free ticket on account of agreeing to serve as a Technical Learning Guide (TLG) in the hands-on labs. This opportunity is only open to Microsoft Certified Trainers (MCTs) and competition was evidently keen this year – so I am glad to have been chosen. Catching up with fellow MCTs like Mark Daunt and meeting up with new ones such as Michael Schmitz was a real pleasure. Of course the down side was that I missed quite a few breakout sessions during the times I was rostered. Nevertheless, I still got to see some of the most important sessions to me, particularly those that centred around Azure and integration technologies. Please have a read of my summary of these on my employer’s blog.
By and far this was my best Australian Ignite/Tech-Ed event experience for many reasons, including:
- The Pro-Integration team from Redmond came all the way out to Australia show everyone what the product group is doing with Logic Apps, Flow, Service Bus, and BizTalk Server
- I was chosen to present an Instructor-Led Lab in Service Fabric – my first ever speaking engagement at Ignite
- I had the rare opportunity to catch up with some fellow MVPs from Perth and Europe.
It was truly phenomenal to see enterprise integration properly represented at an Australian conference, as it is typically overlooked at these events. In addition to at least four breakout sessions on hybrid integration, Scott Guthrie actually performed a live demo of Logic Apps in his keynote! This was a good shout-out to the product team that has worked so hard to bring this technology up to the usability level it now enjoys. I’m glad that Jim Harrer, Jeff Holland, Jon Fancey and Kevin Lam were there to see it!
Teaching the lab in Service Fabric was a thrilling experience, but not without some challenges. The lab itself was broken and required a re-write of the second half, which I had pre-prepared and uploaded to One-Drive here so the students could progress. The main lab content is only available to Ignite attendees, however if you want to have a go at a similar lab you can try these ones available from Microsoft:
Despite the frustration that some attendees expressed about the lab errata and the poor performance of the environment, I was pleased that all the submitted feedback relating to the speaker was very positive!
Finally, perhaps the best part of events like these is the ability to catch up with old friends and meet some new ones. It was a pleasure to hang out with Azure MVP Martin Abbott from Perth and meet a few of his colleagues. It was also great to see Elder Grootenboer and Steef-Jan Wiggers from the Netherlands, who happened to travel to Australia this month on holidays and to speak at some events. Steef-Jan also took time to include me in a V-Log series he’s been working on with various integration MVPs, recording his 3-minute interview with me at the top of Mount Coot-tha on a sunny Brisbane Saturday! And Mexia’s CEO Dean Robertson & myself got to enjoy a nice dinner out with the Microsoft product group and the MVPs.
All good things must come to an end, but it was definitely a memorable week! Now it’s time to start getting ready for the Brisbane edition of the Global Integration Bootcamp on Saturday, 25th March, to be followed not long after by the Global Azure Bootcamp on Saturday 22nd April! I’ve got a few demos and presentations to prepare – but now with plenty of inspiration from Ignite!