by Rob Callaway | Nov 2, 2015 | BizTalk Community Blogs via Syndication
In conversations with students and other integration specialists, I’m discovering more and more how confused some people are about the evolution of cloud-based integration technologies. I suspect that cloud-based integration is going to be big business in the coming years, but this confusion will be an impediment to us all.
To address this I want to write a less technical, very casual, blog post explaining where we are today (November of 2015), and generally how we got here. I’ll try to refrain from passing judgement on the technologies that came before and I’ll avoid theorizing on what may come in the future. I simply want to give a timeline that anyone can use to understand this evolution, along with a high-level description of each technology.
I’ll only speak to Microsoft technologies because that’s where my expertise lies, but it’s worth acknowledging that there are alternatives in the marketplace.
If you’d like a more technical write-up of these technologies and how to use them, Richard Seroter has a good article on his blog that can be found here.
Way, way back in October of 2008 Microsoft unveiled Windows Azure (although it wouldn’t be until February of 2010 that Azure went “live”). On that first day, Azure wasn’t nearly the monster it has become.
It provided a service platform for .NET services, SQL Services, and Live Services. Many people were still very skeptical about “the cloud” (if they even knew what that meant). As an industry we were entering a brave new world with many possibilities.
From an integration perspective, Windows Azure .NET Services offered Service Bus as a secure, standards-based messaging infrastructure.
Over the years, Service Bus has been rebranded several times but the core concepts have stayed the same: reduce the barriers for building composite applications, even when their components have to communicate across organizational boundaries. Initially, Service Bus offered Topics/Subscriptions and Queues as a means for systems and services to exchange data reliably through the cloud.
Service Bus Queues are just like any other queueing technology. We have a queue to which any number of clients can post messages. These messages can be received from the queue later by some process. Transactional delivery, message expiry, and ordered delivery are all built-in features.
I like to call Topics/Subscriptions “smart queues.” We have concepts similar to queues with the addition of message routing logic. That is, within a Topic I can define one or more Subscription(s). Each Subscription is used to identify messages that meet certain conditions and “grab” them. Clients don’t pick up messages from the Topic, but rather from a Subscription within the Topic. A single message can be routed to multiple Subscriptions once published to the Topic.
Sample Service Bus Topic and Subscriptions
If you have a BizTalk Server background, you can essentially think of each Service Bus Topic as a MessageBox database.
Interacting with Service Bus is easy to do across a variety of clients using the .NET or REST APIs. With the ability to connect on-premises applications to cloud-based systems and services, or even connect cloud services to each other, Service Bus offered the first real “integration” features to Azure.
Since its release, Service Bus has grown to include other messaging features such as Relays, Event Hubs, and Notification Hubs, but at its heart it has remained the same and continues to provide a rock-solid foundation for exchanging messages between systems in a reliable and programmable way. In June of 2015, Service Bus processed over 1 trillion (1,000,000,000,000) messages! (Starts at 1:20)
As integration specialists we know that integration problems are more complex than simply grabbing some data from System A and dumping it in System B.
Message transport is important but it’s not the full story. For us, and the integration applications we build, VETRO (Validate, Enrich, Transform, Route, and Operate) is a way of life. I want to validate my input data. I may need to enrich the data with alternate values or contextual information. I’ll most likely need to transform the data from one format or schema to another. Identifying and routing the message to the correct destination is certainly a requirement. Any integration solution that fails to deliver all of these capabilities probably won’t interest me much.
So, in a world where Service Bus is the only integration tool available to me, do I have VETRO? Not really.
I have a powerful, scalable, reliable, messaging infrastructure that I can use to transport messages, but I cannot transform that data, nor can I manipulate that data in a meaningful way, so I need something more.
I need something that works in conjunction with this messaging engine.
Microsoft’s first attempt at providing a more traditional integration platform that provided VETRO-esque capabilities was Microsoft Azure BizTalk Services (MABS) (to confuse things further, this was originally branded as Windows Azure BizTalk Services, or WABS). You’ll notice that Azure itself has changed its name from Windows Azure to Microsoft Azure, but I digress.
MABS was announced publicly at TechEd 2013.
Despite the name, Microsoft Azure BizTalk Services DOES NOT have a common code-base with Microsoft BizTalk Server (on second thought, perhaps the EDI pieces share some code with BizTalk Server, but that’s about all). In the MABS world we could create itineraries. These itineraries contained connections to source and destination systems (on-premises & cloud) and bridges. Bridges were processing pipelines made up of stages. Each stage could be configured to provide a particular type of VETRO function. For example, the Enrich stage could be used to add properties to the context of the message travelling through the bridge/itinerary.
Complex integration solutions could be built by chaining multiple bridges together using a single itinerary.
MABS was our first real shot at building full integration solutions in the cloud, and it was pretty good, but Microsoft wasn’t fully satisfied, and the industry was changing the approach for service-based architectures. Now we want Microservices (more on that in the next section).
The MABS architecture had some shortcomings of its own. For example, there was little or no ability to incorporate custom components into the bridges, and a lack of connectors to source and destination systems.
Over the past couple of years the trending design architecture has been Microservices. For those of you who aren’t already familiar with it, or don’t want to read pages of theory, it boils down to this:
“Architect the application by applying the Scale Cube (specifically y-axis scaling) and functionally decompose the application into a set of collaborating services. Each service implements a set of narrowly related functions. For example, an application might consist of services such as the order management service, the customer management service etc.
Services communicate using either synchronous protocols such as HTTP/REST or asynchronous protocols such as AMQP.
Services are developed and deployed independently of one another.
Each service has its own database in order to be decoupled from other services. When necessary, consistency is between databases is maintained using either database replication mechanisms or application-level events.”
So the shot-callers at Microsoft see this growing trend and want to ensure that the Azure platform is suited to enable this type of application design. At the same time, MABS has been in the wild for just over a year and the team needs to address the issues that exist there. MABS Itineraries are deployed as one big chunk of code, and that does not align well to the Microservices way of doing things. Therefore, need something new but familiar!
Azure App Service is a cloud platform for building powerful web and mobile apps that connect to data anywhere, in the cloud or on-premises. Under the App Service umbrella we have Web Apps, Mobile Apps, API Apps, and Logic Apps.
I don’t want to get into Web and Mobile Apps. I want to get into API Apps and Logic Apps.
API Apps and logic Apps were publicly unveiled in March of 2015, and are currently still in preview.
API Apps provide capabilities for developing, deploying, publishing, consuming, and managing RESTful web APIs. The simple, less sales-pitch sounding version of that is that I can put RESTful services in the Azure cloud so I can easily use them in other Azure App Service-hosted things, or call the API (you know, since it’s an HTTP service) from anywhere else. Not only is the service hosted in Azure and infinitely scalable, but Azure App Service also provides security and client consumption features.
So, API Apps are HTTP / RESTful services running in the cloud. These API Apps are intended to enable a Microservices architecture. Microsoft offers a bunch of API Apps in Azure App Service already and I have the ability to create my own if I want. Furthermore, to address the integration needs that exist in our application designs, there is a special set of BizTalk API Apps that provide MABS/BizTalk Server style functionality (i.e., VETRO).
This is all pretty cool, but I want more. That’s where Logic Apps come in.
Logic Apps are cloud-hosted workflows made up of API Apps. I can use Logic Apps to design workflows that start from a trigger and then execute a series of steps, each invoking an API App whilst the Logic App run-time deals with pesky things like authentication, checkpoints, and durable execution. Plus it has a cool rocket ship logo.
What does all this mean? How can I use these Azure technologies together to build awesome things today?
Service Bus provides an awesome way to get messages from one place to another using either Queues or Topics/Subscriptions.
API Apps are cloud-hosted services that do work for me. For example, hit a SaaS provider or talk to an on-premises system (we call these connectors), transform data, change an XML payload to JSON, etc.
Logic Apps are workflows composed of multiple API Apps. So I can create a composite process from a series of Microservices.
But if I were building an entire integration solution, breaking the process across multiple Logic Apps might make great sense. So I use Service Bus to connect the two workflows to each other in a loosely-coupled way.
Logic Apps and Service Bus working together
And as my integration solution becomes more sophisticated, perhaps I have need for more Logic Apps to manage each “step” in the process. I further use the power of Topics to control the workflow to which a message is delivered.
More Logic Apps and Service Bus Topics provide a sophisticated integration solution
In the purest of integration terms, each Logic App serves as its own VETRO (or subset of VETRO features) component. Decomposing a process into several different Logic Apps and then connecting them to each other using Service Bus gives us the ability to create durable, long-running composite processes that remain loosely-coupled.
Doing VERTO using Service Bus and Logic Apps
Today Microsoft Azure offers the most complete story to date for cloud-based integration, and it’s a story that is only getting better and better. The Azure App Service team and the BizTalk Server team are working together to deliver amazing integration technologies. As an integration specialist, you may have been able to ignore the cloud for the past few years, but in the coming years you won’t be able to get away with it.
We’ve all endeavored to eliminate those nasty data islands. We’ve worked to tear down the walls dividing our systems. Today, a new generation of technologies is emerging to solve the problems of the future. We need people like you, the seasoned integration professional, to help direct the technology, and lead the developers using it.
If any of this has gotten you at all excited to dig in and start building great things, you might want to check out QuickLearn Training’s 5-day instructor-led course detailing how to create complete integration solutions using the technologies discussed in this article. Please come join us in class so we can work together to build magical things.
by noreply@blogger.com Steef-Jan Wiggers | Dec 17, 2014 | BizTalk Community Blogs via Syndication
The beginning of december I visited the Microsoft Campus (Building 92) for the 3rd global BizTalk Summit, which was branded as Integrate2014. The event was organized this time by BizTalk360 along side with Microsoft itself and core partners like Quicklearn, Nevatech, Codit, and LogicalTech SysTalk. It was a three day event filled with numerous sessions by Microsoft themselves, a few MVP’s and partners.
The first important message to me, partners and end-users I believe was that BizTalk Server is going to play an essential role for integration systems and applications on-premise. Besides that its also provide support with connectivity to cloud (Azure with adapters that support SB-connectivity, and relay). That means BizTalk Server can also provide enterprises the ability to build cloud to ground or ground to cloud based solutions (see also Richard Seroter’s course on Pluralsight called Cloud Integration Patterns). The product group explicitly shared that there will be future platform upgrades (full version, or R2). See also my post on the TechNet Wiki Blog: The Future of Integration.
Second even more important message was the new direction for BizTalk Services. MABS 1.0 will change course to adopt architecture called MicroServices. This means that MABS 2.0 will fundamentally change to support this architecture. What the architecture basically boils down to is that granularity will increase. A BizTalk Service solution can consist of many small services like connectors (called triggers and actions), mediation (that supports VETER pattern we know from MABS 1.0), BPM, Mappings, and Business Rule Engine (BRE). During the event we have seen different sessions on MicroServices from different speakers. All sessions were recorded and will be made available soon.
Finally the number of attendees and partners (sponsors) showed me that ecosystem of Microsoft Integration is very alive. Microsoft will continue to innovate it’s on-premise product BizTalk Server and it’s cloud based offering BizTalk Services, which will be named Azure BizTalk Microservices (MABS 2.0). We stand before the door to the next generation of integration solutions. We slowly we move (migrate) integration solution to Azure Platform and build new solutions to communicate with distributed systems (anywhere, anyplace, anytime) and devices.
Other blog post on INTEGRATE 2014 definitely worth to checkout are:
It has been an amazing event and it was awesome to meet with people like Microsoft Integration MVP’s present, Microsoft BizTalk Product Group, Biztalk360 Team, Quicklearn Team, Andrew Slivker (Nevatech), Jason Sander, Dwaine Gilmour, Mandi Ohlinger, Richard Broida, Tom Canter, Rob Fox, Jean Paul Smit, Martin Peters, Rene Brauwers, JoAnn Een, Michael Shea, Koen van Oost, Eldert Grootenboer, Rex van der Laan, Bart Scheurweghs, Rajesh Kolla, Howard Edidin, and list goes on.
BizTalk360 as event organizer have done an outstanding job taking the BizTalk Summit (Integrate) to the next level!
Cheers,
Steef-Jan
by Rene Brauwers | Feb 4, 2014 | BizTalk Community Blogs via Syndication
Next week I will be doing a talk on Windows Azure BizTalk Services with regards on how one can add BAM functionality. During this talk a demo will be the ‘Pièce de résistance’. this demo is based on an article I have written earlier and which can be found on TechNet. Well to cut to the chase… I would not be who I am if I would have not taken this article to the next level and while doing so I encountered a nice challenge.
In my demo there is a scenario in which I have a custom MessageInspector which can be configured as such that it can deliver messages to either a Servicebus Queue/Topic/Relay or BizTalk Service Bridge endpoint. While testing the various scenario’s I encountered a particular issue when I tried to send a message to another bridge. The error message which was ‘reported back stated’
Component xpathExtractor. Pipeline Runtime Error Microsoft.ApplicationServer.Integration.Pipeline.PipelineException: An error occurred while parsing EntityName. Line 5, position 99.
at Microsoft.ApplicationServer.Integration.Pipeline.Activities.XmlActivitiesExceptionHelper.Throw(String message, XmlActivityErrorCode errorCode, String activityName, String stageName, IEnumerable`1 messageProperties)
The above mentioned error was caused by the error-message which was sent back and caused a failure which I could not easily debug any further without a complete rewrite. While I had no time for this I decided to use a different approach in sending the messages. Up to that point I used the WebClient class in combination with the UploadDataAsync Method but I decided to give the OpenWrite method a go. This decision proved to be very useful as my existing exception-handling was not able ‘kick in’ without causing the parent TASK to go in to a faulted state. Thus by using the OpenWrite method I was able to extract he ‘real’ exception. This exception message stated:
“The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.” “The remote certificate is invalid according to the validation procedure.”
At that point I reached the Eureka moment and it all started to make sense. It all boils down to the fact that for development/test purposes we all use a self-signed certificate (created for us when we provisioned out Windows Azure BizTalk Service). This certificate is installed on our client machines, thus allowing us to make requests to the Windows Azure BizTalk Service. This explained why my test-console application did not throw any exceptions when calling the Windows Azure BizTalk Service Bridge in question. However when this code is invoked from within the message inspector which is hosted in Windows Azure.’ it runs into the fact that ‘there is a problem with the security certificate’, this makes sense as it is a self-signed certificate.
So now that I figured out the why, I needed a way to tell my code to ignore these kind of warning when encountered. Well luckily for is, this little gem is available and can be found in the System.Net.ServicePointManager class. This gem is called ServerCertificateValidationCallBack. This CallBack will return false in case the Server Certificate is not complying with the policies (in our case, it is by default not trusted as it is self-signed), so all we need to do is ensure that it always returns true and thus ignoring the security check
please do not do this in production environments! You otherwise might be ending up sending data to a not trusted source (i.e spoofed server etc.)
Below I added the piece of code which implements the ssl validation-check bypass:
using (var client = new WebClient())
{
Uri EndPointAddress = new Uri(this.Endpoint);
//Add WRAP ACCESS TOKEN
client.Headers[HttpRequestHeader.Authorization] = String.Format("WRAP access_token="{0}"", this.AcsToken);
//Add Content Type
client.Headers["Content-Type"] = "application/xml";
//Publish
try
{
//The 'GEM' Ignore validation callback which cause an error
ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) => true;
using (var stream = client.OpenWrite(EndPointAddress, "POST"))
{
byte[] data = System.Text.Encoding.UTF8.GetBytes(messagePayload);
stream.Write(data, 0, data.Length);
}
}
catch (System.Net.WebException wex)
{
AppendExceptionsToLog(wex);
}
catch (ArgumentNullException anex)
{
AppendExceptionsToLog(anex);
}
catch (Exception ex)
{
AppendExceptionsToLog(ex);
}
}
Cheers and hope to see you soon! Especially if you are going to attend the BizTalk Summit 2014 in London this March, don’t hesitate to say Hi 🙂
Microsoft Integration Stencils Pack for Visio 2016/2013 (10,1 MB)
