Big Data integration in high scale scenario using Azure Service Fabric and GrabCaster

Big Data integration in high scale scenario using Azure Service Fabric and GrabCaster

Big Data is one of the most important topic in the last year, the world of integration has changed since the companies start providing the possibility to store our information in the Cloud.
Send the data from our on premise to a database in the cloud can be achieved in different ways, I’m very keen and focused to implement feature in GrabCaster able to solve integration problems in easy way.

GrabCaster is an open source framework and it can be used in every integration project within any environment type, no matter which technologies, transport protocols or data formats are used.

The framework is enterprise ready and it implements all the Enterprise Integration Patterns (EIPs) and offer a consistent model and messaging architecture to integrate several technologies., the internal engine offers all the required features to realize a consistent solution.

The framework can be hosted in Azure Service Fabric which enables us to build and manage scalable and reliable points running at very high density on a shared pool of machines.
Azure Service Fabric servers as the foundation for building next generation cloud and on premise applications and services of unprecedented scale and reliability, for more information about Microsoft Azure Service Fabric check here

Using GrabCaster and the Microsoft Azure stacks I’m able to execute a SQL Server Bulk Insert operation across on premise and the cloud very easily, below how it works.

I’m not going in detail, you can find all the samples and templates in the GrabCaster site.

Download GrabCaster Framework and configure it to use the Azure Redis Cache, this is one of the last messaging provider implemented, what I love about this Azure stack is the pricing and the different options offered by this framework.

Using the SQL Bulk Trigger and Event I’m able to send a large amount of data across the cloud, the engine compacts the stream and it uses the Blobs to move the large amount of records.

Last the I did was moving one million record from a table in an on premise SQL database and another in the cloud.

I installed GrabCaster in Azure Fabric as below.

I configured the SQL bulk trigger in the on premise environment using the json configuration file.

I configured the SQL Bulk event and I sent the configuration to the GrabCaster point in the Azure Fabric using the internal synchronization channel.

I can activate the GrabCaster trigger in different ways, in this case I used the REST API invocation as below.

The trigger is executed and I moved one million records from an on premise SQL Server database into a SQL Server database table in the cloud.

After tested the REST API the developer implemented a simple REST call in a Web UI button.

Below the scenario implemented.

    

There are some important aspects which I appreciate in this approach.

  1. Thank to Microsoft Azure Fabric the solution is always-on, scalable and distributed.
  2. The simplicity into the approach and the configuration.
  3. The extensibility using the REST API call to invocate the trigger.
  4. The using of Redis Cache and Blob which have a very low pricing consume in Microsoft Azure.
Author: Nino Crudele

Nino has a deep knowledge and experience delivering world-class integration solutions using all Microsoft Azure stacks, Microsoft BizTalk Server and he has delivered world class Integration solutions using and integrating many different technologies as AS2, EDI, RosettaNet, HL7, RFID, SWIFT. View all posts by Nino Crudele

Big Data integration in high scale scenario using Azure Service Fabric and GrabCaster

Big Data integration in high scale scenario using Azure Service Fabric and GrabCaster

Big Data is one of the most important topic in the last year, the world of integration has changed since the companies start providing the possibility to store our information in the Cloud.
Send the data from our on premise to a database in the cloud can be achieved in different ways, I’m very keen and focused to implement feature in GrabCaster able to solve integration problems in easy way.

GrabCaster is an open source framework and it can be used in every integration project within any environment type, no matter which technologies, transport protocols or data formats are used.

The framework is enterprise ready and it implements all the Enterprise Integration Patterns (EIPs) and offer a consistent model and messaging architecture to integrate several technologies., the internal engine offers all the required features to realize a consistent solution.

The framework can be hosted in Azure Service Fabric which enables us to build and manage scalable and reliable points running at very high density on a shared pool of machines.
Azure Service Fabric servers as the foundation for building next generation cloud and on premise applications and services of unprecedented scale and reliability, for more information about Microsoft Azure Service Fabric check here

Using GrabCaster and the Microsoft Azure stacks I’m able to execute a SQL Server Bulk Insert operation across on premise and the cloud very easily, below how it works.

I’m not going in detail, you can find all the samples and templates in the GrabCaster site.

Download GrabCaster Framework and configure it to use the Azure Redis Cache, this is one of the last messaging provider implemented, what I love about this Azure stack is the pricing and the different options offered by this framework.

Using the SQL Bulk Trigger and Event I’m able to send a large amount of data across the cloud, the engine compacts the stream and it uses the Blobs to move the large amount of records.

Last the I did was moving one million record from a table in an on premise SQL database and another in the cloud.

I installed GrabCaster in Azure Fabric as below.

I configured the SQL bulk trigger in the on premise environment using the json configuration file.

I configured the SQL Bulk event and I sent the configuration to the GrabCaster point in the Azure Fabric using the internal synchronization channel.

I can activate the GrabCaster trigger in different ways, in this case I used the REST API invocation as below.

The trigger is executed and I moved one million records from an on premise SQL Server database into a SQL Server database table in the cloud.

After tested the REST API the developer implemented a simple REST call in a Web UI button.

Below the scenario implemented.

    

There are some important aspects which I appreciate in this approach.

  1. Thank to Microsoft Azure Fabric the solution is always-on, scalable and distributed.
  2. The simplicity into the approach and the configuration.
  3. The extensibility using the REST API call to invocate the trigger.
  4. The using of Redis Cache and Blob which have a very low pricing consume in Microsoft Azure.
Author: Nino Crudele

Nino has a deep knowledge and experience delivering world-class integration solutions using all Microsoft Azure stacks, Microsoft BizTalk Server and he has delivered world class Integration solutions using and integrating many different technologies as AS2, EDI, RosettaNet, HL7, RFID, SWIFT. View all posts by Nino Crudele

Microsoft Integration Stencils Pack v2.1 for Visio 2016/2013 is now available

Microsoft Integration Stencils Pack v2.1 for Visio 2016/2013 is now available

Because yesterday Portugal was European Champion I decide today to give a gift to the community by making an earlier release of my Microsoft Integration Stencils Pack. After the major release in June 16, I started to realize that with so many stencils it was becoming a little hard to find, or rather, I was […]
Blog Post by: Sandro Pereira

Portugal is the new European champion and how that will affect Integrate2017

Portugal is the new European champion and how that will affect Integrate2017

A goal from Eder, at the 109 minutes of the match, it was worth the European champion title to Portugal, an unprecedented title in Portuguese football, succeeding our neighbor Spain. PORTUGAL are the Euro 2016 champions. No matter what anyone says or does, that will not change. (photo from: UEFA) No matter now saying that […]
Blog Post by: Sandro Pereira

#robustintegration Reviewer – Riaan Gouws, CTO at Synegrate

#robustintegration Reviewer – Riaan Gouws, CTO at Synegrate

Riaan Gouws, @RiaanGouws74

Riaan is Chief Technical Officer at Synegrate, a Southern California based software consulting company specializing in on premise, hybrid and cloud based integration. Synegrate is also a Microsoft Gold Partner in Application Integration with specialized skills in Modern Integration (Azure Micro Services and BizTalk Server), based out of Costa Mesa, California. The company has been operating since 2004 as Inobits Inc. and was rebranded as Synegrate in 2014.

Riaan is a BizTalk Server expert, having worked with the platform since its initial release 15 years ago. He furthermore is a keen supporter of Microsoft’s cloud first strategy and has aligned Synegrate with this strategy. He is also a member of the elite Microsoft Virtual Technology Specialist (VTSP) team, a small group of selected industry experts working as an extension to the Microsoft Technology Specialist teams. He has a strong relationship with Microsoft as a valued partner in implementing enterprise solutions on top of Microsoft Azure.

I got opportunity to work with Riaan during my last project where he was the architect for the BizTalk solution – EDI and EAI, for the enterprise customer. He is amazing person and a great consultant to work with.

Now a days he is mostly involved with customers where they are implementing modern integration strategies using Microsoft Azure. He was the first person I reached out for the reviewer of our book – Robust Cloud Integration with Azure.

Now that he is in the reviewer team of our book, I asked him to share his thoughts on the following couple of questions about the book.

1. What do you think about the outline of the book?

I reviewed the book outline and I think it is well thought out, accurately targeted towards integration professionals, and reads very easily. I especially love Chapter 3 – Getting started with API Apps; I think it is comparable to a theme park thrill ride. It takes the user from introduction to API apps, to advanced topics in a quick 40 pages or so. Love it!

Then the approach you guys are taking with Logic Apps is also great. You start the reader off slow, but then quickly accelerate through the chapters, showing the users basic to advanced concepts on orchestrating data. I also appreciate the fact that you are on cusp of new offerings in Azure, such as Integration Account and Integration Pack.

2. What is your expectation from the book or how do you think this book would be valuable for its reader’s time and money?

I expect the book to take an integration developer/architect, with predominantly an on-premise integration skill and mindset, to being knowledgeable and comfortable building integration solutions in Azure. Synegrate is currently involved with customers where we are implementing modern integration strategies using Microsoft Azure. I see readers gaining a lot of knowledge from the book; knowledge that we have gained from practical experience in the field. I think of this book as something that will allow readers to fast-track their entry into getting modern integration solutions built in Azure.

I thanks Riaan to be part of our team as reviewer and also sharing his thoughts about the book.

I wish him and Synegrate very best for their future endeavors.

Advertisements