Building sentiment analysis solution with Logic Apps

by Steef-Jan Wiggers | Jul 7, 2017 | BizTalk Community Blogs via Syndication

Integrate 2017, a well-organized Microsoft Integration focused event, took place from 26 to 28 of June at Kings Place in London. It attracted 380 plus attendees from 50 different countries and had 28 speakers from around the globe including the Microsoft Product Group. I did a session around Logic Apps from the consumer, end user, and business perspective and used sentiment analysis as for my demo.

Context

To provide you some context. Logic App service was the most prominent technology during the three-day event. This Azure Service became general available a year ago and is starting to build momentum as premier cloud integration capability. Most of all, the service fits rather well in the complete Azure Platform with its connectors to a wide variety of other Azure services and in addition, it connects with SaaS solutions such as Twitter, Zendesk, Salesforce, ServiceNow, PagerDuty, and Slack.

During Integrate 2017 I talked about empowering business with Logic Apps. And my goal was to show the audience the value of Logic Apps for the business. The service is a true iPaaS service according to the definition Wikipedia provides online. And it is a part of Azure, which is multi-tenant, has a subscription model or in the case of Logic Apps it’s consumption based (micro-billing), provides pre-built ready available connectors, deployment/manage/monitoring through the platform.

iPaaS

If you look at how for instance Gartner describes iPaaS then again Logic Apps are a true cloud-native integration platform. Consumers of Logic Apps in Azure can implement data, application, API and process integration projects spanning cloud-resident and on-premises endpoints. I will quote the Gartner report here:

“This is achieved by developing, deploying, executing, managing and monitoring “integration flows” (aka “integration interfaces”) — that is, integration applications bridging between multiple endpoints so that they can work together.”

And the iPaaS capabilities typically include according to Gartner:

• Communication protocol connectors (FTP, HTTP, AMQP, MQTT, Kafka, AS1/2/3/4, etc.)
• Application connectors/adapters for SaaS and on-premises packaged applications
• Several data formats (XML, JSON, ASN.1, etc.)
• Data standards (EDIFACT, HL7, SWIFT, etc.)
• Mapping and transformation of data
• Quality of data
• Routing and Orchestration
• Integration flow development and lifecycle management tools
• Integration flow operational monitoring and management
• Full lifecycle API management

Looking at the above capabilities than Logic Apps in combination with Integration Account and API Management provide those capabilities.

Gartner Quadrant

Logic Apps are positioned in Gartner Quadrant in the Visionaries box, which means that the vendor of the service is able to execute lower than the leaders (in the Quadrant vendors like Dell Boomi and Informatica), have a smaller install base, certain immaturity, timid marketing, reactive sales operation and lack of strategic commit to the market.

My take on that is that Logic Apps is relatively new in the iPaaS market.

• A year ago it became general available. And it is maturing at a fast pace with new feature releases every two weeks with an expanding set of connectors.
• Sales representation from Microsoft at Integrate 2017.
• And finally, the commitment is strong with the Pro Integration Product Group presence at various conferences throughout 2017. This year they have or will attend Ignite, Build, Integrate2017 Europe, Inspire (former WPC), Integrate 2017 US, Integration Bootcamp, Global Integration Bootcamp, Global Azure Bootcamp, and smaller User Group meetings worldwide.

Hence I struggle a bit with the classification of the current state of Logic Apps. I strongly feel the service is close to the border of visionary and leader. It has promised to become a true iPaaS leader.

Benefits

Business can reap the benefits from this service as the attention is towards solving the problem(s) it is facing. Logic Apps is a part of a large Platform. And it can deliver solutions fast as there’s no need for procuring servers, or other infrastructure related capabilities. This accounts for the business that has transformed their business to the cloud and requires cloud-native solutions. That’s what fit for purpose with Logic Apps. And the costs are less and time to market of your solutions is fast.

Use Cases

The connectors provided by Logic App can help you build solutions for various enterprise scenarios. For instance, you leverage cognitive services to identify a person to subsequently grant him access to resources, start an onboarding process, or provide access to a facility. An example of leveraging Cognitive Services is to perform text analysis on tweets, which I will explain in further detail later in this post.

The text analysis can be useful to detect sentiment in a tweet. Particularly on a #hashtag, for instance, a person like Trump, product or service. I mention President Trump here as the current US President uses this social media service quite extensively. And the tweets he produces are evaluated intensively for stock trading.

Dynamics 365

Other thinkable use cases evolve around the Dynamics 365 CRM Online connector. This connector provides connectivity to Dynamics CRM that provides various features like customer service automation, marketing campaigns, and social engagements.

Dynamics 365 has several capabilities or flavors; one is Dynamics for Field Service, which provides a complete Field Service management solution, including service locations, customer assets, preventative maintenance, work order management, resource management, product inventory, scheduling and dispatch, mobility, collaboration, customer billing, and analytics. Therefore, during integrate I talked about leveraging this solution in combination with IoT devices. The picture below shows the data flow from device to the Dynamic Field Service features.

Data from a device can be consumed by IoT Hub service in Azure and pushed to the service bus queue, which can be read by Logic App. The Logi App forwards the data into Dynamics Field Service through the CRM connector. In conclusion, a Logic App or number of them can be part of an end-to-end solution for various field services.

The previous paragraph discussed one of the many use cases possible including Logic Apps. Moreover, there are many other scenarios thinkable since Logic Apps are a part of a bigger platform, which means you leverage them with other Azure Services or create flows to move data around. With sentiment analysis, you can detect sentiment within a text using one of the Cognitive Services API’s. The way sentiment analysis API functions are that it returns a numeric score between 0 and 1 on a given text. Scores close to 1 indicate positive sentiment and scores close to 0 indicate negative sentiment. A score of 0.5 is neutral. With Logic Apps, you can receive tweets within a certain interval (occurrence) based on filter i.e. hashtag and feed the body into Detect Sentiment action.

Sentiment Analysis Solution

To build a solution leveraging the capabilities Cognitive Services deliver with a Logic App, Azure Storage Account, Azure Function and Power BI you need to set up these services up.

Cognitive Service

The setup of the first is basically provisioning of a Cognitive Service instance i.e. API. In the Azure Portal, you find the Cognitive Service in the marketplace. Subsequently, you click on the service you specify a name, choose a subscription, and subsequently which API you like to use.

To detect sentiment analysis in a text you need to choose Text Analytics API, which as the time of writing is still in preview. The Text Analytics API is only available in region West US, and pricing of service varies depending on the tier you require. Below you can see the different pricing options.

As you can see in the picture above the Cognitive Service provides four features:

• Sentiment Analysis
• Key Phrase Extraction
• Topic Detection
• Language Detection

Once you have chosen the required tier you can create this service.

Power BI

The next service is Power BI, which is a part of the Office365 offering and can be found here: https://powerbi.microsoft.com/. You can sign in and start building datasets, dashboards, and reports. For a solution to visualize sentiment you can create a streaming data set. Go to the powerbi.com and “Streaming datasets”, create a dataset of type API, click next and name the dataset and add fields to the streaming data set like shown below.

The Solution

In a solution, I build I created four text fields and one number field. The historic data analysis was enabled to build a collection of the data to be used for a report.

Now both Cognitive Service and Power BI have been setup and next step is to create a storage account in Azure. This account will archive tweets in a blob storage container tweets. Provisioning a storage account is easy and straightforward process. In the marketplace find storage account, select it, specify name, deployment model, purpose (choose blob storage), replication, access tier (cold), secure transfer, subscription, resource group, and location.

The final service required for the solution is a function. The Function in our solution will be provided with the input from the Cognitive Service API response (Score). Azure Functions provide a serverless coding capability using a Browser and the piece’s code you write can run in Azure i.e. within a Function App.

For our solution, we add a GenericWebHook-CSharp. We will rename the function to AnalyseSentimentScore. And in the Develop tab, we see some generic default code, which we will change to the code below.

Architecture

The solution architecture I build looks like the diagram below and resembles a process manager pattern.

This pattern implies that a trigger message is sent to a process manager (Logic App). The process manager is a central processing unit and determines steps based on intermediate results. A tweet is the trigger message that starts a flow in a Logic App. The body is sent to Cognitive Service (Proc A) and the score is sent to a Function (Proc B), which will evaluate the score. The Tweet is stored in blob storage and a few fields are sent to Power BI to fill the dataset. A diagram of a process manager is depicted below.

Implementation

The implementation of the solution is slightly different than from the pattern as after the second intermediate step the tweet data is sent to Azure Blob Storage and Power BI dataset.

The Logic App is implemented with a Twitter trigger, authorized to use my twitter account, with the search text #integrate2017 and interval (frequency) of 5 minutes i.e. every 5 minutes tweets with #integrate2017 will be picked up. Subsequently, this trigger is followed by several actions.

The picture above shows the flow of the Logic App. First, a Twitter triggers then a compose action to create an element part containing the username of the tweet. Subsequently the detect sentiment and the detect key phrases actions. Then the second composes to create a JSON array of the key phrases. And after the second compose the score of the detected sentiment is send to the function, which will return a string (text) of the evaluated score (see also the function). Several tokenized elements are sent to blob storage (see picture below).

And the final step of this solution (Logic App definition) is sending some of the tokenized elements to a dataset row in Power BI dataset.

Now we have walked through the complete Logic App definition and the key actions of the solution.

Integrate 2017 Report

For integrate 2017 I ran the Logic App between 17th of June until the 1st of July. And the event took place between 26th and 28th of June in London. Every 5 minutes the Logic App collected tweets from Twitter with hashtag integrate2017. Over this period of 15 days, 3500 tweets have been aggregated around this event. It started slowly with around 50 tweets until the event started on the 24th with a burst of tweets. Below you can see a report created in Power BI with some visualization of sentiment measured in the tweets.

Around 2/3 of all the tweets, the sentiment was excellent/good, which can be viewed as positive. 1/3 of the tweets were evaluated as moderate. The Cognitive Service Text Analysis capability was unable to determine negative or positive. And finally, a very small percentage was negative (bad). Hence you can conclude that the event was a great event given the sentiment score.

The benefits of building a solution like described above are that with a relative simple Logic App sentiment can be analyzed leveraging several abilities provided by the Cognitive Service. Probably when a business likes to measure sentiment through Social Media channel it can use Logic Apps. Therefore, Logic Apps provide a quick solution in this manner to provide quick insights with low costs. There are no servers necessary and a pro-integration professional can build this type of solutions within a few hours depending on the complexity. Hence it provides quick time to market.

The costs

The interesting part of this solutions is cost. The breakdown of costs for this solution is:

– Logic App (Consumption)
– Function (Consumption)
– Cognitive Service (Tier)
– Storage Account (Volume)
– Power BI (Enterprise Plan)

The Logic App and Function are consumption based and measured on the execution of an action or function. And in general, it can sometimes be hard to predict the workload these services need to process. Hence you need to be aware of this. A good reference with regards to costs with Logic Apps is a post by Rene Brauwers, Tips & Tricks: Cost savings using Logic Apps.

For the Logic App in this solution, 3500 tweets were processed, and the Logic App consists of 8 actions (including the trigger). Hence 28K action calls costs based on the pricing (First 250K actions = €0.000675 / action) approximately 19 euro. And less than a euro for the executions of the Function.

Next, the costs for the Cognitive Service depends on the tier. The free tier could be an option, however, if the workload is too high then you run into rate limiting issues. The S1 Standard can be sufficient and costs 150 Euro a month. Yet you can turn it off after your campaign of measuring sentiment, which could be a few days. In this solution, the costs are 75 euro. Storage of less than 4 Mb of tweets is neglectable. This leaves the costs for Power BI. For the solution, I build I used the pro version, which is around 10 Euro per month. Thus, in total, a sentiment analysis solution costs around a 100 euro.

Conclusion

Depending what the requirements are and perceived value is, Logic Apps combined with other Azure Services and Office365 (Power BI) can be a good fit for purpose for low costs, agility and time to market. Logic Apps are becoming a leader in the iPaaS. On a short term, it will be able to cross the border from visionary to leaders in the Gartner Magic Quadrant. The Product Group is cranking out enhancements on the service and new connectors every two weeks. And they have kept this pace since the General Availability of the service a year ago. Nevertheless, the competition is strong however I am confident Logic Apps will be amongst the leaders.

Author: Steef-Jan Wiggers

Steef-Jan Wiggers has over 15 years’ experience as a technical lead developer, application architect and consultant, specializing in custom applications, enterprise application integration (BizTalk), Web services and Windows Azure. Steef-Jan is very active in the BizTalk community as a blogger, Wiki author/editor, forum moderator, writer and public speaker in the Netherlands and Europe. For these efforts, Microsoft has recognized him a Microsoft MVP for the past 5 years. View all posts by Steef-Jan Wiggers