by Sowmiya Subramanian | Oct 5, 2017 | BizTalk Community Blogs via Syndication
This blog helps to perform various operations with BAM views using the command prompt such as:
- deploy BAM Views
- remove BAM definitions
- authorize user accounts for BAM View
First, I have created a BAM view named as “BAMOperation” with the Activities Order and Status with items as shown below.
The BAM management utility deployment command line tool, bm.exe., allows to apply, modify, and remove BAM definitions. You can use, amongst others, below commands:
- deploy-all – Deploys a BAM definition
- remove-all – Removes a BAM definition
- update-all – Updates a BAM definition
- add-account – add accounts on BAM definition
Deploy a BAM View and Activities
The BAM Definition file (BAMOperation.xml) becomes deployed using the BAM Command line utility.
Deploying of BAM definition file will create corresponding database tables, views, etc., which will organize the data in a relational view. Follow the below steps to deploy a BAM View:
- Open Command Prompt as administrator
- Navigate to BizTalk Server Installation path/ Tracking folder and use the below command:
bm.exe deploy-all –DefinitionFile:” <<FileLocation>> BAMOperation.xml”
which will deploy a “BAMOperation” view in BAM Portal as shown below.
Remove a BAM Definition
Administrators can use the remove-view / remove-activity command to remove a view and Activity from the BAM Primary Import database. Follow the below steps to remove an existing activity/view:
- Open Command Prompt as administrator.
- Navigate to BizTalk Server Installation path/ Tracking folder and use the below command.
To Remove a BAM Definition File:
bm.exe deploy-all –DefinitionFile:” <<FileLocation>> BAMOperation.xml”
Removing a BAM definition file will remove the related tables, views, etc. which were deployed in the database and server.
Remove a BAM Activity
To remove a BAM Activity, the user needs to remove the BAM View first and then the BAM Activity they wanted to remove. Execute the below-shown command to remove the BAM View first and then the BAM Activity.
To remove a BAM View:
bm.exe remove-view -name: {view name}
To remove a BAM Activity
bm.exe remove-activity -name: {activity name}
The above command which will remove the specified activity alone form the database.
Update a BAM Activity
The User is also able to update an existing View/Activity. To update an Activity/ View, the user should remove the activity/view first by using the above-mentioned commands and then follow the below steps to update any changes in the existing View /Activity.
For example, I have removed the above “Status” activity and I have updated the activity as shown below.
We then use the below command to update the changes to the existing view:
bm.exe update-all –DefinitionFile:” <<FileLocation>> BAMOperation.xml”
The above command will add a table, view in the database and it will update the previous activity in the BAM Portal as shown below.
Add an Account to BAM Definition
Administrators use the add-account command to associate users with BAM views and protect the BAM Excel Spreadsheet views from unauthorized access. When users save BAM views, the views reference a SQL connection string that is hidden within the workbook. The workbook is protected, but you must also ensure that the document is protected.
When you associate users with BAM views, you restrict access to the views to only the users or groups to whom you grant access by using the below command.
bm.exe add-account -AccountName: DomainName UserName -View:(viewname)
by Sandro Pereira | Oct 5, 2017 | BizTalk Community Blogs via Syndication
It is not the first time, neither will be the last that I encountered similar problems like this one or the same problem with earlier versions, we call it DLL hell (or nightmare) but I think that all BizTalk Administrator are familiar with it and vaccinated for the problem. Some months ago, while trying to communicate with an Oracle database within Visual Studio in a brand-new BizTalk Server 2016 Developer environment to generate the proper Schemas, we faced with the following Oracle.DataAccess problem:
Error saving properties.
(System.ArgumentException) Invalid binding.
(System.IO.FileNotFoundException) Could not load file or assembly ‘Oracle.DataAccess, Version=4.121.1.0, Culture=neutral, PublicKeyToken=89b483f429c47342’ or one of its dependencies. The system cannot find the file specified.
Cause
When installing the Oracle WCF Adapter for BizTalk Server 2016 there is a design-time requirement to use Oracle.DataAccess Version 4.121.1.0.
BizTalk Server 2016 requires that specific Oracle.DataAccess version, however, us we were able to very on the GAC, the DLL that existed in our environment had a different version.
Trying to find the correct ODP.NET 11.2.0.1.2 version under Oracle website can be a challenge
Note: depending on the BizTalk Server version that you are using, this the required Oracle.DataAccess version may change.
Solution
Trying to find the correct ODP.NET 11.2.0.1.2 version under Oracle website can be a challenge, so one of the easier and fast ways to solve this problem is using Assembly Binding Redirection in the machine configuration file (Machine.config):
- 32-bit: c:WindowsMicrosoft.NETFramework[version]configmachine.config
- 64-bit: c:WindowsMicrosoft.NETFramework64[version]configmachine.config
Note: You should apply this in both 32 and 64-bit machine configuration files.
By using the <assemblyBinding> Element for <runtime> that will contain all the information about assembly version redirection and the locations of assemblies.
In this case, you should apply the following configurations:
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
<assemblyIdentity name="Oracle.DataAccess"
publicKeyToken="89b483f429c47342" />
<bindingRedirect oldVersion="4.121.1.0" newVersion="x.xxx.x.x" />
</dependentAssembly>
</assemblyBinding>
</runtime>
By doing this when BizTalk Server will look to the Oracle.DataAccess version which not exists in your environment, it will be redirected to the existing DLL version.
For example, in our case we used:
<!--<runtime />-->
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
<assemblyIdentity name="Oracle.DataAccess" publicKeyToken="89b483f429c47342" />
<bindingRedirect oldVersion="4.121.1.0" newVersion="4.121.2.0" />
</dependentAssembly>
</assemblyBinding>
</runtime>
Just to be on the safe side, you should add this configuration in both 32 and 64-bit in .NET Framework 2.0 and 4.0 machine configuration files.
Author: Sandro Pereira
Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community. View all posts by Sandro Pereira
by Gautam | Oct 1, 2017 | BizTalk Community Blogs via Syndication
Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform?
Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.
If you want to receive these updates weekly, then don’t forget to Subscribe!
On-Premise Integration:
Cloud and Hybrid Integration:
Microsoft Ignite 2017
Feedback
Hope this would be helpful. Please feel free to let me know your feedback on the Integration weekly series.
by michaelstephensonuk | Oct 1, 2017 | BizTalk Community Blogs via Syndication
Recently I had a chat with a few people at a company about integration solutions and a question came up which I remember not having been asked in a while. “Will the business users be able to modify the solution after its live”. Back in the day you used to get asked this quite often but I don’t remember being asked this one for a while. I do know that the technology landscape has changed a lot since I last got asked this and after a very interesting discussion I thought id share my thinking of this space.
Why would the user want to modify the solution
There are a few different things that come up as the reason for this desire. It’s a good idea in this discussion with a customer to find out their drivers, they may not always be the same. Some of the common ones include:
- IT guys are expensive so we don’t want to constantly need to pay a premium for changes
- IT take too long to make the changes
- A change may seem very simple yet IT seem to turn it into a project
- We need to react to customer’s needs and changes quickly
What are the ways in which the user may modify the solution
Different solutions will have different things that can be changed or tuned but if we think of common solutions we see these are some of the areas we can change things:
- Config settings in the solution which are used to drive decision logic. EG a threshold for an order which is considered large. Depending upon where the config settings are stored they may be able to be modified by the business user
- Business Rules may be editable by a business user
- Flow logic, the solution may have a workflow which could be edited by the user
- On boarding new customers/partners is a part of the integration solution the business user may wish to take control of
- Data mapping and transformation rules
Im sure there are probably plenty of other areas but above are just a few off the top of my head
Should the user be allowed to modify the solution
Assuming the reason that the business users want to be able to take control of modifying the solution or parts of it is valid then a decision over if this should be allowed or not is likely to come down to the desire of the product owner or sponsor and the governance areas of the solution.
What we need to consider is that if we elevate the business users permissions to control parts of the integration solution, does this mean we trade off any other parts of the solution. In this area we need to consider the areas of the solution that the business user will often have low regard for such as performance, testing, application lifecycle management, security and risk. If for example we go and grant permissions for the business user to modify the work flow of an integration process then the changes are they will reactively go an do this in response to a change required by their customer. They may not think to communicate the planned change, they may not test it properly and may think their change has worked by they don’t understand that they have now broken 3 things down stream which have been affected by this change.
The one think you can guarantee is that the business user will very rarely have an understanding of dependencies and consequences within the system. I say this quite boldly because often many IT people do not have this understanding either. When it comes to making changes, even if you have a very good IT architecture, treating the change like pulling out a piece of a Jenga puzzle is a good approach to make. First you need to work out if this is a good piece to be messing around with or not. Maybe its pretty safe and you can let your business user get on with it and at other times you need to be very cautious and through.
Having a view on the changes the business would like to make overlayed with some kind of heat map of your architecture will tell you the safe Jenga pieces for the business to take control of and the areas of risk when you need to keep in IT.
When you then let the business make these changes themselves, you still need to implement a change management process for them to follow. You don’t want to be in a position where the business user made a change which wasn’t reflected back into the system source configuration so that next time there is a major release the change is regressed. Teams still need to work together.
Types of solution
Once you have identified areas and rules around the business user making changes in isolation, I guess you have laid out your rules for engagement as you begin democratizing integration in certain areas.
I would expect that you find solutions would fall into certain types which are more and less acceptable for the business to change.
Citizen Integrator
Citizen Integrator solutions are those ones which were either built by the business or built by IT but can be handed over to be looked after day to day by business users. These may be solutions for individuals or solutions for teams.
In this space you will find super users in the business such as Excel guru’s begin to thrive. They can build solutions for their team and themselves and really innovate.
One of the challenges from an IT perspective is that if the Citizen Integrator builds a solution that becomes too important or if it becomes relevant to be managed by regulatory rules which the business function may not be aware of but IT have dedicated functions to support.
How do you as an IT function stop bob the HR junior intern from building the mission critical staff holiday system replacement project without anyone in IT being aware?
Id expect the business user to be able to make the following changes in this type of solution:
Lightweight Low Risk Integration
Lightweight integration projects may be good candidates for business users to “modify”. I would consider light weight to mean not lots of load, not very complex and fairly well isolated. In this scenario you may choose to allow a business super user to some changes in certain areas but it is likely some areas may require a more advanced skillset.
Id expect the business user to be able to make the following changes in this type of solution:
Mission Critical & High Risk
In a mission critical integration solution I would expect that there would be a pretty thourgh change management process that would control changes to the system. In these cases the consequences of breaking something out-weigh the benefits of quick changes. I would expect that most changes would involve a degree of impact analysis followed by a controlled change process involving making the change, deploying it in a repeatable way, testing the change and making the change live.
The intention of the process overhead is to remove risk of things going wrong and sometimes removing the risk of politics in the organisation if something was broken that affected the core business.
I would expect in this case the attitude to the business super user making changes would be:
Those examples above are just some of the ones you might come across. It really depends on the organisation and its attitude to risk and change. An example you might find is a mission critical system which has certain parts of the system which are very safe for business users to modify. An example might be an architecture where Dynamics CRM is used to provide a lot of the settings and configuration for a customer and then integration processes use these settings as required. This gives the user a place where they can safely modify some parts of the system which are used by others. I think the key point here though is it comes back to a heat map of your architecture so you know the safe Jenga pieces and the unsafe ones.
Technology Comparison
Up until this point I have tried to think about things agnostic of technology, but if we also bring in this angle it gets even more interesting. Back a number of years ago when we had a small set of tools available there were only a few limited choices in this space, but now we have many more choices.
Flow & Power Apps
If we know we have requirements for the business user to take an active part in the solution and its maintenance then Flow and Power Apps give us an ecosystem which is built for these users. In a solution we can incorporate these tools to provide a safe(ish) area for the business user to do their bits. We can even give the business user control of the entire solution if its appropriate.
Flow and Power Apps should be a part of our integration architecture anyway and give us a big tick in the box of empowering the business user to be an active stakeholder in an integration solution.
Logic Apps
Logic Apps are a really interesting one, they have the features or a high power mission critical integration tool but the ability to sandbox logic apps in resource groups means it is possible for us to use some logic apps for those IT only use cases and have other Logic Apps where the business user could be granted access to Azure and the resource group to be able to manage and modify if it was appropriate.
BizTalk
BizTalk is one of the tools where there are not that many choices for the business user. It is unlikely we would want the business user to make changes to anything which is not then handed over to IT for deployment. That said in an agile environment a BizTalk developer and business subject matter expert working closely on a solution can be a very good way to work.
Rules, Cross Reference Data, port settings and Configuration settings are the most likely candidates for a desire to change but I think the risks of doing this without an ALM process would outweigh any benefits.
One point to note with BizTalk is that BizTalk 360 provides a number of features which can allow a business user to manage their integration solution. While BizTalk might be one of the less friendly tools to allow a business super user to make changes, BizTalk 360 can allow the person to manage their messages and process instances if they want. This can be done in a safe way.
SSIS & Data Factory
Azure Data Factory and SSIS are like BizTalk in that it would be difficult to get a business user to be able to do anything with them. They are a fairly closed environment and require a significant skill set to do anything with them. Id see these as IT only tools.
Service Bus
Service Bus is an interesting one, you could imagine a scenario where the business user might request that all new customer messages are now also sent to a new application they are buying. Conceptually its not too much of a leap to see an advanced user setting up a new subscription from the new customer topic to the new application worker queue. In the real world however I can imagine that most of these changes would require some additional work around the applications either sending or receiving messages so I think the service bus change would be unlikely to be done in isolation.
I think a business user may also struggle to understand message flows without an upfront visualization tool.
With this in mind I suspect Service Bus would be unlikely to be a good fit for requirements for business users to modify.
Event Grid
My first thought is that event grid would fall into the same space as service bus messaging, but maybe in the future as the solution matures the fact that the data is events rather than messages may mean that there are certain scenarios that the business may dynamically change their interest in. In wonder about scenarios like a university clearing time where perhaps a super user gets a really cool idea of a way to improve the likelihood of a new student signing up and would like to explore events for just 2 or 3 conditions for a couple of days. Self service subscription to events in this nature could be a really powerful way of experimentation within the business.
Summary
I think the answer to the initial question has changed a lot over the last few years, its great to be in the position where we have loads of technical options and rather than being limited in how we could include those requirements into a solution its now a case of making sure we don’t go too over the top and remember that there is still an important place for governance and best practices in addition to agility and flexibility.
Exciting times really
by Sandro Pereira | Sep 29, 2017 | BizTalk Community Blogs via Syndication
This is probably the quickest and smallest update that I made in my Microsoft Integration (Azure and much more) Stencils Pack: only 1 new stencil and I only do it for its importance, since it is definitely one of Microsoft’s fastest growing business these days, and the Ignite context: the new Azure logo.
This will be probably the first Visio pack containing this shape.
The Microsoft Integration (Azure and much more) Stencils Pack v2.6.1 is composed by 13 files:
- Microsoft Integration Stencils v2.6.1
- MIS Apps and Systems Logo Stencils v2.6.1
- MIS Azure Portal, Services and VSTS Stencils v2.6.1
- MIS Azure SDK and Tools Stencils v2.6.1
- MIS Azure Services Stencils v2.6.1
- MIS Deprecated Stencils v2.6.1
- MIS Developer v2.6.1
- MIS Devices Stencils v2.6.1
- MIS IoT Devices Stencils v2.6.1
- MIS Power BI v2.6.1
- MIS Servers and Hardware Stencils v2.6.1
- MIS Support Stencils v2.6.1
- MIS Users and Roles Stencils v2.6.1
That will help you visually represent Integration architectures (On-premise, Cloud or Hybrid scenarios) and Cloud solutions diagrams in Visio 2016/2013. It will provide symbols/icons to visually represent features, systems, processes and architectures that use BizTalk Server, API Management, Logic Apps, Microsoft Azure and related technologies.
- BizTalk Server
- Microsoft Azure
- · Azure App Service (API Apps, Web Apps, Mobile Apps and Logic Apps)
- API Management
- Event Hubs
- Service Bus
- Azure IoT and Docker
- SQL Server, DocumentDB, CosmosDB, MySQL, …
- Machine Learning, Stream Analytics, Data Factory, Data Pipelines
- and so on
- Microsoft Flow
- PowerApps
- Power BI
- Office365, SharePoint
- DevOpps: PowerShell, Containers
- And many more…
You can download Microsoft Integration (Azure and much more) Stencils Pack from:
Microsoft Integration Stencils Pack for Visio 2016/2013 (11,4 MB)
Microsoft | TechNet Galler
Author: Sandro Pereira
Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community. View all posts by Sandro Pereira
by Praveena Jayanarayanan | Sep 27, 2017 | BizTalk Community Blogs via Syndication
Biztalk360 comes with a lot of exciting features in every release. One of the important functionalities in BizTalk360 is the monitoring with the autocorrect options. BizTalk360 is the one-stop monitoring solution for BizTalk server. We can not only monitor the artefacts, but also the SQL jobs. Yes, the SQL jobs present in the SQL server can also be monitored. We can also set the autocorrect (enable/disable) functionality for the SQL jobs.
There may be separate servers for BizTalk databases and BizTalk360 database or even single server hosting all the databases. The jobs in all these servers can be monitored via BizTalk360. But then, can all the users have access to monitor and autocorrect the SQL jobs? In this blog, I am going to explain about the permissions required by the users for monitoring SQL jobs and setting autocorrect functionality which we learnt from one of the support tickets.
Customer’s case:
Our support team often get some interesting tickets which do not directly deal with the functionality and features of BizTalk360. Some tickets may be related to performance, access permissions, AD users etc. Each ticket experience is a new learning for our support engineers. Let’s see one such case of a customer related to the access permission for the databases in the SQL server.
The customer got the below exception when they tried to set up monitoring for SQL jobs.
The sp_help_job is the stored procedure used to list the SQL jobs running in the server. This job returns information about jobs that are used by SQL Server Agent to perform automated activities in SQL Server. There are SQL jobs that get installed and scheduled automatically to maintain the health of the BizTalk environment.
BizTalk360 allows to set the threshold for SQL jobs (Monitoring -> Manage Mapping ->SQL Server Instances ->SQL Jobs) to list out those SQL jobs and perform the automatic operation this “sp_help_job” job is being used.
The exception in the above screenshot comes because of a missing permission for the BizTalk360 service account while accessing the SQL server. We have our support article in place which describes about the permissions for the SQL jobs. The customer has given the permission according to this article. But they were facing the error again when trying to enable autocorrect feature for SQL jobs.
In this error message, it says “Only members of sysadmin role are allowed to update or delete jobs owned by a different login”
This means that only if the service account has the SYSADMIN permission, then it can enable/disable the sql jobs from BizTalk360. But some of the customers would not prefer to provide SYSADMIN permission for the service account due to some security policies. So, what happens in such case? Let’s go ahead and check the resolution given. Before that lets have a quick glance at SQl jobs and permissions.
The SQL jobs:
A job is a series of operations performed by SQL Server Agent sequentially. A job can run on one local server or on multiple remote servers. The jobs are used to define administrative tasks that can be run one or more times and monitored for success or failure. SQL server agent runs these scheduled jobs. A job can be edited only by its owner or by the members of the sysadmin role.
The SQL job permissions:
SQL server has the following msdb database fixed roles through which the SQL server can be accessed and controlled. The roles from least to most privileged are:
- SQLAgentUserRole
- SQLAgentReaderRole
- SQLAgentOperatorRole
Can we have a brief look at each one of them?
SQLAgentUserRole:
This is the least privileged role. It has permissions on only operators, local jobs, and job schedules. Members of SQLAgentUserRole have permissions on only local jobs and job schedules that they own. They cannot use multi-server jobs (master and target server jobs), and they cannot change job ownership to gain access to jobs that they do not already own.
SQLAgentReaderRole:
This role includes all the SQLAgentUserRole permissions as well as permissions to view the list of available multi-server jobs, their properties, and their history. Members of this role can also view the list of all available jobs and job schedules and their properties, not just those jobs and job schedules that they own. SQLAgentReaderRole members cannot change job ownership to gain access to jobs that they do not already own.
SQLAgentOperatorRole:
This is the most privileged role which includes all the permissions of the above-mentioned roles. They have additional permissions on local jobs and schedules. They can execute, stop, or start all local jobs, and they can delete the job history for any local job on the server. They can also enable or disable all local jobs and schedules on the server. SQLAgentOperatorRole members cannot change job ownership to gain access to jobs that they do not already own.
The below table summarizes some of the properties for all these roles.
| Database Role |
Action – Create/modify/delete
Action – Enable/Disable
|
|
Local Jobs |
Multiserver jobs |
Job schedules |
| SQLAgentUserRole |
Yes
Yes
(Owned jobs)
|
No
No
|
Yes
Yes
(Owned schedules)
|
| SQLAgentReaderRole |
Yes
Yes
(Owned jobs)
|
No
No
|
Yes
Yes
(Owned schedules)
|
| SQLAgentOperatorRole |
Yes
Yes
|
No
No
|
Yes (Owned schedules)
Yes
|
Of all the above-mentioned SQL database roles, the SYSADMIN is the highest privileged role which has the administrator rights on the SQL server.
The resolution provided:
As mentioned earlier, the BizTalk360 service account would require the SYSADMIN permission to monitor and autocorrect the SQL jobs. But in some customer scenarios, they would not prefer to provide the SYSADMIN permissions. In that case, we need to see what is the minimum level of permission that we can provide to the service account for monitoring the SQL jobs.
Our support team did an extensive testing to check for various scenarios and permissions for the service account. The outcome of the testing is given below:
As the table summarizes, when the BizTalk360 service account is given the permissions as SQLAgentUserRole or SQLAgentReaderRole, it can only view the SQL jobs and cannot perform any operations on them. But when the SQLAgentOperatorRole is given for the service account, the auto correct functionality will work for the SQL jobs. The SYSADMIN permission is not required for this. This role is the highest privileged role next to the SYSADMIN.
Conclusion:
Hence, for setting the autocorrect functionality (enable/disable) the SQL jobs, the BizTalk360 service account needs to be given the SQLAgentOperatorRole permission to the system database, if SYSADMIN permission is not preferred to be given.
PS: BizTalk360 will not do any operation by itself until monitoring has configured for any of the available SQL Jobs and enable the Auto-correction ability. In case, you don’t wish to monitor the SQL jobs you can avoid the permissions shown in the above image.
If you have any questions, contact us at [email protected]. Also, feel free to leave your feedback in our forum.
Author: Praveena Jayanarayanan
I am working as Senior Support Engineer at BizTalk360. I always believe in team work leading to success because “We all cannot do everything or solve every issue. ‘It’s impossible’. However, if we each simply do our part, make our own contribution, regardless of how small we may think it is…. together it adds up and great things get accomplished.” View all posts by Praveena Jayanarayanan
by Sriram Hariharan | Sep 27, 2017 | BizTalk Community Blogs via Syndication
This episode of Azure Logic Apps Monthly Update comes to us directly from #MSIgnite. It is one of those episodes with a special guest and this episode featured Sarah Fender from the Azure Security Center team. The Pro Integration team are at #MSIgnite that’s happening between September 25-29, 2017 at Orlando, FL. I’ll try to give you a very crisp recap of the proceedings during the event and the important announcements from the #MSIgnite event.
Azure Security Center
Sarah started off talking about the Azure Security Center feature. Security Center provides unified security management and threat protection for Azure workloads, workloads running on-premises and on other cloud platforms. It basically assesses the security of the cloud and on-premise workloads and offers out of the box insights. In addition, Security Center offers some built in security controls such as Just in Time VM access that will help to lock down access to virtual machines, and Adaptive Access Controls that help to lock down on machines to prevent any malware execution. Security Center also monitors the hybrid cloud using advanced concepts like Machine Learning and provides rich graphical data to administrators.
Security Center keeps a look into all the different incidents in the environment such as SQL Injection, security incidents, suspicious processes and so on and provides insights which will be very helpful for IT teams to keep a track of the issues in the environment.
At #MSIgnite, the Azure Security Center team introduced the new experience of Investigation Dashboard. With this feature, organizations can easily respond to the incident and understand the intricate details about the security incident. The investigation path defines the attack path and the graphical view displays the detailed information such as severity of the attack, attack detected by information and so on. The investigation dashboard also lists the entities and now supports the Playbooks that are nothing but Logic Apps being triggered from Security Center when a certain alert is fired.
You can run a Playbook from the Security Center through the integration with Azure Logic Apps. Users can pre-define a Logic App that will actually take a corrective action when there is an attack you can allow the investigation dashboard to automatically execute that particular Logic App (through Playbook) to execute the corrective action. For e.g., when a vulnerability attack is detected with a very high severity, post a message on the slack channel for the users to get notified.
After all these updates from Sarah, it was time for the Logic Apps trio comprising of Jeff Hollan, Kevin Lam and Jon Fancey to provide the latest updates on Logic Apps. Kevin Lam started off by giving the latest updates-
What’s New in Azure Logic Apps?
- Custom Connectors – Enables the option to extend your endpoints and register them as connectors in Logic Apps.
- Large Message Support – This functionality is now available in the designer. Using this functionality, you can move large files up to 1 GB (between) for specific connectors (blob, FTP).
- Variables append to array – append capability to aggregate data within loops in the designer. Kevin Lam gave a pro tip here for all users –
Remember to turn on sequential for for-each to achieve this scenario.
- Nested foreach and do-until – is now available in the designer.
- Enable high throughput scenarios – You can configure the number of scale units within the code view to enable the high throughput scenarios. Say, you can take one Logic App definition that runs in a scale unit and span it across 16/32/64 scale units to get increased throughput. This is called ludicrous mode (as Kevin had it on the PPT).
- Maximum retries count (Custom Retry Policy) has been increased from 4 to 10.
- Now you can export (Publish) Logic Apps to PowerApps and Flow
- Emit correlation tracking id from the trigger to OMS – This gives full traceability across the process that’s happening across the Logic App.
- Expression intellisense – This is now available in the designer. When you are typing an expression, you will see the same intelligent view that you see when you are typing in Visual studio.
- Schedule based batching – In addition to batching based on message count, you can batch messages based on the schedule.
New Connectors
- Azure Security Center Trigger
- Log Analytics Data Collector – add information to Log Analytics from Log Analytics
- ServiceNow – create tickets, read & write into ServiceNow
- DateTime Actions
- Azure Event Grid Publish
- Adobe Sign – This was a big announcement from Microsoft at #MSIgnite – collaboration with Adobe
- O365 Groups
- Skype for Business
- LinkedIn
- Apache Impala
- FlowForma
- Bizzy
What’s in Progress?
- Concurrency Control (code-view live) – Say, your Logic App is executing in a faster way than you want it to actually work. In this case, you can make Logic Apps to slow down (restrict the number of Logic Apps running in parallel). This is possible today in the code-view where you can define say, only 10 Logic Apps can execute at a particular time in parallel. Therefore, when 10 Logic Apps are executing in parallel, the Logic Apps logic will stop polling until one of the 10 Logic Apps finish execution and then start polling for data.
- SOAP – Native SOAP support to consume cloud and on-premise SOAP services. This is one of the most requested features on UserVoice.
- Expression Tracing – You can actually get to see the intermediate values for complex expressions
- Foreach failure navigation – If there are lots of iterations in the foreach loop and few of them failed; instead of having to look for which one actually failed, you can navigate to the next failed action inside a for each loop easily to see what happened.
- Functions + Swagger – You can automatically render the Azure functions annotated with Swagger. This functionality will be going live by end of August.
- HTTP OAuth with Certificates
- Complex Conditions within the designer
- Bulk resubmit in OMS
- Batch configuration in Integration Account
- Connectors
- Workday
- Marketo
- Compute
- Containers
Watch the recording of this session here
[embedded content]
Community Events Logic Apps team are a part of
- INTEGRATE 2017 USA – October 25 – 27, 2017 at Redmond. Register for the event today. Scott Guthrie, Executive Vice President at Microsoft will be delivering the keynote speech. You can also avail Day Passes for the event (available for Wednesday and Thursday).
- ServerlessConf – 2 days of sessions on Serverless with Hackathon during October 2017
- Workday Rising – October 9 – 12 at Chicago
- CONNECT 2017 on October 9, 2017 at DeFabrique, Utrecht
Feedback
If you are working on Logic Apps and have something interesting, feel free to share them with the Azure Logic Apps team via email or you can tweet to them at @logicappsio. You can also vote for features that you feel are important and that you’d like to see in logic apps here.
The Logic Apps team are currently running a survey to know how the product/features are useful for you as a user. The team would like to understand your experiences with the product. You can take the survey here.
If you ever wanted to get in touch with the Azure Logic Apps team, here’s how you do it!
Previous Updates
In case you missed the earlier updates from the Logic Apps team, take a look at our recap blogs here –
Author: Sriram Hariharan
Sriram Hariharan is the Senior Technical and Content Writer at BizTalk360. He has over 9 years of experience working as documentation specialist for different products and domains. Writing is his passion and he believes in the following quote – “As wings are for an aircraft, a technical document is for a product — be it a product document, user guide, or release notes”. View all posts by Sriram Hariharan
by Steef-Jan Wiggers | Sep 27, 2017 | BizTalk Community Blogs via Syndication
September 2017, the last month at Macaw and about to onboard on a new journey at Codit Company. And I looking forward to it. It will mean more travelling, speaking engagements and other cool things. #Cyanblue is the new blue.
Below a picture of Tomasso, Eldert, me, Dominic (NoBuG), and Kristian in Olso (top floor or Communicate office).
I did a talk about Event Grid at NoBug wearing my Codit shirt for the first time.
Month September
September was a month filled with new challenges. I onboarded the Middleware Friday team and released two episodes (31 and 33):
Moreover, I really enjoyed doing these type of videos and looking forward to create a few more as I will be presenting an episide every alternating week. Subsequently, Kent will continu with episodes focussed around Microsoft Cloud offerings such as Microsoft Flow. And my focus will be integration in general.
In September I did a few blog posts on my own blog and BizTalk360 blog:
This month I only read one book. Yet it was a good book called: The Subtle Art of Not Giving a F*ck from Mark Manson.
Music
My favorite albums in September were:
- Chelsea Wolfe – Hiss Spun
- Satyricon – Deep Calleth Upon Deep
- Cradle Of Filth – Cryptoriana: The Seductiveness Of Decay
- Enter Shikari – The Spark
- Myrkur – Mareridt
- Arch Enemy – Will To Power
- Wolves In The Throne Room – Thrice Woven
Running
In September I continued with training and preparing for next months half marathons in London and Amsterdam.
October will be filled with speaking engagements ranging from Integration Monday to Integrate US 2017 in Redmond.
Cheers,
Steef-Jan
Author: Steef-Jan Wiggers
Steef-Jan Wiggers is all in on Microsoft Azure, Integration, and Data Science. He has over 15 years’ experience in a wide variety of scenarios such as custom .NET solution development, overseeing large enterprise integrations, building web services, managing projects, designing web services, experimenting with data, SQL Server database administration, and consulting. Steef-Jan loves challenges in the Microsoft playing field combining it with his domain knowledge in energy, utility, banking, insurance, health care, agriculture, (local) government, bio-sciences, retail, travel and logistics. He is very active in the community as a blogger, TechNet Wiki author, book author, and global public speaker. For these efforts, Microsoft has recognized him a Microsoft MVP for the past 7 years. View all posts by Steef-Jan Wiggers
by Steef-Jan Wiggers | Sep 25, 2017 | BizTalk Community Blogs via Syndication
A couple of weeks ago Azure Event Grid service became available in public preview. This service enables centralized management of events in a uniform way. Moreover, it scales with you when the number of events increases. This is made possible by the foundation the Event Grid relies on Service Fabric. Not only does it auto scale you also do not have to provision anything besides an Event Topic to support custom events (see the blog post Routing an Event with a custom Event Topic).
Event Grid is serverless, therefore you only pay for each action (Ingress events, Advanced matches, Delivery attempts, Management calls). Moreover, the price will be 30 cents per million actions in the preview and will be 60 cents once the service will be GA.
Azure Event Grid can be described as an event broker that has one of more event publishers and subscribers. Furthermore, Event publishers are currently Azure blob storage, resource groups, subscriptions, event hubs and custom events. Finally, more will be available in the coming months like IoT Hub, Service Bus, and Azure Active Directory. Subsequently, there are consumers of events (subscribers) like Azure Functions, Logic Apps, and WebHooks. And on the subscriber side too more will be available with Azure Data Factory, Service Bus and Storage Queues for instance.
To view Microsoft’s Roadmap for Event Grid please watch the Webinar of the 24th of August on YouTube.
Event Grid Preview for Azure Storage
Currently, to capture Azure Blob Storage events you will need to register your subscription through a preview program. Once you have registered your subscription, which could take a day or two, you can leverage Event Grid in Azure Blob Storage only in Central West US!
The Microsoft documentation on Event Grid has a section “Reacting to Blob storage events”, which contains a walk-through to try out the Azure Blob Storage as an event publisher.
Scenario
Having registered the subscription to the preview program, we can start exploring its capabilities. Since the landing page of Event Grid provides us some sample scenarios, let’s try out the serverless architecture sample, where one can use Event Grid to instantly trigger a Serverless function to run image analysis each time a new photo is added to a blob storage container. Hence, we will build a demo according to the diagram below that resembles that sample.
An image will be uploaded to a Storage blob container, which will be the event source (publisher). Subsequently, the Storage blob container belongs to a Storage Account containing the Event Grid capability. And finally, the Event Grid has three subscribers, a WebHook (Request Bin) to capture the output of the event, a Logic App to notify me a blob has been created and an Azure Function that will analyze the image created in the blob storage, by extracting the URL from the event message and use it to analyze the actual image.
Intelligent routing
The screenshot below depicts the subscriptions on the events on the Blob Storage account. The WebHook will subscribe to each event, while the Logic App and Azure Function are only interested in the BlobCreated event, in a particular container(prefix filter) and type (suffix filter).
Besides being centrally managed Event Grid offers intelligent routing, which is the core feature of Event Grid. You can use filters for event type, or subject pattern (pre- and suffix). Moreover, the filters are intended for the subscribers to indicate what type of event and/or subject they are interested in. When we look at our scenario the event subscription for Azure Functions is as follows.
- Event Type : Blob Created
- Prefix : /blobServices/default/containers/testcontainer/
- Suffix : .jpg
The prefix, a filter object, looks for the beginsWith in the subject field in the event. And in addition the suffix looks for the subjectEndsWith in again the subject. Consequently, in the event above, you will see that the subject has the specified Prefix and Suffix. See also Event Grid subscription schema in the documentation as it will explain the properties of the subscription schema. The subscription schema of the function is as follows:
<pre>{
"properties": {
"destination": {
"endpointType": "webhook",
"properties": {
"endpointUrl": "https://imageanalysisfunctions.azurewebsites.net/api/AnalyseImage?code=Nf301gnvyHy4J44JAKssv23578D5D492f7KbRCaAhcEKkWw/vEM/9Q=="
}
},
"filter": {
"includedEventTypes": [ "<strong>blobCreated</strong>"],
"subjectBeginsWith": "<strong>/blobServices/default/containers/testcontainer/</strong>",
"subjectEndsWith": "<strong>.jpg</strong>",
"subjectIsCaseSensitive": "true"
}
}
}</pre>
Azure Function Event Handler
The Azure Function is only interested in a Blob Created event with a particular subject and content type (image .jpg). This will be apparent once you inspect the incoming event to the function.
<pre>[{
"topic": "/subscriptions/0bf166ac-9aa8-4597-bb2a-a845afe01415/resourceGroups/rgtest/providers/Microsoft.Storage/storageAccounts/teststorage666",
"<strong>subject</strong>": "<strong>/blobServices/default/containers/testcontainer/</strong>blobs/NinoCrudele.<strong>jpg</strong>",
"<strong>eventType</strong>": "<strong>Microsoft.Storage.BlobCreated</strong>",
"eventTime": "2017-09-01T13:40:33.1306645Z",
"id": "ff28299b-001e-0045-7227-23b99106c4ae",
"data": {
"api": "PutBlob",
"clientRequestId": "206999d0-8f1b-11e7-a160-45670ee5a425",
"requestId": "ff28299b-001e-0045-7227-23b991000000",
"eTag": "0x8D4F13F04C48E95",
"contentType": "image/jpeg",
"contentLength": 32905,
"blobType": "<strong>BlockBlob</strong>",
"url": "https://teststorage666.blob.core.windows.net/testcontainer/NinoCrudele.jpg",
"sequencer": "0000000000000AB100000000000437A7",
"storageDiagnostics": {
"batchId": "f11739ce-c83d-425c-8a00-6bd76c403d03"
}
}
}]</pre>
The same intelligence applies for the Logic App that is interested in the same event. The WebHook subscribes to all the events and lacks any filters.
The scenario solution
The solution contains a storage account (blob), a registered subscription for Event Grid Azure Storage, a Request Bin (WebHook), a Logic App and a Function App containing an Azure function. The Logic App and Azure Function subscribe to the BlobCreated event with the filter settings.
The Logic App subscribes to the event once the trigger action is defined. The definition is shown in the picture below.
Note that the resource name has to be specified explicitly (custom value) as the resource type Microsoft.Storage has been set explicitly too. The resource types currently available are Resource Groups, Subscriptions, Event Grid Topics and Event Hub Namespaces, while Storage is still in a preview program. Therefore, registration as described earlier is required. As a result with the above configuration, the desired events can be evaluated and processed. In case of the Logic App, it is parsing the event and sending an email notification.
Image Analysis Function
The Azure Function is interested in the same event. And as soon as the event is pushed to Event Grid once a blob has been created, it will process the event. The URL in the event https://teststorage666.blob.core.windows.net/testcontainer/NinoCrudele.jpg will be used to analyse the image. The image is a picture of my good friend Nino Crudele.
This image will be streamed from the function to the Cognitive Services Computer Vision API. The result of the analysis can be seen in the monitor tab of the Azure Function.
The result of the analysis with high confidence is that Nino is smiling for the camera. We, as humans, would say that this is obvious, however do take into consideration that a computer is making the analysis. Hence, the Computer Vision API is a form of Artificial Intelligence (AI).
The Logic App in our scenario will parse the event and sent out an email. The Request Bin will show the raw event as is. And in case I, for instance, delete a blob, then this event will only be caught by the WebHook (Request Bin) as it is interested in any event on the Storage account.
Summary
Azure Event Grid is unique in its kind as now other Cloud vendor has this type of service that can handle events in a uniform and serverless way. Although it is still early days as this service is in preview a few weeks. However, with expansion of event publishers and subscribers, management capabilities and other features it will mature in the next couple of months.
The service is currently only available in, West Central US and West US. However, over the course of time it will become available in every region. And once it will become GA the price will increase.
Working with Storage Account as a source (publisher) of events unlocked new insights in the Event Grid mechanisms. Moreover, it shows the benefits of having one central service in Azure for events. And the pub-sub and push of events are the key differentiators towards the other two services Service Bus and Event Hubs. Therefore, no longer do you have to poll for events and/or develop a solution for it. To conclude the Service Bus Team has completed the picture for messaging and event handling.
Author: Steef-Jan Wiggers
Steef-Jan Wiggers has over 15 years’ experience as a technical lead developer, application architect and consultant, specializing in custom applications, enterprise application integration (BizTalk), Web services and Windows Azure. Steef-Jan is very active in the BizTalk community as a blogger, Wiki author/editor, forum moderator, writer and public speaker in the Netherlands and Europe. For these efforts, Microsoft has recognized him a Microsoft MVP for the past 5 years. View all posts by Steef-Jan Wiggers
by BizTalk Team | Sep 25, 2017 | BizTalk Community Blogs via Syndication
If you missed the chance to attend INTEGRATE 2017 in London this year, now is your chance to participate in INTEGRATE 2017 USA at the Microsoft Redmond Campus. Come see Scott Guthrie, Executive Vice President for the Cloud and Enterprise division, deliver the keynote address. Have a chance to network with Microsoft employees along with Microsoft Integration MVPs.
Further details and registration information can be found at https://www.biztalk360.com/integrate-2017-usa/
