September 16, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

September 16, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform and Azure iPaaS?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

If you want to receive these updates weekly, then don’t forget to Subscribe!

 

Microsoft Announcements and Updates

 

Community Blog Posts

 

Videos

 

Podcasts

 

How to get started with iPaaS design & development in Azure?

Feedback

Hope this would be helpful. Please feel free to reach out to me with your feedback and questions.

BizTalk360 Application Server – Hardware Requirements

BizTalk360 Application Server – Hardware Requirements

Introduction

Managing the infrastructure of BizTalk/SQL Servers is a vital task to BizTalk/System Administrators. Manually monitoring the infrastructure of BizTalk Environment is a cumbersome task.  BizTalk360 is the operational, monitoring and analytics tool which is used to manage the health of BizTalk and SQL Servers in an efficient way. Many customers approach the BizTalk360 Support Team to provide the Hardware Requirements of BizTalk360.

In our Documentation portal, we have provided the BizTalk360 installation document with hardware requirements for a BizTalk360 instance which contains one BizTalk Group. We provided the prerequisites with both software and hardware requirements, considering the BizTalk360 is set up in a stand-alone server.

We also have BizTalk360 Installation guide to set up BizTalk360. This article is focused on the hardware requirements of Windows Server to install the BizTalk360 application.

BizTalk Group Configuration

In most organizations, BizTalk Groups are set up in at least 3 environments (Production, Staging, QA/Dev). Considering these configurations of BizTalk Groups, we are suggesting the BizTalk360 application servers hardware requirements with BizTalk360 Gold tier. Having the Gold tier, and not Platinum, means that you will not have the ability to use the BizTalk360 Analytics features, but this will also save you from needing additional resources which come along with the Platinum features. However, with the mentioned hardware requirements, we kept the possibility open for upgrading to the Platinum tier, without the need for additional hardware requirements.

BizTalk-Group-Configuration

BizTalk360 – Application Server Configuration

Goal: Create a separate instance of BizTalk360 Application Servers for twos version of BizTalk Server environments (BizTalk Server 2016 & 2013 R2).

Scenario 1

We have written down the requirements for the BizTalk360 Application servers for BizTalk Server 2013 R2 environments, as well as BizTalk Server 2016 environments. Each of these BizTalk360 Application servers will be used to operate and monitor 3 different BizTalk environments. The specifications for these servers have also been written with that requirement in mind.

You will also find the requirements for the SQL server, which will host 2 SQL Server instances. Each SQL instance will host a BizTalk360 database, which will connect to either the BizTalk360 application server for monitoring the BizTalk Server 2016 environments or the one for monitoring the BizTalk Server 2013 R2 environments.

The system requirements to install BizTalk360 are considered to configure 3 BizTalk Groups in per BizTalk360 installation.

BizTalk360 Application Server

Resource

Requirements

Computer and Processor

A computer with an Intel Pentium-compatible CPU that is 1 GHz or higher quad processors. The 64-bit versions of BizTalk Server require a 64-bit operating system running on an x64 based system.

Memory

16 GB or higher

Hard Disk

Minimum 20 GB of available hard-disk space for a complete installation including the operating system and all prerequisite software. The hard disk must be NTFS formatted.

 

BizTalk360 – Database Server (2 SQL Instances)

Resource

Requirements

Computer and Processor

A computer with an Intel Pentium-compatible CPU that is 1 GHz or higher quad processors. The 64-bit versions of BizTalk Server require a 64-bit operating system running on an x64 based system.

Memory

16 GB or higher

Hard Disk (Database)

Minimum 200 GB of available hard-disk space for a complete installation including the operating system and all prerequisite software. The hard disk must be NTFS formatted.

 

Why do we need these specifications?

In complex BizTalk environments, multiple BizTalk Servers and SQL Servers are configured. Based on these BizTalk Group configurations, we arrived at the BizTalk360 Application Server and BizTalk360 Database Server Configuration.

BizTalk360 Application Server

  1. Processor: To process the various data from (2 or more) BizTalk Servers and SQL Servers, BizTalk360 Application Server requires a high-end processor.
    • WMI Queries are used to collect the Event Log Sources from BizTalk and SQL Servers

    • Querying against the BizTalk databases like Message Box and Tracking Database

    • PerfMon is used to collect System Resources and Performance Metrics analytics data

  2. Hard Disk: Disk Storage of BizTalk360 Application Server is necessary to hold the following data
    • Message Content (Archived Messages)

    • User Profile Images

    • Processed Email Content

    • Custom Widget Scripts

BizTalk360 Database Server

BizTalk360 collects Event Log Sources and Performance Metrics from BizTalk and SQL Server and stores that data into the BizTalk360 database. It also collects Tracking Data to determine the Message Patterns and Transmission Failure Rates.

The BizTalk360 Administrator makes sure the Data Purging Jobs are in healthy status to manage the growth of BizTalk360 database.

BizTalk360 High Availability

Many customers are configuring the BizTalk360 Monitoring Service and Analytics Service in high availability with Windows server cluster setup.  In this case, you need two BizTalk360 Application servers, with the configuration mentioned in the above table.

Note: Customers can install all three components in high availability configuration

  • BizTalk360 Web Application
  • Monitoring Service
  • Analytics Service

High-Availability-Status

Users can manage the BizTalk360 Monitoring and Analytics services in the Settings -> BizTalk360 Health. Read this article to know more about BizTalk360 High Availability status.

Scenario 2

Some customers use different instances of BizTalk360 for the same version of a BizTalk Group in QA, Staging and Production environments. In this scenario, three different Windows servers are required for BizTalk360 Application configuration. Similarly, three SQL Servers are required to configure the BizTalk360 databases. If they want to set up high availability of BizTalk360 in production, then one BizTalk360 Application Server and one SQL Server are added to the list.

BizTalk360-Application-Configuration

Conclusion

Once all the setup and configurations are in place, it is a quick and seamless task to install/upgrade BizTalk360. The customer suggestion and feedback are always heard and addressed, which helps us to improve the product and provide better service.

The post BizTalk360 Application Server – Hardware Requirements appeared first on BizTalk360.

Performance Data Collection Optimization

Performance Data Collection Optimization

BizTalk360 aims to offer capabilities out-of-the-box from tools like the Windows Performance Monitor. The BizTalk360 Analytics service collects the performance data for the various server types like BizTalk, SQL, IIS, and Windows. Based on the server type selection, the Analytics service will start collecting the performance data and that can be visualized in BizTalk360 Analytics widgets or the user can push that data to third-party APM’s (Application Performance Monitoring) like New Relic, AppDynamics or Dynatrace.

BizTalk360 also allows administrators to automatically execute queries against the Tracking database, at a specific interval, to view, analyze and troubleshoot the tracked data. For the Tracking data, BizTalk360 provides a similar user experience to the performance data collection. This allows the user to pick the metrics they want to collect.         

In this article we will look into detailed information about:

  1. Performance Data Collection of each type (BizTalk, Windows, SQL, IIS)
  2. Tracking Data Collection
  3. Analytics Custom widgets in BizTalk360

Performance Data Collection

To analyze the performance of a BizTalk environment, BizTalk360 is equipped with Performance Data Collection in the Analytics section. Once the user enabled any one of the server types (BizTalk, SQL, IIS, and Windows) in any selected environment, the BizTalk360 Analytics service will start to collect all the related counters.

For enabling the performance data, navigate to the Settings section of BizTalk360. In the Analytics configuration, you can manage the performance data collection in the Manage Analytics section.

Once the performance data collection is enabled for a server, the Analytics service will start to collect counter data on the next polling cycle.

For a larger environment, the Analytics service may need to collect multiple counter data, but the user might not require all those data. For instance, if a user wants only the BizTalk message related counter data to analyze the performance of the BizTalk environment, the Analytics service collects all the BizTalk related counters.

Perfromance-Grid

Performance Data collection Optimization

To address this challenge, BizTalk360 allows the user to manage the metrics collection from the v9.0 phase 2. To narrow down the data collection at a server level, the user can optimize the data collection using the “Manage Metrics” option for each server in an environment.

With the “Manage Metrics” option, the user can choose the required performance metrics based on types, so that BizTalk360 Analytics will start collecting data only for those metrics.

Let’s take a deep dive on how a user can manage the metrics collection at the server level.

In BizTalk360, each server’s counter is segregated into 4 types:

  1. BizTalk
  2. Windows
  3. IIS
  4. SQL

Most of the BizTalk environments will be in a multi-server setup where at least BizTalk Server and SQL Server are configured in different machines. A user may expect BizTalk related counters from BizTalk Server or SQL related counters from SQL Server. A user may not want SQL/IIS related performance metrics data in a BizTalk server machine. By using the “Manage Metrics” option in each server, the user can choose the metrics needed for each server. This improves the overall performance of the Analytics service and avoids the growth of the database.

Microsoft-Teams

Tracking Data Collection

BizTalk360 allows the user to run queries against the Tracking database and display the result in a widget. In some scenarios, retaining the tracking data longer than the actually required duration, causes the database to grow exponentially. In other scenarios, the environment doesn’t follow a strict purging policy. For both cases, it is hard to query against the Tracking database due to its size.

In a larger database, querying against the Tracking database and displaying the data, results in a graphical form in the widget. This could degrade the performance of the queries and impact on the user experience of the Analytics dashboard.

To overcome these challenges, we created a service in v8.3 that collects the tracking data periodically in small chunks and avoiding expensive queries on the Tracking database, and metrics are collected rather than pulling all the tracking data into BizTalk360 database.

Once decided to go with new this data collection method, BizTalk360 provides a customizable interface where user can pick the metrics they wanted to collect. A similar user experience to the performance data collection in BizTalk360 allows the user to fine-tune the data collection at an environment level.

Tracking-Data-settings

Analytics Custom Widget in BizTalk360

BizTalk360 is loaded with customizable information and a variety of widgets to choose from, which can be added to your dashboards. The BizTalk360 Analytics dashboard is loaded with performance data widgets and tracking database widgets, like the Messaging performance and transmission failure rate widgets.

Widget-Metrics

Users can add custom widgets to both the Analytics dashboard and custom dashboards in BizTalk360. The product provides rich options for adding widgets in these dashboards. Users can choose

  • Date Range (24 hours, 7 days, 30 days, custom date range)
  • Comparing data with previously generated data
  • Graph type (line, column, area)

Based on the selected date range, the performance data gets collected from the BizTalk servers.

Once the analytics components, for which the data needs to be collected, are enabled, the different metrics for the custom widgets will get enabled. With the custom widgets, you can choose different metrics for which you want to view the data. Users can choose messaging performance/transmission failure of tracking data or performance data from the drop-down.

Conclusion

Do you wish to see more information on the widgets or from the product itself? Then, please put in your suggestions and feedback in our user voice portal. The existing ideas can also be voted for. We at BizTalk360 aim at providing the features that fulfill the customer requirements. It is from this feedback portal that the features get picked up for every release.

For more information about how these widgets work, check our Documentation Portal.

The post Performance Data Collection Optimization appeared first on BizTalk360.

September 8, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

September 8, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform and Azure iPaaS?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

If you want to receive these updates weekly, then don’t forget to Subscribe!

 

Microsoft Announcements and Updates

 

Community Blog Posts

 

Videos

 

Podcasts

 

How to get started with iPaaS design & development in Azure?

Feedback

Hope this would be helpful. Please feel free to reach out to me with your feedback and questions.

BizTalk360 V9.0.2 Release Blog

BizTalk360 V9.0.2 Release Blog

It’s time to upgrade to the latest version of BizTalk360 V9.0 Phase2. We always aim to improve our product based on customer’s feedback to meet their business needs. This release comes with a handful of most customer asks features and enhancements.

A brief summary of features and enhancement we introduced in V9.2

Host Instance Monitoring – This feature is extended to support “AtleastOneActive” monitoring state for Clustered and Non-Cluster Host Instances.

Dynatrace Integration We have integrated Dynatrace in BizTalk360; with this, you can see the BizTalk Server related performance metrics in Dynatrace.

SQL Query Custom Report – In BizTalk Reporting, we introduced an additive widget called Custom SQL Query widget through which SQL queries can be mapped to get the top 100 query result as a report based on the scheduled configuration.

BizTalk Group Dashboard – In BizTalk360’s Monitoring dashboard it was not possible to view the status of all the mapped artifacts in a single view; the user needs to select the alarm each time to view the status of the mapped artifacts of each alarm. To overcome this challenge, we have introduced the BizTalk Group Dashboard which gives a consolidated view of the status of all the artifacts which are mapped to all the alarms in an environment.

Governance Audit (User Access Policies and License Activities) – BizTalk360 Activities will give a clear view of user actions with respect to Licenses and User Access Policies. License activities include activation, deactivation, remove and manual activation and deactivation. User Access Policy activities will include create, update, delete and import operation under the following section i.e. Manage Users, Manage Application Groups and Manage Custom User profiles

Performance Data Collection Improvements – To improve the performance data collection we are providing an option for the user to select the required performance counters on each type. This way, BizTalk360 will start collecting data only for the selected counters.

Messaging Flow Patterns – Analytics Messaging Flow Patterns can be determined from the BizTalk Management database. Users can switch the Messaging Flow determination technique in the Message Patterns configuration. The Dynamic Pattern option is used to determine the Message Patterns from the tracking database (earlier versions). 

Let us jump into more detail of the new features

Host Instance Monitoring

A BizTalk Server Host is a logical container that contains the BizTalk Server runtime processes in which you deploy items such as adapter handlers, receive locations (including pipelines), and orchestrations.  The host instance is the runtime process where the message processing, receiving, and transmitting occurs. Obviously, it is very important to ensure your host instances are running in your BizTalk environment. BizTalk360 already made it easy, by introducing the Host Instance Monitoring functionality. In BizTalk360, you can monitor the BizTalk Host Instances by setting the Expected State. If the expected state doesn’t match with the current state of the host instance, you will get notified.

Out of the box, BizTalk360 allows users to monitor the Host Instances using the “AtleastOneActive” state. 

If this state is assigned as the Expected State, then the monitoring service will verify that across all the instances for the respective clustered host, at least one host instance is active, guaranteeing that the server is running, and no downtime happened for that host/server.

The user can also enable AutoCorrect for Clustered and Non-Clustered host instances monitoring.

Host Instance Monitoring

Dynatrace Integration

Dynatrace is an APM tool which has a generic capability to monitor operating systems, network protocols, system metrics, network infrastructure, applications. By Integrating BizTalk360 with Dynatrace you can view the BizTalk Server performance metrics in your Dynatrace environments; which does not need any agent to be installed in the BizTalk Servers to collect data and it does not require any configuration changes at the server level.

BizTalk360 Analytics Service holds a sub-service for Dynatrace which creates custom metrics in your Dynatrace environment for all the counters which you have enabled in the BizTalk360 Analytics configuration section. Each minute the sub-service will start pushing the collected BizTalk Server performance data to the respective custom metrics. These custom metrics data can be mapped to a custom chart and viewed in your Dynatrace dashboard.           

Dynatrace                                        

 SQL Query Custom Report

With the Biztalk360 reporting users can generate PDF documents of critical performance metrics and send emails to the user at specific time periods, depending on the requirement in Reporting Section. Now with this version, you can get your SQL query results as a report just by simply mapping the configured secure SQL queries to the reporting schedule. This avoids switching between multiple tools.

Note: BizTalk360 consists of 6 default queries which the user can only execute. In addition to that, you can create new custom SQL queries as per the business need.

Reports can be generated on Daily, Weekly, and Monthly. 

SQL Query Custom Reports

BizTalk Group Dashboard

Monitoring and Notification is one among the most important feature in BizTalk360.In which user can use the Monitoring dashboard to monitor their BizTalk Environment and get notified if any violation persists. But there is one usability problem exist where user cannot see the consolidated view of all the alarms and their mapped artifact in a single view. To Overcome this, we brought the “BizTalk Group Dashboard”.

The BizTalk Group Dashboard is designed to monitor all the artifacts (and view all errors) which are mapped to all the configured alarms in BizTalk360, in a single view. The “BizTalk Group Dashboard” will automatically pick up all the artifacts (which are mapped to any of the BizTalk360 alarms) and displays the status of the artifacts in a graphical manner. Also, the error/warning details of the displayed artifacts are shown in a grid view. The BizTalk Group dashboard checks the status of the artifacts in every 60 seconds.

BizTalkGroup_Dashboard

Governance Audit (User Access Policies and License Activities)

In addition to BizTalk360 auditing Activities, now user can audit the user Access policies and license activities details in BizTalk360 activities.

In License auditing, users will be able to see the activities like License Activation, Deactivation and Removal of licenses, along with the details like Environment name, Product License Type, Environment Type, Order number, License code, License edition, and Max BizTalk servers.

In User Access Policy auditing, BizTalk360 captures the activities like,

  • Adding Super/Custom users
  • Modifying the access permission for existing users
  • Removing users from an environment
  • Creating, Modifying or Deleting the Application Groups
  • Creating, Modifying or Deleting the Custom User Profiles

BizTalkGroup_Dashboard

Enhancements

Below are some of the major improvements we have done in this release

Performance Data Collection Improvements

In earlier versions, user can enable the performance data collection for the various server types, say BizTalk, SQL, IIS, Windows, where BizTalk360 will start collecting performance data for all the counters of the selected type. From this version on, to improve the performance, we are providing an option for the user to select the required performance counters on each type. This way, BizTalk360 will start collecting data only for the selected counters.        

By using “Manage Metrics” option user can manage the performance metrics data collection for each server in the configured. User can choose what type of data they want in the environment level from the neat tree view diagram.  Based on the selection analytics service collects the performance data which avoids performance impact.

PerformanceMetrics

Messaging Flow Patterns

In complex BizTalk environment, it is important to visualize the patterns of how messages are processed within the system or integrate with multiple disparate systems to handle the message transfer from one point to another. It could be a simple direct transfer between the two ports, or through a complex messaging route passing through orchestrations to perform certain validations, etc. It is helpful to BizTalk Administrators or Business users to track the flow of messages in different stages. For this purpose, the BizTalk Admin Console has the capability to view the workflow of messages, although that feature will mainly be of use for BizTalk Administrators.

we are bringing management capabilities like hiding and deleting of message flows.

The Hide Operation can be used on the message patterns that are decommissioned or less frequently used. The Delete Operation can be applied on the flows that are no longer used in BizTalk Group.

MessageFlowPatterns

FIPS Compliance Support in Cryptography

BizTalk360 now supports AE5 cryptography algorithm to support FIPS Compliance in Encryption/Decryption mechanism.

 SQL Server Monitoring Improvements

  • The platinum Users can now add up to 4 servers for monitoring.
  • The SQL Server added for monitoring now can be removed.
  • The SQL Server which is not reachable can also be added for monitoring, However, the monitoring works seamlessly only when the server is reachable.

SQL Collation support in BizTalk360

BizTalk Server can support both case sensitive and case insensitive collation. To make BizTalk360 more compatible with BizTalk Server, we have improved BizTalk360 in such a way that it will support case sensitive collation as well.

Analytics Widget Improvements

  • Information about the different legends in the BizTalk Messaging Performance widget has been added.
  • The date/time range for the last 24 hours has been modified to show the time range for the last 24 hours instead of 12 hours

Support for rules created using BRE Pipeline Framework

The rules created using .NET classes with BRE pipeline framework are getting displayed in BizTalk360

ESB Exceptions

In ESB Exception user can provide the Action Type and Comments while deleting the faults, the same will be audited for reference. Also when the message is edited before resubmit then it will be audited as ‘resubmit with data modification’.

Besides these new features and enhancements, we have fixed a few bugs. Refer  release notes for more information.

Conclusion

Considering the feedback from our customers, BizTalk360 will continue to provide more useful features. Why not give BizTalk360 a try! It takes only 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity of your own BizTalk Environments. 

The post BizTalk360 V9.0.2 Release Blog appeared first on BizTalk360.

September 1, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

September 1, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform and Azure iPaaS?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

If you want to receive these updates weekly, then don’t forget to Subscribe!

 

Microsoft Announcements and Updates

 

Community Blog Posts

 

Videos

 

Podcasts

 

How to get started with iPaaS design & development in Azure?

Feedback

Hope this would be helpful. Please feel free to reach out to me with your feedback and questions.

BizTalk JSON Schema Wizard: Error in JSON Instance File.XMLNodeConverter can only convert JSON that begins with an object

BizTalk JSON Schema Wizard: Error in JSON Instance File.XMLNodeConverter can only convert JSON that begins with an object

This week while trying to use the BizTalk Server 2016 JSON Schema Wizard to generate an XML schema from a specific JSON file to be sent to a Logic App I got the following error:

Error in JSON Instance File.XMLNodeConverter can only convert JSON that begins with an object. Path ‘’, line 1, position 1.

The JSON file had the following format:

[
  {
    "IntID": 208,
    "ItemLogID": 14255826,
    "Step": "IN",
    "BusinessUnit": "TST",
  },
  {
    "IntID": 209,
    "ItemLogID": 14257419,
    "Step": "IN",
    "BusinessUnit": "TST",
  }
]

Basically, I’m trying to send a list of “objects” that in my case are Locks, to be processed.

Cause

The cause of this “problem” is that a JSON array of objects, it may not make sense in a BizTalk Server XML world as Morten la Cour very well written in this forum. At that sentence, I will add: “at it his being done today”. Because with a few improvements could be smarter and support this type of messages.

Why? The BizTalk
JSON Schema Wizard is a simple and “stupid” converter, it will ask only for you
to provide a root node and a namespace that it will add to the XML Schema and
the XML JSON representation because it will require that information to be able
to uniquely identify this type of message.

But it will not understand what is a “not identified” object array because it needs to give it a Record name in the XML equivalent.

So, that means
that it will not support JSON arrays?

No, it will support JSON arrays if you provide a field name to that array, i.e., instead of having:

[
…
]

You should have:

{
   "field name": [
   …
   ]
}

Solution

If you don’t have the control over that JSON message,
you may need to create a custom pipeline component to add or remove this field
name that will identify the array in the XML equivalent message.

If you have control over the structure of the JSON
message the simple way is to modify the structure of the message to include a
field name to identify the array. Lucky it was my case, so I modify the
original structure descrived above to be:

  {
    "locks": [
    {
      "IntID": 208,
      "ItemLogID": 14255826,
      "Step": "IN",
      "BusinessUnit": "TST",
    },
    {
      "IntID": 209,
      "ItemLogID": 14257419,
      "Step": "IN",
      "BusinessUnit": "TST",
    }
  ]
}

Now, if I try to run the BizTalk JSON Schema Wizard
against this message it will be able to create the JSON Schema.

The post BizTalk JSON Schema Wizard: Error in JSON Instance File.XMLNodeConverter can only convert JSON that begins with an object appeared first on SANDRO PEREIRA BIZTALK BLOG.

My Pluralsight course—Getting Started with Concourse—is now available!

My Pluralsight course—Getting Started with Concourse—is now available!

Design software that solves someone’s “job to be done“, build it, package it, ship it, collect feedback, learn, and repeat. That’s the dream, right? For many, shipping software is not fun. It’s downright awful. Too many tickets, too many handoffs, and too many hours waiting. Continuous integration and delivery offer some relief, as you keep producing tested, production-ready artifacts. Survey data shows that we’re not all adopting this paradigm as fast as we should. I figured I’d do my part by preparing and delivering a new video training course about Concourse.

I’ve been playing a lot with Concourse recently, and published a 3-part blog series on using it to push .NET Core apps to Kubernetes. It’s an easy-to-use CI system with declarative pipelines and stateless servers. Concourse runs jobs on Windows or Linux, and works with any programming language you use.

My new hands-on Pluralsight course is ~90 minutes long, and gives you everything you need to get comfortable with the platform. It’s made up of three modules. The first module looks at key concepts, the Concourse architecture, and user roles, and we set up our local environment for development.

The second module digs deep into the primitives of Concourse: tasks, jobs and resources. I explain how to configure each, and then we go hands on with each. There are aspects that took me a while to understand, so I worked hard to explain these well!

The third and final module looks at pipeline lifecycle management and building manageable pipelines. We explore troubleshooting and more.

Believe it or not, this is my 20th course with Pluralsight. Over these past 8 years, I’ve switched job roles many times, but I’ve always enjoyed learning new things and sharing that information with others. Pluralsight makes that possible for me. I hope you enjoy this new course, and most importantly, start doing CI/CD for more of your workloads!

August 26, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

August 26, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform and Azure iPaaS?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

If you want to receive these updates weekly, then don’t forget to Subscribe!

 

Microsoft Announcements and Updates

Community Blog Posts

 

Videos

 

Podcasts

 

How to get started with iPaaS design & development in Azure?

Feedback

Hope this would be helpful. Please feel free to reach out to me with your feedback and questions.

Building an Azure-powered Concourse pipeline for Kubernetes  – Part 3: Deploying containers to Kubernetes

Building an Azure-powered Concourse pipeline for Kubernetes – Part 3: Deploying containers to Kubernetes

So far in this blog series, we’ve set up our local machine and cloud environment, and built the initial portion of a continuous delivery pipeline. That pipeline, built using the popular OSS tool Concourse, pulls source code from GitHub, generates a Docker image that’s stored in Azure Container Registry, and produces a tarball that’s stashed in Azure Blob Storage. What’s left? Deploying our container image to Azure Kubernetes Service (AKS). Let’s go.

Generating AKS credentials

Back in blog post one, we set up a basic AKS cluster. For Concourse to talk to AKS, we need credentials!

From within the Azure Portal, I started up an instance of the Cloud Shell. This is a hosted Bash environment with lots of pre-loaded tools. From here, I used the AKS CLI to get the administrator credentials for my cluster.

az aks get-credentials --name seroter-k8s-cluster --resource-group demos --admin

This command generated a configuration file with URLs, users, certificates, and tokens.

I copied this file locally for use later in my pipeline.

Creating a role-binding for permission to deploy

The administrative user doesn’t automatically have rights to do much in the default cluster namespace. Without explicitly allowing permissions, you’ll get some gnarly “does not have access” errors when doing most anything. Enter role-based access controls. I created a new rolebinding named “admin” with admin rights in the cluster, and mapped to the existing clusterAdmin user.

kubectl create rolebinding admin --clusterrole=admin --user=clusterAdmin --namespace=default

Now I knew that Concourse could effectively interact with my Kubernetes cluster.

Giving AKS access to Azure Container Registry

Right now, Azure Container Registry (ACR) doesn’t support an anonymous access strategy. Everything happens via authenticated users. The Kubernetes cluster needs access to its container registry, so I followed these instructions to connect ACR to AKS. Pretty easy!

Creating Kubernetes deployment and service definitions

Concourse is going to apply a Kubernetes deployment to create pods of containers in the cluster. Then, Concourse will apply a Kubernetes service to expose my pod with a routable endpoint.

I created a pair of configurations and added them to the ci folder of my source code.

The deployment looks like:

apiVersion: extensions/v1beta1
 kind: Deployment
 metadata:
   name: demo-app
   namespace: default
   labels:
     app: demo-app
 spec:
   replicas: 1
   template:
     metadata:
       labels:
         app: demo-app
     spec:
       containers:
       - name: demo-app
         image: myrepository.azurecr.io/seroter-api-k8s:latest
         imagePullPolicy: Always
         ports:
         - containerPort: 8080
       restartPolicy: Always 

This is a pretty basic deployment definition. It points to the latest image in the ACR and deploys a single instance (replicas: 1).

My service is also fairly simple, and AKS will provision the necessary Azure Load Balancer and public IP addresses.

 apiVersion: v1
 kind: Service
 metadata:
   name: demo-app
   namespace: default
   labels:
     app: demo-app
 spec:
   selector:
     app: demo-app
   type: LoadBalancer
   ports:
     - name: web
       protocol: TCP
       port: 80
       targetPort: 80 

I now had all the artifacts necessary to finish up the Concourse pipeline.

Adding Kubernetes resource definitions to the Concourse pipeline

First, I added a new resource type to the Concourse pipeline. Because Kubernetes isn’t a baked-in resource type, we need to pull in a community definition. No problem. This one’s pretty popular. It’s important than the Kubernetes client and server are expecting the same Kubernetes version, so I set the tag to match my AKS version.

resource_types:
- name: kubernetes
  type: docker-image
  source:
    repository: zlabjp/kubernetes-resource
    tag: "1.13"

Next, I had to declare my resource itself. It has references to the credentials we generated earlier.

resources:
- name: azure-kubernetes-service
  type: kubernetes
  icon: azure
  source:
    server: ((k8s-server))
    namespace: default
    token: ((k8s-token))
    certificate_authority: |
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----

There are a few key things to note here. First, the “server” refers to the cluster DNS server name in the credentials file. The “token” refers to the token associated with the clusterAdmin user. For me, it’s the last “user” called out in the credentials file. Finally, let’s talk about the certificate authority. This value comes from the “certificate-authority-data” entry associated with the cluster DNS server. HOWEVER, this value is base64 encoded, and I needed a decoded value. So, I decoded it, and embedded it as you see above.

The last part of the pipeline? The job!

jobs:
- name: run-unit-tests
  [...]
- name: containerize-app
  [...]
- name: package-app
  [...]
- name: deploy-app
  plan:
  - get: azure-container-registry
    trigger: true
    passed:
    - containerize-app
  - get: source-code
  - get: version
  - put: azure-kubernetes-service
    params:
      kubectl: apply -f ./source-code/seroter-api-k8s/ci/deployment.yaml -f ./source-code/seroter-api-k8s/ci/service.yaml
  - put: azure-kubernetes-service
    params:
      kubectl: |
        patch deployment demo-app -p '{"spec":{"template":{"spec":{"containers":[{"name":"demo-app","image":"myrepository.azurecr.io/seroter-api-k8s:'$(cat version/version)'"}]}}}}' 

Let’s unpack this. First, I “get” the Azure Container Registry resource. When it changes (because it gets a new version of the container), it triggers this job. It only fires if the “containerize app” job passes first. Then I get the source code (so that I can grab the deployment.yaml and service.yaml files I put in the ci folder), and I get the semantic version.

Next I “put” to the AKS resource, twice. In essence, this resource executes kubectl commands. The first command does a kubectl apply for both the deployment and service. On the first run, it provisions the pod and exposes it via a service. However, because the container image tag in the deployment file is to “latest”, Kubernetes actually won’t retrieve new images with that tag after I apply a deployment. So, I “patched” the deployment in a second “put” step and set the deployment’s image tag to the semantic version. This triggers a pod refresh!

Deploy and run the Concourse pipeline

I deployed the pipeline as a new revision with this command:

fly -t rs set-pipeline -c azure-k8s-final.yml -p azure-k8s-final

I unpaused the pipeline and watched it start up. It quickly reached and completed the “deploy to AKS” stage.

But did it actually work? I jumped back into the Azure Cloud Shell to check it out. First, I ran a kubectl get pods command. Then, a kubectl get services command. The first showed our running pod, and the second showed the external IP assigned to my pod.

I also issued a request to that URL in the browser, and got back my ASP.NET Core API results.

Also to prove that my “patch” command worked, I ran the kubectl get deployment demo-app –output=yaml command to see which container image my deployment referenced. As you can see below, it no longer references “latest” but rather, a semantic version number.

With all of these settings, I now have a pipeline that “just works” whenever I updated my ASP.NET Core source code. It tests the code, packages it up, and deploys it to AKS in seconds. I’ve added all the pipelines we created here to GitHub so that you can easily try this all out.

Whatever CI/CD tool you use, invest in automating your path to production.