Creating a continuous integration pipeline in Concourse for a test-infused ASP.NET Core app

Creating a continuous integration pipeline in Concourse for a test-infused ASP.NET Core app

Trying to significantly improve your company’s ability to build and run good software? Forget Docker, public cloud, Kubernetes, service meshes, Cloud Foundry, serverless, and the rest of it. Over the years, I’ve learned the most important place you should start: continuous integration and delivery pipelines. Arguably, “apps on pipeline” is the most important “transformation” metric to track. Not “deploys per day” or “number of microservices.” It’s about how many apps you’ve lit up for repeatable, automated deployment. That’s a legit measure of how serious you are about being responsive and secure.

All this means I needed to get smarter with Concourse, one of my favorite tools for CI (and a little CD). I decided to build an ASP.NET Core app, and continuously integrate and deliver it to a Cloud Foundry environment running in AWS. Let’s go!

First off, I needed an app. I spun up a new ASP.NET Core Web API project with a couple REST endpoints. You can grab the source code here. Most of my code demos don’t include tests because I’m in marketing now, so YOLO, but a trustworthy pipeline needs testable code. If you’re a .NET dev, xUnit is your friend. It’s maintained by my friend Brad, so I basically chose it because of peer pressure. My .csproj file included a few references to bring xUnit into my project:

  • “Microsoft.NET.Test.Sdk” Version=”15.7.0″
  • “xunit” Version=”2.3.1″
  • “xunit.runner.visualstudio” Version=”2.3.1″

Then, I created a class to hold the tests for my web controller. I included one test with a basic assertion, and another “theory” with an input data set. These are comically simple, but prove the point!

   public class TestClass {
        private ValuesController _vc;
public TestClass() {
            _vc = new ValuesController();

        public void Test1(){
            Assert.Equal("pivotal", _vc.Get(1));

        public void Test2(int value) {
            Assert.Equal("public", _vc.GetPublicStatus(value));

When I ran dotnet test against the above app, I got an expected error because the third inline data source led to a test failure, since my controller only returns “public” companies when the input value is between 1 and 10. Commenting the offending inline data source led to a successful test run.


Ok, the app was done. Now, to put it on a pipeline. If you’ve ever used shameful swear words when wrangling your CI server, maybe it’s worth joining all the folks who switched to Concourse. It’s a pretty straightforward OSS tool that uses a declarative model and containers for defining and running pipelines, respectively. Getting started is super simple. If you’re running Docker on your desktop, that’s your easiest route. Just grab this Docker Compose file from the Concourse GitHub repo. I renamed mine to docker-compose.yml, jumped into a Terminal session, switched to the folder holding this YAML file, and ran docker compose up -d. After a second or two, I had a PostgreSQL server (for state) and a Concourse server. PROVE IT, you say. Hit localhost:8080, and you’ll see the Concourse dashboard.


Besides this UX, we interface with Concourse via a CLI tool called fly. I downloaded it from here. I then used fly to add my local environment as a “target” to manage. Instead of plugging in the whole URL every time I interacted with Concourse, I created an alias (“rs”) using fly -t rs login -c http://localhost:8080. If you get a warning to sync your version of fly with your version of Concourse, just enter fly -t rs sync and it gets updated. Neato.

Next up? The pipeline. Pipelines are defined in YAML and are made up of resources and jobs. One of the great things about a declarative model, is that I can run my CI tests against any Concourse by just passing in this (source-controlled) pipeline definition. No point-and-ciick configurations, no prerequisite components to install. Love it. First up, I defined a couple resources. One was my GitHub repo, the second was my target Cloud Foundry environment. In the real world, you’d externalize the Cloud Foundry credentials, and call out to files to build the app, etc. For your benefit, I compressed to a single YAML file.

- name: seroter-source
  type: git
    branch: master
- name: pcf-on-aws
  type: cf
    skip_cert_check: false
    username: XXXXX
    password: XXXXX
    organization: seroter-dev
    space: development

Those resources tell Concourse where to get the stuff it needs to run the jobs. The first job used the GitHub resource to grab the source code. Then it used the Microsoft-provided Docker image to run the dotnet test command.

- name: aspnetcore-unit-tests
    - get: seroter-source
      trigger: true
    - task: run-tests
      privileged: true
        platform: linux
        - name: seroter-source
            type: docker-image
              repository: microsoft/aspnetcore-build
            path: sh
            - -exc
            - |
                cd ./seroter-source
                dotnet restore
                dotnet test

Concourse isn’t really a CD tool, but it does a nice basic job of getting code to a defined destination. The second job deploys the code to Cloud Foundry. It also uses the source code resource and only fires if the test job succeeds. This ensures that only fully-tested code makes its way to the hosting environment. If I were being more responsible, I’d take the results of the test job, drop it into an artifact repo, and then use that artifact for deployment. But hey, you get the idea!

- name: aspnetcore-unit-tests
- name: deploy-to-prod
    - get: seroter-source
      trigger: true
      passed: [aspnetcore-unit-tests]
    - put: pcf-on-aws
        manifest: seroter-source/manifest.yml

That was it! I was ready to deploy the pipeline (pipeline.yml) to Concourse. From the Terminal, I executed fly -t rs set-pipeline -p test-pipeline -c pipeline.yml. Immediately, I saw my pipeline show up in the Concourse Dashboard.


After I unpaused my pipeline, it fired up automatically.


Remember, my job specified a Microsoft-provided container for building the app. Concourse started this job by downloading the Docker image.


After downloading the image, the job kicked off the dotnet test command and confirmed that all my tests passed.


Terrific. Since my next job was set to trigger when the first one succeeded, I immediately saw the “deploy” job spin up.


This job knew how to publish content to Cloud Foundry, and used the provided parameters to deploy the app in a few seconds. Note that there are other resource types if you’re not a Cloud Foundry user. Nobody’s perfect!


The pipeline run was finished, and I confirmed that the app was actually deployed.


Finished? Yes, but I wanted to see a failure in my pipeline! So, I changed my xUnit tests and defined inline data that wouldn’t pass. After committing code to GitHub, my pipeline kicked off automatically. Once again it was tested in the pipeline, and this time, failed. Because it failed, the next step (deployment) didn’t happen. Perfect.


If you’re looking for a CI tool that people actually like using, check out Concourse. Regardless of what you use, focus your energy on getting (all?) apps on pipelines. You don’t do it because you have to ship software every hour, as most apps don’t need it. It’s about shipping whenever you need to, with no drama. Whether you’re adding features or patching vulnerabilities, having pipelines for your apps means you’re actually becoming a customer-centric, software-driven company.


Categories: .NET, ASP.NET Web API, Cloud, Cloud Foundry, DevOps, Docker, General Architecture, Microservices, OSS, Pivotal

Here’s where you’ll find me over the Summer

Here’s where you’ll find me over the Summer

I mean, you’ll mainly find me in Seattle, where I actually live. But, I’m also speaking on a variety of topics at a few shows over the next few months, and thought I’d point those out.

If the thing that connects your other things together isn’t resilient, you’re in trouble. In this talk, I’ll take a look at some core availability patterns for application/data integration, and then review how to configure Azure’s integration services for high availability. This is always a terrific show with compelling speakers, and the hosts at BizTalk360 always do a bang-up job putting it on.

I know a few things about product ownership, such as how to be a good product owner, and a bad one. Primarily because I’ve been both. In this talk at the “big Agile” show, I’ll look at the role and what a product owner should do. I’m starting to work on this presentation now, and will likely list 10+ things that you should do, and a few things to avoid. This will be my second time speaking at this conference, and I enjoy hearing from so many folks focused on software teams and getting code to production.

I’ve been geeking out on space exploration books and movies lately, and thought it’d be fun to translate the lessons from an iconic NASA mission to the everyday challenges faced by software engineers. Here, I’ll show how some of the key ideas applied by NASA engineers reinforce some of the best practices when designing complex software systems. I didn’t attend the inaugural edition of this conference last year, but I’m jazzed to be part of it this time around.

I hope I’ll see you at some of these! If you’ll be at any one of them (or all three, if you’re my stalker!), do let me know.


Categories: BizTalk, Cloud, DevOps, General Architecture, Messaging, Microservices, Microsoft Azure

How to use the Kafka interface of Azure Event Hubs with Spring Cloud Stream

How to use the Kafka interface of Azure Event Hubs with Spring Cloud Stream

When I think of the word “imposter” my mind goes to movies where the criminal is revealed after their disguise is removed. But imposters don’t have to be evil geniuses. Sometimes imposters are good. You may buy generic types of pharmaceuticals or swap out beef for a meat-less hamburger. The goal is to get the experience of what you’re after, but through secondary means. In that sense, Azure Event Hubs is a fantastic imposter.

Now to be sure, Azure Event Hubs is a terrific standalone cloud service. If you need to reliably ingest tons of events in a scalable way, there aren’t many (any?) better options.

Microsoft’s gotten into the habit of putting facades onto their cloud services to turn them into credible imposters. For instance, Azure Cosmos DB has its own native interface, but also ones that mimic MongoDB and Apache Cassandra. Azure Event Hubs got into the action by recently adding an Apache Kafka interface. If you have apps or tools that use those interfaces, it’s much easier to adopt the Azure “equivalents” now. Azure isn’t actually offering MongoDB, Cassandra, or Kafka, but their first party services resemble them enough that you don’t have to change code to get the benefits of Azure’s global scale. Good strategy.

Java developers everywhere use Spring Cloud Stream to talk to popular message brokers like RabbitMQ and Apache Kafka. I thought it’d be fun to see if Spring Cloud Stream “just works” with the new Event Hubs interface. LET’S SEE WHAT HAPPENS.

Creating our Azure Event Hub

I started in the Microsoft Azure portal. From there, I chose to add a new Event Hubs namespace which holds all my actual Event Hubs. For kicks, I chose the “basic” pricing tier which allows a single consumer group and a hundred connections. I set the “enable Kafka” flag and configured three throughput units (each Event Hubs partition scales to a maximum of one throughput unit).


Once I had an Event Hubs namespace, I added an Event Hub to it. I gave the Event Hub a name, and three partitions to spread the data. If I had chosen a plan besides “basic”, I would have also been able to set the message retention period beyond one day.


That’s it. Might be the simplest possible way to create a Kafka-like service!

Spring Boot project setup

The whole point of Spring Boot is to eliminate boilerplate code and make it easier to focus on building robust apps. For example, you don’t want to mess with all that broker-specific logic when you want to pass messages or events around. Spring Boot and Spring Cloud Stream make it straightforward.

To get going, I went to Devs create over 800,000 projects per month from here, so I added two more to the mix. My first project (source code here) added Spring Cloud Stream and Web dependencies. This is my message producer. Then I created a second project (source code here) with the Spring Cloud Stream dependency. This one acts as my message consumer.


This is a Maven project, so I added one more dependency directly to the pom.xml file. This one tells my project to activate the Kafka-specific objects upon application startup.

Configuring the producer

With my projects created, it was time to configure them. First up, the producer.

I made this one simple for demo purposes. First, I added Spring Boot annotations on the primary class. The @RestController one exposes annotated operations as HTTP endpoints. The second annotation was @EnableBinding(Source.class) which set this up as a Spring Cloud Stream object that used channels identified in the default “Source” class.

Next, I autowired a Source object that gets autoconfigured by the Spring Boot process at startup. Finally, I defined a method that responds to HTTP POST requests and sends a message to the “output” channel. This is where the message is sent to Event Hubs, but notice that my code is completely unaware of that fact.

public class EventHubsKafkaApplication {

  public static void main(String[] args) {, args);

  Source mySource;

  @RequestMapping(method=RequestMethod.POST, path="/")
  public String PublishMessage(@RequestBody String company) {

    return "success";

That’s all the code I needed to test this out. All that was left for my producer was to configure the file. This lets me set some of the properties needed to connect to my message broker. Before setting them, I went back to the Microsoft Azure portal and grabbed the connection string for my Event Hub.

With that connection string in hand, I set up my file. First, I defined my brokers. In this case, it’s the fully qualified domain name of my Event Hubs namespace. Next I set the channel destination, in this case, the Event Hub named “eh1.” Finally, I configured my security settings, which includes the connection string. required username="$ConnectionString" password="Endpoint=sb://;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=";

With that, the producer app was done.

Configuring the consumer

The consumer app is even simpler! Its main class also has an @EnableBinding annotation, and this one binds to the channels in the default Sink class. Then, it makes use of the very handy @StreamListener which grabs data from the sink and handles content negotiation.

public class EventHubsKafkaConsumerApplication {

  public static void main(String[] args) {, args);

  public void logMessages(String msg) {
    System.out.println("message is: " + msg);

This was all the code needed to talk to a message broker or event stream processor and do something when a message arrives. Amazing.

The application properties for the consumer are virtually identical to the producer. The only difference is the second property that refers to the input channel, versus the producer that refers to the output channel. required username="$ConnectionString" password="Endpoint=sb://;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=";

With that, I had two working apps.

Testing everything

I first ran a simple test. This involved starting up both the producer and the consumer apps. The producer noticed that I had three partitions, and handled accordingly. The consumer recognized the three partitions as well, and joined an anonymous consumer group (as we didn’t specify one).


I kicked up Postman and posted a JSON message to the REST endpoint exposed by my app. I sent in messages with company names of company-1, company-2, and company-3. You can see here that they show up in my consumer app!


To make sure this wasn’t some other witchcraft going on, I checked the Microsoft Azure portal, and sure enough, see the connections and messages counted.


Simple, right?

BONUS: Messing with Kafka Consumer Groups

I wanted to try something else, too. Kafka offers “consumer groups” where a single consumer instance within the group gets the message. This lets you load balance by having multiple instances of your consumer app, without each one getting a copy of the message. Every consumer group gets the message, so the same message may get read many times by different consumer groups. Azure Event Hubs has the same concept, and it behaves the same way. Spring Cloud Stream also offers this abstraction, even when the underlying broker (e.g. RabbitMQ) doesn’t offer it.

I could set the consumer group property directly in the file. Just add “” to it. But I wanted to pass this in at application startup so that I could start up multiple instances of the app with different consumer group values.

I started up an instance of my consumer (java -jar event-hubs-kafka-consumer-0.0.1-SNAPSHOT.jar – and passed in that property. Notice that it read the full log (because this is the first time the consumer group saw it), and the consumer group is indicated in the log.


Interestingly, when I started a second instance, it didn’t grab what was already read by the consumer group (expected), but when I sent in three more events, BOTH instances got a copy (not expected). Saw some other weirdness when I set the “partitioned=true” on the consumer, and provided an “instanceIndex” number for each consumer app instance.


What’s ALSO interesting is that even though I set up an Event Hubs namespace for a single consumer group, I can apparently create as many as I want with the Kafka interface. Here, I started up another instance of my consumer, but with a different consumer group ID (java -jar event-hubs-kafka-consumer-0.0.1-SNAPSHOT.jar –, and it also read the full log, as you’d expect from a new consumer group. In fact, I created two new ones (app2, app3) and both worked.


So, it seems like the consumer group limits aren’t applying here. I checked the consumer groups in Azure to see if these were being created behind the scenes, but using the Azure API, I still only saw the $Default one there. I have no idea where the Kafka consumer groups show up but they’re clearly in place. Otherwise, I wouldn’t have seen the correct behavior as each new consumer group came online!


I’ll chalk it up to one of two possible things: (1) I’m a mediocre programmer so I probably screwed something up, or (2) it’s an alpha product and everything might not be wired up just yet. Or both!

Regardless, it’s VERY simple to try out a Kafka-compatible interface on a cloud-hosted service thanks to Azure Event Hubs and Spring Cloud Stream. Kafka users should keep an eye on Azure Event Hubs as a legit option for a cloud-hosted event stream processor.


Categories: Cloud, Microservices, Microsoft Azure, OSS, Pivotal, Spring

Creating an Azure VM Scale Set from a legacy, file-sharing, ASP.NET app

Creating an Azure VM Scale Set from a legacy, file-sharing, ASP.NET app

In an ideal world, all your apps have good test coverage, get deployed continuously via pipelines, scale gracefully, and laugh in the face of component failure. That is decidedly not the world we live in. Yes, cloud-native apps are the goal for many, but that’s not what most people have stashed in their data center. Can those apps take some advantage of cloud platforms? For example, what if I had a classic ASP.NET Web Forms app that depends on local storage, but needs better scalability? I could refactor the app—and that might be the right thing to do—or do my best to take advantage of VM-level scaling options in the public cloud. In this demo, I’ll take the aforementioned app, and get it running Azure VM Scale Sets without any code changes.

I’ve been messing with Azure VM Scale Sets as part of a new Pluralsight course that I’m almost done building. The course is all about creating highly-available architectures on Microsoft Azure. Scale Sets make it easy to build and manage fleets of identical virtual machines. In our case here, I want to take an ASP.NET app and throw it into a Scale Set. This exercise requires four steps:

  1. Create and configure a Windows virtual machine in Microsoft Azure. Install IIS, deploy the app, and make sure everything works.
  2. Turn the virtual machine into an image. Sysprep the machine and create an image in Azure for the Scale Set to use.
  3. Create the Azure VM Scale Set. Run a command, watch it go. Configure the load balancer to route traffic to the fleet.
  4. Create a custom extension to update the configuration on each server in the fleet. IIS gets weird on sysprep, so we need Azure to configure each existing (and new) server.

Ok, let’s do this.

Step 1: Create and configure a Windows virtual machine in Microsoft Azure.

While I could take a virtual machine from on-premises and upload it, let’s start from scratch and build a fresh environment.

First off, I went to the Microsoft Azure portal and initiated the build of a new Windows Server VM.


After filling out the required fields and triggering the build, I had a snazzy new VM after a few minutes. I clicked the “connect” button on the portal to get a local RDP file with connection details.


Before connecting the VM, I needed to set up a file share. This ASP.NET app reads files from a file location, then submits the content to an endpoint. If the app uses local storage, then that’s a huge problem for scalability. If that VM disappears, so does the data! So we want to use a durable network file share that a bunch of VMs can share. Fortunately, Azure has such a service.

I went into the Azure Portal and provisioned a new storage account, and then set up the file structure that my app expects.


How do I get my app to use this? My ASP.NET app gets its target file location from a configuration property in its web.config file. No need to chase down source code to use a network file share instead of local storage! We’ll get to that shortly.

With my storage set up, I proceeded to connect to my virtual machine. Before starting the RDP session, I added a link to my local machine so that I could transfer the app’s code to the server.


Once connected, I proceeded to install the IIS web server onto the box. I also made sure to add ASP.NET support to the web server, which I forget to do roughly 84% of the time.


Now I had a web server ready to go. Next up? Copying files over. Here, I just took content from a local folder and put it into the wwwroot folder on the server.


My app was almost ready to go, but I still needed to update the web.config to point to my Azure file storage.


Now, how does my app authenticate with this secure file share? There’s a few ways you could try and do it. I chose to create a local user with access to the file share, and run my web app in an application pool acting as that user. That user was named seroterpluralsight.


What are the credentials? The name of the user should be the name of the Azure storage account, and the user’s password is the account key.


Finally, I created a new IIS application pool (pspool) and set the identity to the serverpluralsight user.


With that, I started up the app, and sure enough, was able to browse the network file share without any issue.


Step 2: Turn the virtual machine into an image

The whole point of a Scale Set is that I have a scalable set of uniform servers. When the app needs to scale up, Azure just adds another identical server to the pool. So, I need a template!

Note: There are a couple ways to approach this feature. First, you could just build a Scale Set from a generic OS image, and then bootstrap it by running installers to prepare it for work. This means you don’t have to build and maintain a pre-built image. However, it also means it takes longer for the new server to become a useful member of the pool. Bootstrapping or pre-building images are both valid options. 

To create a template from a Windows machine, I needed to sysprep it. Doing this removes lots of user specific things, including mapped drives. So while I could have created a mapped drive from Azure File Storage and accessed files from the ASP.NET app that way, the drive goes away when I sysprep. I decided to just access the file share via the network path and not deal with a mapped drive.


With the machine now generalized and shut down, I returned to the Azure Portal and clicked the “capture” button. This creates an Azure image from the VM and (optionally) destroys the original VM.


Step #3: Create the Azure VM Scale Set

I now had everything needed to build the Scale Set. If you’re bootstrapping a server (versus using a pre-built image) you can create a Scale Set from the Azure Portal. Since I am using a pre-built image, I had to dip down to the CLI. To make it more fun, I used the baked-in Azure Cloud Shell instead of the console on my own machine. Before crafting the command to create the Scale Set, I grabbed the ID of the VM template. You can get this by copying the Resource ID from the Azure image page on the Portal.


With that ID, I put together the command for instantiating the Scale Set.

az vmss create -n psvmss -g pluralsight-practice --instance-count 2 --image /subscriptions/[subscription id]/resourceGroups/pluralsight-practice/providers/Microsoft.Compute/images/[image id] --authentication-type password --admin-username legacyuser --admin-password [password] --location eastus2 --upgrade-policy-mode Automatic --load-balancer ps-loadbalancer --backend-port 3389

Let’s unpack that. I specified a name for my Scale Set (“psvmss”) told it which resource group to add this to (“pluralsight-practice”), set a default number of VM instances, pointed it to my pre-built image, set password authentication for the VMs and provided credentials, set the geographic location, told the Scale Set to automatically apply changes, and defined a load balancer (“ps-loadbalancer”). After a few minutes, I had a Scale Set.


Neato. Once that Scale Set is in place, I could still RDP into individual boxes, but they’re meant to be managed as a fleet.

Step #4: Create a custom extension to update the configuration on each server in the fleet.

As I mentioned earlier, we’re not QUITE done yet. When you sysprep a Windows box that has an IIS app pool with a custom user, the server freaks out. Specifically, it still shows that user as the pool’s identity, but the password gets corrupted. Seems like a known thing. I could cry about it, or do something to fix it. Fortunately, Azure VMs (and Scale Sets) have the idea of “custom script extensions.” These are scripts that can apply to one or many VMs. In my case, what I needed was a script that reset the credentials of the application pool user.

First, I created a new Powershell script (“config-app-pool.ps1”) that set the pool’s identity.

Import-Module WebAdministration

Set-ItemProperty IIS:AppPoolspspool -name processModel -value @{userName="seroterpluralsight"; password="[password]";identitytype=3}

I uploaded that file to my Azure Storage account. This gives me a storage location that the Scale Set can use to retrieve these settings later.

Next, I went back to the Cloud Shell to create couple local files used by the extension command. First, I created a file called public-settings.json that stored the location of the above Powershell script.


"fileUris": [""]


Then I created a protected-settings.json file. These values get encrypted are only decrypted on the VM when the script runs.


"commandToExecute": "powershell -ExecutionPolicy Unrestricted -File config-app-pool.ps1", "storageAccountName": "seroterpluralsight", "storageAccountKey": "[account key]"


That file tells the extension what to actually do with the file it downloaded from Azure Storage, and what credentials to use to access Azure Storage.

Ok, now I could setup the extension. Once the extension is in place, it applies to every VM in the Scale Set now, or in the future.

az vmss extension set --resource-group pluralsight-practice --vmss-name psvmss --name customScriptExtension --publisher Microsoft.Compute --settings ./public-settings.json --protected-settings ./protected-settings.json

Note that if you’re doing this against Linux boxes, the “name” and “publisher” have different values.

That’s pretty much it. Once i extended the generated load balancer with rules to route on port 80, I had everything I needed.


After pinging the load balanced URL, I saw my “legacy” ASP.NET application served up from multiple VMs, all with secure access to the same file share. Terrific!


Long term, you’ll be better off refactoring many of your apps to take advantage of what the cloud offers. A straight up lift-and-shift often resembles transferring debt from one credit card to another. But, some apps don’t need many changes at all to get some incremental benefits from cloud, and Scale Sets could be a useful route for you.

2017 in Review: Reading and Writing Highlights

2017 in Review: Reading and Writing Highlights

kid-3What a fun year. Lots of things to be grateful for. Took on some more responsibility at Pivotal, helped put on a couple conferences, recorded a couple dozen podcast episodes, wrote news/articles/eMags for, delivered a couple Pluralsight courses (DevOps, and Java related), received my 10th straight Microsoft MVP award, wrote some blog posts, spoke at a bunch of conferences, and added a third kid to the mix.

Each year, I like to recap some of the things I enjoyed writing and reading. Enjoy!

I swear that I’m writing as much as I ever have, but it definitely doesn’t all show up in one place anymore! Here are a few things I churned out that made me happy.

I plowed through thirty four books this year, mostly on my wonderful Kindle. As usual, I choose a mix of biographies, history, sports, religion, leadership, and mystery/thriller. Here’s a handful of the ones I enjoyed the most.

  • Apollo 8: The Thrilling Story of the First Mission to the Moon, by Jeffrey Kluger (@jeffreykluger). Brilliant storytelling about our race to the moon. There was a perfect mix of character backstory, science, and narrative. Really well done.
  • Boyd: The Fighter Pilot Who Changed the Art of War, by Robert Coram (@RobertBCoram). I had mixed feelings after finishing this. Boyd’s lessons on maneuverability are game-changing. His impact on the world is massive. But this well-written story also highlights a man obsessed; one who grossly neglected his family. Important book for multiple reasons.
  • The Game: Inside the Secret World of Major League Baseball’s Power Brokers, by Jon Pessah (@JonPessah). Gosh, I love baseball books. This one highlights the Bud Selig era as commissioner, the rise of steroid usage, complex labor negotiations, and the burst of new stadiums. Some amazing behind-the-scenes insight here.
  • Not Forgotten: The True Story of My Imprisonment in North Korea, by Kenneth Bae. One might think that an American held in captivity by North Koreans longer than anyone since the Korean War would be angry. Rather, Bae demonstrates sympathy and compassion for people who aren’t exposed to a better way. Good story.
  • Shoe Dog: A Memoir by the Creator of Nike, by Phil Knight (@NikeUnleash). I went and bought new Nikes after this. MISSION ACCOMPLISHED PHIL KNIGHT. This was a fantastic book. Knight’s passion and drive to get Blue Ribbon (later, Nike) off the ground was inspiring. People can create impactful businesses even if they don’t feel an intense calling, but there’s something special about those that do.
  • Dynasty: The Rise and Fall of the House of Cesar, by Tom Holland (). This is somewhat of a “part 2” from Holland’s previous work. Long, but engaging, this book tells the tale of the first five emperors. It’s far from a dry history book, as Holland does a admirable job weaving specific details into an overarching story. Books like this always remind me that nothing happens in politics today that didn’t already happen thousands of years ago.
  • Avenue of Spies: A True Story of Terror, Espionage, and One American Family’s Heroic Resistance in Nazi-Occupied Paris, by Alex Kershaw (). Would you protect the most vulnerable, even if your life was on the line as a result? Many during WWII faced that choice. This book tells the story of one family’s decision, the impact they had, and the hard price they paid.
  • Stalling for Time: My Life as an FBI Hostage Negotiator, by Gary Noesner. Fascinating book that explains the principles of hostage negotiation, but also lays out the challenge of introducing it to an FBI conditioned to respond with force. Lots of useful nuggets in here for people who manage complex situations and teams.
  • The Things Our Fathers Saw: The Untold Stories of the World War II Generation from Hometown, USA, by Matthew Rozell (). Intensely personal stories from those who fought in WWII, with a focus on the battles in the Pacific. Harrowing, tragic, inspiring. Very well written.
  • I Don’t Have Enough Faith to Be an Atheist, by Norman Geisler () and Frank Turek (). Why are we here? Where did we come from? This book outlines the beautiful intersection of objective truth, science, philosophy, history, and faith. It’s a compelling arrangement of info.
  • The Late Show, by Michael Connelly (). I’d read a book on kangaroo mating rituals if Connelly wrote it. Love his stuff. This new cop-thriller introduced a multi-dimensional lead character. Hopefully Connelly builds a new series of books around her.
  • The Toyota Way: 14 Management Principles from the World’s Greatest Manufacturer, by Jeffrey Liker. Ceremonies and “best practices” don’t matter if you have the wrong foundation. Liker’s must-read book lays out, piece by piece, the fundamental principles that help Toyota achieve operational excellence. Everyone in technology should read this and absorb the lessons. It puts weight behind all the DevOps and continuous delivery concepts we debate.
  • One Mission: How Leaders Build a Team of Teams, by Chris Fussell (). I read, and enjoyed, Team of Teams last year. Great story on the necessity to build adaptable organizations. The goal of this book is to answer *how* you create an adaptable organization. Fussell uses examples from both military and private industry to explain how to establish trust, create common purpose, establish a shared consciousness, and create spaces for “empowered execution.”
  • Win Bigly: Persuasion in a World Where Facts Don’t Matter, by Scott Adams (). What do Obama, Steve Jobs, Madonna, and Trump have in common? Remarkable persuasion skills, according to Adams. In his latest book, Adams deconstructs the 2016 election, and intermixes a few dozen persuasion tips you can use to develop more convincing arguments.
  • Value Stream Mapping: How to Visualize Work and Align Leadership for Organizational Transformation, by Karen Martin () and Mike Osterling (). How does work get done, and are you working on things that matter? I’d suspect that most folks in IT can’t confidently answer either of those questions. That’s not the way IT orgs were set up. But I’ve noticed a change during the past year+, and there’s a renewed focus on outcomes. This book does a terrific job helping you understand how work flows, techniques for mapping it, where to focus your energy, and how to measure the success of your efforts.
  • The Five Dysfunctions of a Team, by Patrick Lencioni (). I’ll admit that I’m sometimes surprised when teams of “all stars” fail to deliver as expected. Lencioni spins a fictitious tale of a leader and her team, and how they work through the five core dysfunctions of any team. Many of you will sadly nod your head while reading this book, but you’ll also walk away with ideas for improving your situation.
  • Setting the Table: The Transforming Power of Hospitality in Business, by Danny Meyer (). How does your company make people feel? I loved Meyer’s distinction between providing a service and displaying hospitality in a restaurant setting, and the lesson is applicable to any industry. A focus on hospitality will also impact the type of people you hire. Great book that that leaves you hungry and inspired.
  • Extreme Ownership: How U.S. Navy SEALs Lead and Win, by Jocko Willink () and Leif Babin (). As a manager, are you ready to take responsibility for everything your team does? That’s what leaders do. Willink and Babin explain that leaders take extreme ownership of anything impacting their mission. Good story, with examples, of how this plays out in reality. Their advice isn’t easy to follow, but the impact is undeniable.
  • Strategy: A History, by Sir Lawrence Freedman (). This book wasn’t what I expected—I thought it’d be more about specific strategies, not strategy as a whole. But there was a lot to like here. The author looks at how strategy played a part in military, political, and business settings.
  • Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity, by Kim Scott (). I had a couple hundred highlights in this book, so yes, it spoke to me. Scott credibly looks at how to guide a high performing team by fostering strong relationships. The idea of “radical candor” altered my professional behavior and hopefully makes me a better boss and colleague.
  • The Lean Startup: How’s Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses, by Eric Ries (). A modern classic, this book walks entrepreneurs through a process for validated learning and figuring out the right thing to build. Ries sprinkles his advice with real-life stories as proof points, and offers credible direction for those trying to build things that matter.
  • Hooked: How to Build Habit-Forming Products, by Nir Eyal (). It’s not about tricking people into using products, but rather, helping people do things they already want to do. Eyal shares some extremely useful guidance for those building (and marketing) products that become indispensable.
  • The Art of Action: How Leaders Close the Gap between Plans, Actions, and Results, by Stephen Bungay. Wide-ranging book that covers a history of strategy, but also focuses on techniques for creating an action-oriented environment that delivers positive results.

Thank you all for spending some time with me in 2017, and I look forward to learning alongside you all in 2018.


Categories: General Architecture, .NET, Cloud, Cloud Foundry, Microsoft Azure, Messaging, OSS, Microservices, Pivotal, Spring

Can’t figure out which SpringOne Platform sessions to attend? I’ll help you out.

Can’t figure out which SpringOne Platform sessions to attend? I’ll help you out.

Next week is SpringOne Platform (S1P). This annual conference is where developers from around the world learn about about Spring, Cloud Foundry, and modern architecture. It’s got a great mix of tech talks, product demos, and transformational case studies. Hear from software engineers and leaders that work at companies like Pivotal, Boeing, Mastercard, Microsoft, Google, FedEx, HCSC, The Home Depot, Comcast, Accenture, and more.

If you’re attending (and you are, RIGHT?!?), how do you pick sessions from the ten tracks over three days? I helped build the program, and thought I’d point out the best talks for each type of audience member.

The “multi-cloud enthusiast”

Your future involves multiple clouds. It’s inevitable. Learn all about the tech and strategies to make it more successful.

The “bleeding-edge developer”

The vast major of S1P attendees are developers who want to learn about the hottest technologies. Here are some highlights for them.

The “enterprise change agent”

I’m blown away by the number of real case studies at this show. If you’re trying to create a lasting change at your company, these are the talks that prep you for success.

The “ambitious operations pro”

Automation doesn’t spell the end of Ops. But it does change the nature of it. These are talks that forward-thinking operations folks want to attend to learn how to build and manage future tech.

The “modern app architect”

What a fun time to be an architect! We’re expected to deliver software with exceptional availability and scale. That requires a new set of patterns. You’ll learn them in these talks.

The “curious data pro”

How we collect, process, store, and retrieve data is changing. It has to. There’s more data, in more formats, with demands for faster access. These talks get you up to speed on modern data approaches.

The “plugged-in manager”

Any engineering lead, manager, or executive is going to spend considerable time optimizing the team, not building software. But that doesn’t mean you shouldn’t be up-to-date on what your team is working with. These talks will make you sound hip at the water cooler after the conference.

Fortunately all these sessions will be recorded and posted online. But nothing beats the in-person experience. If you haven’t bought a ticket, it’s not too late!

Trying Out Microsoft’s Spring Boot Starters

Trying Out Microsoft’s Spring Boot Starters

Do you build Java apps? If so, there’s a good chance you’re using Spring Boot. Web apps, event-driven apps, data processing apps, you name it, Spring Boot has libraries that help. It’s pretty great. I’m biased; I work for the company that maintains Spring, and taught two Pluralsight courses about Spring Cloud. But there’s no denying the momentum:

If a platform matters, it works with Spring Boot. Add Microsoft Azure to the list. Microsoft and Pivotal engineers created some Spring Boot “starters” for key Azure services. Starters make it simple to add jars to your classpath. Then, Spring Boot handles the messy dependency management for you. And with built-in auto-configuration, objects get instantiated and configured automatically. Let’s see this in action. I built a simple Spring Boot app that uses these starters to interact with Azure Storage and Azure DocumentDB (CosmosDB). 

I started at Josh Long’s second favorite place on the Internet: Here, you can bootstrap a new project with all sorts of interesting dependencies, including Azure! I defined my app’s group and artifact IDs, and then chose three dependencies: web, Azure Storage, and Azure Support. “Azure Storage” brings the jars in for storage, and “Azure Support” activates other Azure services when you reference their jars.


I downloaded the resulting project and opened it in Spring Tool Suite. Then I added one new starter to my Maven POM file:


That’s it. From these starters, Spring Boot pulls in everything our app needs. Next, it was time for code. This basic REST service serves up product recommendations. I wanted to store each request for recommendations as a log file in Azure Storage and a record in DocumentDB. I first modeled a “recommendations” that goes into DocumentDB. Notice the topmost annotation and reference to a collection.

package seroter.demo.bootazurewebapp;

public class RecommendationItem {

        private String recId;
        private String cartId;
        private String recommendedProduct;
        private String recommendationDate;
        public String getCartId() {
                return cartId;
        public void setCartId(String cartId) {
                this.cartId = cartId;
        public String getRecommendedProduct() {
                return recommendedProduct;
        public void setRecommendedProduct(String recommendedProduct) {
                this.recommendedProduct = recommendedProduct;
        public String getRecommendationDate() {
                return recommendationDate;
        public void setRecommendationDate(String recommendationDate) {
                this.recommendationDate = recommendationDate;
        public String getRecId() {
                return recId;
        public void setRecId(String recId) {
                this.recId = recId;

Next I defined an interface that extends DocumentDbRepository.

package seroter.demo.bootazurewebapp;
import org.springframework.stereotype.Repository;
import seroter.demo.bootazurewebapp.RecommendationItem;

public interface RecommendationRepo extends DocumentDbRepository<RecommendationItem, String> {

Finally, I build the REST handler that talks to Azure Storage and Azure DocumentDB.  Note a few things. First, I have a pair of autowired variables. These reference beans created by Spring Boot and injected at runtime. In my case, they should be objects that are already authenticated with Azure and ready to go.

public class BootAzureWebAppApplication {
        public static void main(String[] args) {
, args);

        //for blob storage
        private CloudStorageAccount account;

        //for Cosmos DB
        private RecommendationRepo repo;

In the method that actually handles the HTTP POST, I first referenced the Azure Storage Blob containers and add a file there. I got to use the autowired CloudStorageAccount here. Next, I created a RecommendationItem object and loaded it into the autowired DocumentDB repo. Finally, I returned a message to the caller.

@RequestMapping(value="/recommendations", method=RequestMethod.POST)
public String GetRecommendedProduct(@RequestParam("cartId") String cartId) throws URISyntaxException, StorageException, IOException {

        //create log file and upload to an Azure Storage Blob
        CloudBlobClient client = account.createCloudBlobClient();
        CloudBlobContainer container = client.getContainerReference("logs");

        String id = UUID.randomUUID().toString();
        String logId = String.format("log - %s.txt", id);
        CloudBlockBlob blob = container.getBlockBlobReference(logId);
        //create the log file and populate with cart id

        //add to DocumentDB collection (doesn't have to exist already)
        RecommendationItem r = new RecommendationItem();
        r.setRecommendationDate(new Date().toString());;

        return "Happy Fun Ball (Y777-TF2001)";

Excellent. Next up, creating the actual Azure services! From the Azure Portal, I created a new Resource Group called “boot-demos.” This holds all the assets related to this effort. I then added an Azure Storage account to hold my blobs.


Next, I grabbed the connection string to my storage account.


I took that value, and added it to the file in my Spring Boot app.;AccountName=bootapplogs;AccountKey=[KEY VALUE]<span                           data-mce-type="bookmark"                              id="mce_SELREST_start"                                data-mce-style="overflow:hidden;line-height:0"                                style="overflow:hidden;line-height:0"                         ></span>;

Since I’m also using DocumentDB (part of CosmosDB), I needed an instance of that as well.


Can you guess what’s next? Yes, it’s credentials. Specifically, I needed the URI and primary key associated with my Cosmos DB account.


I snagged those values and also put them into my file.

azure.documentdb.key=[KEY VALUE]

That’s it. Those credentials get used when activating the Azure beans, and my code gets access to pre-configured objects. After starting up the app, I sent in a POST request.


I got back a “recommended product”, but more importantly, I didn’t get an error! When I looked back at the Azure Portal, I saw two things. First, I saw a new log file in my newly created blob container.


Secondly, I saw a new database and document in my Cosmos DB account.


That was easy. Spring Boot apps, consuming Microsoft Azure services with no fuss.

Note that I let my app automatically create the Blob container and DocumentDB database. In real life you might want to create those ahead of time in order to set various properties and not rely on default values.

Bonus Demo – Running this app in Cloud Foundry

Let’s not stop there. While the above process was simple, it can be simpler. What if I don’t want to go to Azure to pre-provision resources? And what if I don’t want to manage credentials in my application itself? Fret not. That’s where the Service Broker comes in.

Microsoft created an Azure Service Broker for Cloud Foundry that takes care of provisioning resources and attaching those resources to apps. I added that Service Broker to my Pivotal Web Services (hosted Cloud Foundry) account.


When creating a service instance via the Broker, I needed to provide a few parameters in a JSON file. For the Azure Storage account, it’s just the (existing or new) resource group, account name, location, and type.

  "resourceGroup": "generated-boot-demo",
  "storageAccountName": "generatedbootapplogs",
  "location": "westus",
  "accountType": "Standard_LRS"

For DocumentDB, my JSON file called out the resource group, account name, database name, and location.

  "resourceGroup": "generated-boot-demo",
  "docDbAccountName": "generatedbootappdocs",
  "docDbName": "recommendations",
  "location": "westus"

Sweet. Now to create the services. It’s just a single command for each service.

cf create-service azure-documentdb standard bootdocdb -c broker-documentdb-config.json

cf create-service azure-storage standard bootstorage -c broker-storage-config.json<span                              data-mce-type="bookmark"                              id="mce_SELREST_start"                                data-mce-style="overflow:hidden;line-height:0"                                style="overflow:hidden;line-height:0"                         ></span>

To prove it worked, I snuck a peek at the Azure Portal, and saw my two new accounts.


Finally, I removed all the credentials from the file, packaged my app into a jar file, and added a Cloud Foundry manifest. This manifest tells Cloud Foundry where to find the deployable asset, and which service(s) to attach to. Note that I’m referencing the ones I just created.

- name: boot-azure-web-app
  memory: 1G
  instances: 1
  path: target/boot-azure-web-app-0.0.1-SNAPSHOT.jar
  - bootdocdb
  - bootstorage

With that, I ran a “cf push” and the app was deployed and started up by Cloud Foundry. I saw that it was successfully bound to each service, and the credentials for each Azure service were added to the environment variables. What’s awesome is that the Azure Spring Boot Starters know how to read these environment variables. No more credentials in my application package. My environments variables for this app in Cloud Foundry are shown here.


I called my service running in Cloud Foundry, and as before, I got a log file in Blob storage and a document in Document DB.

These Spring Boot Starters offer a great way to add Azure services to your apps. They work like any other Spring Boot Starter, and also have handy Cloud Foundry helpers to make deployment of those apps super easy. Keep an eye on Microsoft’s GitHub repo for these starters. More good stuff coming.


Categories: Cloud, Cloud Foundry, Microsoft Azure, Spring

Adding circuit breakers to your .NET applications

Adding circuit breakers to your .NET applications

Apps fail. Hardware fails. Networks fail. None of this should surprise you. As we build more distributed systems, these failures create unpredictability. Remote calls between components might experience latency, faults, unresponsiveness, or worse. How do you keep a failure in one component from creating a cascading failure across your whole environment?

In his seminal book Release It!, Michael Nygard introduced the “circuit breaker” software pattern. Basically, you wrap calls to downstream services, and watch for failure. If there are too many failures, the circuit “trips” and the downstream services isn’t called any longer. Or at least for a period of time until the service heals itself.

How do we use this pattern in our apps? Enter Hystrix from Netflix OSS. Released in 2012, this library executes each call on a separate thread, watches for failures in Java calls, invokes a fallback operation upon failure, trips a circuit if needed, and periodically checks to see if the downstream service is healthy. And it has a handy dashboard to visualize your circuits. It’s wicked. The Spring team worked with Netflix and created a easy-to-use version for Spring Boot developers. Spring Cloud Hystrix is the result. You can learn all about it in my most recent Pluralsight course.

But why do Java developers get to have all the fun? Pivotal released an open-source library called Steeltoe last year. This library brings microservices patterns to .NET developers. It started out with things like a Git-backed configuration store, and service discovery. The brand new update offers management endpoints and … an implementation of Hystrix for .NET apps. Note that this is for .NET Framework OR .NET Core apps. Everybody gets in on the action.

Let’s see how Steeltoe Hystrix works. I built an ASP.NET Core service, and than called it from a front-end app. I wrapped the calls to the service using Steeltoe Hystrix, which protects my app when failures occur.

Dependency: the recommendation service

This service returns recommended products to buy, based on your past purchasing history. In reality, it returns four products that I’ve hard-coded into a controller. LOWER YOUR EXPECTATIONS OF ME.

This is an ASP.NET Core MVC Web API. The code is in GitHub, but here’s the controller for review:

namespace core_hystrix_recommendation_service.Controllers
    public class RecommendationsController : Controller
        // GET api/recommendations
        public IEnumerable<Recommendations> Get()
            Recommendations r1 = new Recommendations();
            r1.ProductId = "10023";
            r1.ProductDescription = "Women's Triblend T-Shirt";
            r1.ProductImage = "";

            Recommendations r2 = new Recommendations();
            r2.ProductId = "10040";
            r2.ProductDescription = "Men's Bring Back Your Weekend T-Shirt";
            r2.ProductImage = "";

            Recommendations r3 = new Recommendations();
            r3.ProductId = "10057";
            r3.ProductDescription = "H2Go Force Water Bottle";
            r3.ProductImage = "";

            Recommendations r4 = new Recommendations();
            r4.ProductId = "10059";
            r4.ProductDescription = "Migrating to Cloud Native Application Architectures by Matt Stine";
            r4.ProductImage = "";

            return new Recommendations[] { r1, r2, r3, r4 };

Note that the dependency service has no knowledge of Hystrix or how the caller invokes it.

Caller: the recommendations UI

The front-end app calls the recommendation service, but it shouldn’t tip over just because the service is unavailable. Rather, bad calls should fail quickly, and gracefully. We could return cached or static results, as an example. Be aware that a circuit breaker is much more than fancy exception handling. One big piece is that each call executes in its own thread. This implementation of the bulkhead patterns prevents runaway resource consumption, among other things. Besides that, circuit breakers are also machinery to watch failures over time, and allow the failing service to recover before allowing more requests.

This ASP.NET Core app uses the mvc template. I’ve added the Steeltoe packages to the project. There are a few Nuget packages to choose from. If you’re running this in Pivotal Cloud Foundry, there’s a set of packages that make it easy to integrate with Hystrix dashboard embedded there. Here, let’s assume we’re running this app somewhere else. That means I need the base package “Steeltoe.CircuitBreaker.Hystrix” and “Steeltoe.CircuitBreaker.Hystrix.MetricsEvents” which gives me a stream of real-time data to analyze.

<Project Sdk="Microsoft.NET.Sdk.Web">
    <PackageReference Include="Microsoft.AspNet.WebApi.Client" Version="5.2.3" />
    <PackageReference Include="Microsoft.AspNetCore.All" Version="2.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="2.0.0" />
    <PackageReference Include="Steeltoe.CircuitBreaker.Hystrix" Version="1.1.0" />
    <PackageReference Include="Steeltoe.CircuitBreaker.Hystrix.MetricsEvents" Version="1.1.0" />
    <DotNetCliToolReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Tools" Version="2.0.0" />

I built a class (“RecommendationService”) that calls the dependent service. This class inherits from HystrixCommand. There are a few ways to use these commands in calling code. I’m adding it to the ASP.NET Core service container, so my constructor takes in a IHystrixCommandOptions.

//HystrixCommand means no result, HystrixCommand<string> means a string comes back
public class RecommendationService: HystrixCommand<List<Recommendations>>
  public RecommendationService(IHystrixCommandOptions options):base(options) {

I’ve got inherited methods to use thanks to the base class. I call my dependent service by overriding Run (or RunAsync). If failure happens, the RunFallback (or RunFallbackAsync) is invoked and I just return some static data. Here’s the code:

protected override List<Recommendations> Run()
  var client = new HttpClient();
  var response = client.GetAsync("http://localhost:5000/api/recommendations").Result;

  var recommendations = response.Content.ReadAsAsync<List<Recommendations>>().Result;

  return recommendations;

protected override List<Recommendations> RunFallback()
  Recommendations r1 = new Recommendations();
  r1.ProductId = "10007";
  r1.ProductDescription = "Black Hat";
  r1.ProductImage = "";

  List<Recommendations> recommendations = new List<Recommendations>();

  return recommendations;

My ASP.NET Core controller uses the RecommendationService class to call its dependency. Notice that I’ve got an object of that type coming into my constructor. Then I call the Execute method (that’s part of the base class) to trigger the Hystrix-protected call.

public class HomeController : Controller
  public HomeController(RecommendationService rs) { = rs;

  RecommendationService rs;

  public IActionResult Index()
    //call Hystrix-protected service
    List<Recommendations> recommendations = rs.Execute();

    //add results to property bag for view
    ViewData["Recommendations"] = recommendations;

    return View();

Last thing? Tying it all together. In the Startup.cs class, I added two things to the ConfigureServices operation. First, I added a HystrixCommand to the service container. Second, I added the Hystrix metrics stream.

// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)

  //add QueryCommand to service container, and inject into controller so it gets config values
  services.AddHystrixCommand<RecommendationService>("RecommendationGroup", Configuration);

  //added to get Metrics stream

In the Configure method, I added couple pieces to the application pipeline.

// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
   if (env.IsDevelopment())



   app.UseMvc(routes =>
       name: "default",
       template: "{controller=Home}/{action=Index}/{id?}");


That’s it. Notice that I took advantage of ASP.NET Core’s dependency injection, and known extensibility points. Nothing unnatural here.

You can grab the source code for this from my GitHub repo.

Testing the circuit

Let’s test this out. First, I started up the recommendation service. Pinging the endpoint proved that I got back four recommended products.


Great. Next I started up the MVC app that acts as the front-end. Loading the page in the browser showed the four recommendations returned by the service.


That works. No big deal. Now let’s turn off the downstream service. Maybe it’s down for maintenance, or just misbehaving. What happens?


The Hystrix wrapper detected a failure, and invoked the fallback operation. That’s cool. Let’s see what Hystrix is tracking in the metrics stream. Just append /hystrix/ to the URL and you get a data stream that’s fully compatible with Spring Cloud Hystrix.


Here, we see a whole bunch of data that Hystrix is tracking. It’s watching request count, error rate, and lots more. What if you want to change the behavior of Hystrix? Amazingly, the .NET version of Hystrix in Steeltoe has the same broad configuration surface that classic Hystrix does. By adding overrides to the appsettings.json file, you can tweak the behavior of commands, the thread pool, and more. In order to see the circuit actually open, I stretched the evaluation window (from 10 to 20 seconds), and reduced the error limit (from 20 to 3). Here’s what that looked like:

"hystrix": {
  "command": {
      "default": {
        "circuitBreaker": {
          "requestVolumeThreshold": 3
        "metrics" : {
          "rollingStats": {
            "timeInMilliseconds" : 20000

Restarting my service shows new threshold in the Hystrix stream. Super easy, and very powerful.


BONUS: Using the Hystrix Dashboard

Look, I like reading gobs of JSON in the browser as much as the next person with too much free time. However, normal people like dense visualizations that help them make decisions quickly. Fortunately, Hystrix comes with an extremely data-rich dashboard that makes it simple to see what’s going on.

This is still a Java component, so I spun up a new project from and added a Hystrix Dashboard dependency to my Boot app. After adding a single annotation to my class, I spun up the project. The Hystrix dashboard asks for a metrics endpoint. Hey, I have one of those! After plugging in my stream URL, I can immediately see tons of info.


As a service owner or operator, this is a goldmine. I see request volumes, circuit status, failure counts, number of hosts, latency, and much more. If you’ve got a couple services, or a couple hundred, visualizations like this are a life saver.


As someone who started out their career as a .NET developer, I’m tickled to see things like this surface. Steeltoe adds serious juice to your .NET apps and the addition of things like circuit breakers makes it a must-have. Circuit breakers are a proven way to deliver more resilient service environments, so download my sample apps and give this a spin right now!


Categories: .NET, Cloud, Microservices, Pivotal, Spring

Surprisingly simple messaging with Spring Cloud Stream

Surprisingly simple messaging with Spring Cloud Stream

You’ve got a lot of options when connecting microservices together. You could use service discovery and make direct calls. Or you might use a shared database to transfer work. But message brokers continue to be a popular choice. These range from single purpose engines like Amazon SQS or RabbitMQ, to event stream processors like Azure Event Hubs or Apache Kafka, all the way to sophisticated service buses like Microsoft BizTalk Server. When developers choose any one of those, they need critical knowledge to be use them effectively. How can you shrink the time to value and help developers be productive, faster? For Java developers, Spring Cloud Stream offers a valuable abstraction.

Spring Cloud Stream offers an interface for developers that requires no knowledge of the underlying broker. That broker, either Apache Kafka or RabbitMQ, gets configured by Spring Cloud Stream. Communication to and from the broker is also done via the Stream library.

What’s exciting to me is that all brokers are treated the same. Spring Cloud Stream normalizes behavior, even if it’s not native to the broker. For example, want a competing consumer model for your clients, or partitioned processing? Those concepts behave differently in RabbitMQ and Kafka. No problem. Spring Cloud Stream makes it work the same, transparently. Let’s actually try both of those scenarios.

Competing consumers through “consumer groups”

By default, Spring Cloud Stream sets up everything as a publish-subscribe relationship. This makes it easy to share data among many different subscribers. But what if you want multiple instances of one subscriber (for scale out processing)? One solution is consumer groups. These don’t behave the same in both messaging brokers. Spring Cloud Stream don’t care! Let’s build an example app using RabbitMQ.

Before writing code, we need an instance of RabbitMQ running. The most dead-simple option? A Docker container for it. If you’ve got Docker installed, the only thing you need to do is run the following command:

-docker run -d –hostname local-rabbit –name demo-rmq -p 15672:15672 -p 5672:5672 rabbitmq:3.6.11-management


After running that, I have a local cache of the image, and a running container with port mapping that makes the container accessible from my host.

How do we get messages into RabbitMQ? Spring Cloud Stream supports a handful of patterns. We could publish on a schedule, or on-demand. Here, let’s build a web app that publishes to the bus when the user issues a POST command to a REST endpoint.

Publisher app

First, build a Spring Boot application that leverages spring-cloud-starter-stream-rabbit (and spring-boot-starter-web). This brings in everything I need to use Spring Cloud Stream, and RabbitMQ as a destination.


Add a new class that acts as our REST controller. A simple @EnableBinding annotation lights this app up as a Spring Cloud Stream project. Here, I’m using the built-in “Source” interface that defines a single communication channel, but you can also build your own.

public class BriefController {

In this controller class, add an @Autowired variable that references the bean that Spring Cloud Stream adds for the Source interface. We can then use this variable to directly publish to the bound channel! Same code whether talking to RabbitMQ or Kafka. Simple stuff.

public class BriefController {
  //refer to instance of bean that Stream adds to container
  Source mysource;

  //take in a message via HTTP, publish to broker
  @RequestMapping(path="/brief", method=RequestMethod.POST)
  public String publishMessage(@RequestBody String payload) {


    //send message to channel

    return "success";

Our publisher app is done, so all that’s left is some basic configuration. This configuration tells Spring Cloud Stream how to connect to the right broker. Note that we don’t have to tell Spring Cloud Stream to use RabbitMQ; it happens automatically by having that dependency in our classpath. No, all we need is connection info to our broker, an explicit reference to a destination (without it, the RabbitMQ exchange would be called “output”), and a command to send JSON.


#rabbitmq settings for Spring Cloud Stream to use

Consumer app

This part’s almost too easy. Here, build a new Spring Boot application and only choose the spring-cloud-starter-stream-rabbit dependency.

In the default class, decorate it with @EnableBinding and use the built-in Sink interface. Then, all that’s left is to create a method to process any messages found in the broker. To do that, we decorate the operation with @StreamListener, and all the content type handling is done for us. Wicked.

public class BlogStreamSubscriberDemoApplication {

  public static void main(String[] args) {, args);

  public void logfast(String msg) {

The configuration for this app is straightforward. Like above, we have connection details for RabbitMQ. Also, note that the binding now references “input”, which was the name of the channel in the default “Sink” interface. Finally, observe that I used the SAME destination as the source, to ensure that Spring Cloud Stream wires up my publisher and subscriber successfully. For kicks, I didn’t yet add the consumer group settings.


#rabbitmq settings for Spring Cloud Stream to use

Test the solution

Let’s see how this works. First, start up three instances of the subscriber app. I generated a jar file, and started up three instances in the shell.


When you start up these apps, Spring Cloud Stream goes to work. Log into the RabbitMQ admin console, and notice that one exchange got generated. This one, named “legalbriefs”, maps to the name we put in our configuration file.


We also have three queues that map to each of the three app instances we started up.


Nice! Finally, start up the publisher, and post a message to the /briefs endpoint.


What happens? As expected, each subscriber gets a copy of the message, because by default, everything happens in a pub/sub fashion.


Add consumer group configuration

We don’t want each instance to get a copy. Rather, we want these instances to share the processing load. Only one should get each message. In the subscriber app, we add a single line to our configuration file. This tells Spring Cloud Stream that all the instances form a single consumer group that share work.

#adds consumer group processing

After regenerating the subscriber jar file, and starting up each file, we see a different setup in RabbitMQ. What you see is a single, named queue, but three “consumers” of the queue.


Send in two different messages, and see that each is only processed by a single subscriber instance. This is a simple way to use a message broker to scale out processing.


Doing stateful processing using partitioning

Partitioning feels like a related, but different scenario than consumer groups. Partitions in Kafka introduce a level of parallel processing by writing data to different partitions. Then, each subscriber pulls from a given partition to do work. Here in Spring Cloud Stream, partitioning is useful for parallel processing, but also for stateful processing. When setting it up, you specify a characteristic that steers messages to a given partition. Then, a single app instance processes all the data in that partition. This can be handy for event processing or any scenario where it’s useful for related messages to get processed by the same instance. Think counters, complex event processing, or time-sensitive calculations.

Unlike with consumer groups. partitioning requires configuration changes to both publishers AND subscribers.  On the publisher side, all that’s needed is (a) the number of partitions, and (b) the expression that describes how data is partitioned. That’s it. No code changes.

#adding configuration for partition processing

On the subscriber side, you set the number of partitions, and set the “partitioned” property equal to “true.” What’s also interesting, but logical, is that as each subscriber starts, you need to give it an “index” so that Spring Cloud Streams knows which partition it should read from.

#add partition processing

Let’s start everything up again. The publisher starts up the same as before. Now though, each subscriber instance starts up with a “”” flag that specifies which index applies.


In RabbitMQ, the setup is different than before. Now, we have three queues, each with a different “routing key” that corresponds to its partition.


Send in a message, and notice that all messages with the same attorney name go to one instance. Change the case number, and see that all messages still go to the same place. Switch the attorney ID, and observe that a different partition (likely) gets it. If you have more data varieties than you do partitions, you’ll see a partition handle more than one set of data. No problem, just know that happens.



It shouldn’t have to be hard to deal with message brokers. Of course there are plenty of scenarios where you want to flex the advanced options of a broker, but there are also many cases where you just want a reliable intermediary. In those cases, Spring Cloud Stream makes it super easy to abstract away knowledge of the broker, while still normalizing behavior across the unique engines.

In my latest Pluralsight course, I spent over an hour digging into Spring Cloud Stream, and another ninety minutes working with Spring Cloud Data Flow. That project helps you quickly string together Stream applications. Check it out for a deeper dive!


Categories: BizTalk, Cloud, DevOps, General Architecture, Messaging, Microservices, Pivotal, Spring

My new Pluralsight course about coordinating Java microservices with Spring Cloud, is out!

My new Pluralsight course about coordinating Java microservices with Spring Cloud, is out!

Home Cloud My new Pluralsight course about coordinating Java microservices with Spring Cloud, is out!

No microservice is an island. Oh sure, we all talk about isolated components and independent teams. But your apps and services get combined and used in all sorts of ways. It’s inevitable. As you consider how your microservices interact with each other, you’ll uncover new challenges. Fortunately, there’s technology that’s made those challenges easier to solve.

Spring Boot is hot. Like real hot. At this point, it’s basically the de facto way for Java developers build modern apps.

Spring Cloud builds on Spring Boot and introduces all sorts of distributed systems capabilities to your code. Last year I delivered a Pluralsight course that looked at building services with it. But that was only half of the equation. A big portion of Spring Cloud simplifies interactions between services. So, I set out to do a sequel course on Spring Cloud, and am thrilled that the course came out today.


This course, Java Microservices with Spring Cloud: Coordinating Services, focuses on helping you build a more effective, maintainable microservices architecture. How do you discover services? Can you prevent cascading failures? What are your routing options? How does messaging play a role? Can we rethink data integration and do it better? All those questions get answered in this ~6 hour course. The six modules of the course include:

  1. Introducing Spring Cloud and microservices coordination scenarios. A chat about the rise of microservices, problems that emerge, and what Spring Cloud is all about.
  2. Locating services at runtime using service discovery. The classic “configuration management DB” can’t keep up with the pace of change in a microservices architecture. No, you need a different way to see a live look at what services exist, and where they are. Enter Spring Cloud Eureka. We take a deep look at how to use it to register and discovery services.
  3. Protecting systems with circuit breakers. What happens when a service dependency goes offline? Bad things. But you can fail fast and degrade gracefully by using the right approach. We dig into Spring Cloud Hystrix in this module and look at the interesting ways to build a self-healing environment.
  4. Routing your microservices traffic. A centralized load balancer may not be the right fit for a fast-changing environment. Same goes for API gateways. This module looks at Spring Cloud Ribbon for client-side load balancing, and Spring Cloud Zuul as a lightweight microproxy.
  5. Connecting microservices through messaging. Message brokers offer one convenient way to link services in a loosely coupled way. Spring Cloud Stream is a very impressive library that makes messaging easy. Add competing consumers, partition processing, and more, whether you’re using RabbitMQ or Apache Kafka underneath. We do a deep dive in this module.
  6. Building data processing pipelines out of microservices. It’s time to rethink how we process data, isn’t it? This shouldn’t be the realm of experts, and require bulky, irregular updates. Spring Cloud Data Flow is one of the newest parts of Spring Cloud, and promises to mess with your mind. Here, we see how to do real-time or batch processing by connecting individual microservices together. Super fun.

It’s always a pleasure creating content for the Pluralsight audience, and I do hope you enjoy the course!


Categories: Cloud, DevOps, General Architecture, Messaging, Microservices, Spring