Year 2016 in review, Integration is still relevant

Year 2016 in review, Integration is still relevant

Last few hours before 2016 ends and what a year it has been as so many things happened. The year started off with finalizing and publishing an eBook on extensibility of BizTalk Server, which is available for free. My buddy Saravana made available through BizTalk360 website.


The publishing of the book marked the first major milestone in 2016. The collaboration of Eldert, Johann and myself as authors and three awesome reviewers in Mark, James and Sandro delivered a 150-page book on the power of .NET and BizTalk.
With the release of the book, Eldert and myself embarked on a speaking tour to promote the book, which we did in front of the BizTalk User Group in the Netherlands, online through integration Monday and in Sweden (Stockholm, Gothenburg). Thanks Mikael, Johan and Peter!!!


In the spring (May) I presented at Integrate 2016 on IoT end to end leveraging the Microsoft Azure Platform services. And this was in front of almost 400 people.



Who would have thought that integration is irrelevant? It’s not! Thanks for  Saravana and the BizTalk360 team for making this three-day event happening. Integrate 2016 was a success and for me my second major milestone in 2016.


Before Integrate 2016 Kent, his wife Melissa travelled through the UK and Scotland enjoying various country sides. One of the best days was hanging out with Michael in Newcastle. A really nice place with some great bars and restaurants.


After Integrate I went to TUGA IT, a large two-day conference in Lisbon. Thanks to Sandro I got invited to speak together with Nino. The three of us did a full day workshop on integration on Friday and a few talks on Saturday.


We had a blast and I really enjoyed Lisbon together with Nino and Sandro. Thanks to Niko Neugebauer and other Portuguese involved! Before I left I spend a full day exploring Lisbon with Sandro.


After Integrate and TUGA IT I went to Sweden with Eldert to talk about BizTalk Extensibility and another topic Azure Web Jobs. Sweden and its user groups are awesome, great welcome, excellent hosting environments and crowd. In Stockholm, we had a great user group event with several speakers including Mikael, Johan, Eldert, Robin, Ashwin and myself. The next day Eldert and myself went to Gothenburg to speak on extensibility and Web Jobs.

The summer holidays I spent in Italy and started to seriously prepare for third major milestone in 2016, the NYC marathon. In 2013 I ran the Berlin marathon together with Kent and decided or at least I said I will do another, preferably New York. Once Kent and his brother Kurt decided they wanted to run it when they turned 40 I said I am in!


The race was brutal and the bridges were quite the experience and challenge. It was a psychical and mental test yet I managed to finish with 5 hours.


My Canadian buddies finished just a few minutes later and then we enjoyed our medals with some great food and beers.


Thanks Kent and Kurt for letting me join in this endeavor. Our next challenge might be in 2018.


The next stop after the Marathon was the MVP Summit, which started the day after NYC Marathon. And to start things of was a nice game of the Seahawks playing the Bills. Lots of MVP and some MS folks went to the game.


The MVP Summit was a great experience as there was lots to share and engagements with the product teams.


You always run into someone you know ☺


This summit took place during the elections, hence we spent an evening together at Dan and Ola, who organized a great barbeque.


The outcome was not what we thought it would be, however, we still had a great time.


One of the traditions during the Summit is visiting Joey’s in Bellevue, which of course we did.


And Moksha our favorite Indian Cuisine place.


Another thing we did was visiting Tord, PG BizTalk Server and his family in Redmond.

And the BizTalk Crew was reunited once again.


We know BizTalk Server is in good hands now!
I really enjoyed my stay in Seattle. I went to another football game with Kent. We watched the Washington Huskies.


Before I left Seattle, I spent some time with my great friend Tom and his wife JoAnn. We watched the Seahawks play the Patriots on Sunday in a bar. Yes, more American Football, I love it. And it was an epic game, with the Hawks winning the game and cheers in the bar!


It was quite a trip, New York, running the Marathon followed by the MVP Summit. Something to remember as long as I will live.

In November and December I spend time writing some quest blogs for Saravana, a LinkedIn article on today’s integration and presenting at our Local user group.


In 2017 I, will kick the tires with some speaking engagements in Australia. In February I, will spent two and a half week with Mick, Eldert, Rene, Dean, Bill, Dan and many others. This will be followed by the global integration boot camp in March, where I’ll be speaking at the location of motion10 together with Eldert, Rob, Gijs, and Tomasso. In April and May I will be on the road to probably to speak in Portugal, Sweden and some other countries or venues. It will be an interesting year 2017!

P.S. I lost some kind of bet I have to wear some jersey with a name of a soccer player I really like, NOT!!!
















Cheers,
Steef-Jan

2016 Year in Review, Looking Ahead to 2017

2016 Year in Review, Looking Ahead to 2017

It is that time of year where I like to reflect back on what the previous year has brought and also set my bearings for the road ahead.  If you are interested in reading my 2015 recap, you can find it here.

Personal

2016 was a milestone birthdate for myself and my twin brother. In order to celebrate, and try to deny getting old for another year, we decided to run a marathon in New York City.  The NYC Marathon is one of the 6 major marathons in the world so it acted as a fantastic backdrop for our celebration.  Never one to turn down an adventure, my good friend Steef-Jan Wiggers joined us for this event.  As you may recall, Steef-Jan and I ran the Berlin Marathon (another major) back in 2013.

Marathon

The course was pretty tough.  The long arching bridges created some challenges for me, but I fought through it and completed the race.  We all finished within about 10 minutes of each and had a great experience touring the city.

Before Race 

Kurt, Kent and Steef-Jan in the TCS tent before the race

AfterRace

At the finish line with the hardware.

After Pary

Celebrating our victory at Tavern on the Green in Central Park.

Speaking

Traveling and speaking is something I really like to do and the MVP program has given me many opportunities to scratch this itch. I also need to thank my previous boss and mentor Nipa Chakravarti for all of the support that she has provided which made all of this possible.

In Q2, I once again had a chance to head to Europe to speak at BizTalk360’s Integrate Event with the Microsoft Product Group.  My topic was on Industrial IoT and some of the project work that we had been working on. You can find a recording of this talk here.

Speaking

On stage….

Integrate

I really like this photo as it reminds me of the conversation I was having with Sandro.  He was trying to sell me a copy of his book, and I was trying to convince him that if he gave me a free copy, that I could help him sell more.  Sandro has to be one of the hardest working MVPs I know who is recognized as one of the top Microsoft Integration Gurus.  If you have ever having a problem in BizTalk, there is a good chance he has already solved it.  You can find his book here in both digital and physical versions.

Integrate Group

BizTalk360 continues to be an integral part of the Microsoft Integration community.  Their 2016 event had record attendance from more than 20 countries.  Thank-you BizTalk360 for another great event and for building a great product.  We use BizTalk360 everyday to monitor our BizTalk and Azure services. 

On a bit of a different note, this past year we had a new set of auditors come in for SOX compliance.  For the first time, that I have experienced, the auditors were really interested in how we were monitoring our interfaces and what our governance model was.  We passed the audit with flying colours, but that was really related to having BizTalk360.  Without it, our results would not have been what they were.

Q3

Things really started to heat up in Q3.  My first, of many trips, was out to Toronto to speak at Microsoft Canada’s Annual General meeting. I shared the stage with Microsoft Canada VP Chris Barry as we chatted about Digital Transformation and discuss our experiences with moving workloads to the cloud.

AGM

Next up was heading to the south east United States to participate in the BizTalk Bootcamp. This is my third time presenting at the event.  I really enjoy speaking at this event as it is very well run and is in a very intimate setting.  I have had the chance to meet some really passionate integration folks at this meetup so it was great to catch up once again.  Thank-you Mandi Ohlinger and the Microsoft Pro Integration team for having me out in Charlotte once again.

BizTalkBootcamp

At the Bootcamp talking about Azure Stream Analytics Windowing.

The following week, I was off to Atlanta to speak at Microsoft Ignite.  Speaking at a Microsoft premier conference like Ignite (formerly TechEd) has been a bucket list item so this was a really great opportunity for me.  At Ignite, I was lucky enough to participate in two sessions.  The first session that I was involved in was a customer segment as part of the PowerApps session with Frank Weigel and Kees Hertogh.  During this session I had the opportunity to show off one of the apps my team has built using PowerApps.  This app was also featured as part of a case study here.

Powerapps

On stage with PowerApps team.

Next up, was a presentation with John Taubensee of the Azure Messaging team.  Once again my presentation focused on some Cloud Messaging work that we had completed earlier in the year.  Working with the Service Bus team has been fantastic this year.  The team has been very open to our feedback and has helped validate different use cases that we have.  In addition to this presentation, I also had the opportunity to work on a customer case study with them.  You can find that document here. Thanks Dan Rosanova, John Taubensee, Clemens Vasters and Joe Sherman for all the support over the past year.

Ignite

Lastly, at the MVP Summit in November, I had the opportunity to record a segment in the Channel 9 studio.  Having watched countless videos on Channel 9, this is always a neat experience.  The segment is not public yet, but I will be sure to post when it is.  Once again, I had the opportunity to hang out with Sandro Pereira before our recordings.

Sandro

Channel9Studio

In the booth, recording.

Channel9StudioCouch

Prepping in the Channel 9 studio

Writing

I continue to write for InfoQ on Richard Seroter’s Cloud Editorial team.  It has been a great experience writing as part of this team.  Not only do I get exposed to some really smart people, I get exposed to a lot of interesting topics that only fuels my career growth.  In total, I wrote 46 articles but here are my top 5 that I either really enjoyed writing or learned a tremendous amount about.

  • Integration Platform as a Service (iPaaS) Virtual Panel In this article, I had the opportunity to interview some thought leaders in the iPaaS space from some industry leading organizations.  Thank-you Jim Harrer (Microsoft), Dan Diephouse (MuleSoft) and Darren Cunningham (SnapLogic) for taking the time to contribute to this feature.  I hope to run another panel in 2017 to gauge how far iPaaS has come.
  • Building Conversational and Text Interfaces Using Amazon Lex – After researching this topic, I immediately became interested in Bots and Deep Learning.  It was really this article that acted as a catalyst for spending more time in this space and writing about Google and Microsoft’s offerings.
  • Azure Functions Reach General Availability Something that I like to do, when possible, is to get a few sound bytes from people involved in the piece of news that I am covering.  I met Chris Anderson at the Integrate event earlier in the year, so it was great to get more of his perspective when writing this article.
  • Microsoft PowerApps Reaches General Availability – Another opportunity to interview someone directly involved in the news itself.  This time it was Kees Hertogh, a Senior Director of Product Marketing at Microsoft. 
  • Netflix Cloud Migration Complete – Everyone in the industry knows that Netflix is a very innovative company and has disrupted and captured markets from large incumbents.  I found it interesting to get more insight into how they have accomplished this.  Many people probably thought the journey was very short, but what I found was that it wasn’t the case.  It was a very methodical approach that actually took around 8 years to complete.

Another article that I enjoyed writing was for the Microsoft MVP blog called Technical Tuesday.  My topic focused on Extending Azure Logic Apps using Azure Functions. The article was well received and I will have another Technical Tuesday article published early in the new year.

Back to School

Blockchain

I left this topic off of the top 5 deliberately as I will talk about it here, but it absolutely belongs up there. Back in June, I covered a topic for InfoQ called Microsoft Introduces Project Bletchley: A Modular Blockchain Fabric.  I really picked up this topic out of our Cloud queue as my boss at the time had asked me about Blockchain and I didn’t really have a good answer. After researching and writing about the topic, I had the opportunity to attend a Microsoft presentation in Toronto for Financial organizations looking to understand Blockchain.  At the Microsoft event (you can find similar talk here), Alex Tapscott gave a presentation about Blockchain and where he saw it heading.  ConsenSys, a Microsoft partner and Blockchain thought leader was also there talking about the Brooklyn Microgrid. I remember walking out the venue that day thinking everything was about to change.  And it did.  I needed to better understand blockchain.

For those that are not familiar with blockchain, simply put, it is a paradigm that focuses on using a distributed ledger for recording transactions and providing the ability to execute smart contracts against these transactions.  An underlying principle of blockchain is to address the transfer of trust amongst different parties.  Historically, this has been achieved through intermediaries that act as a “middleman” between trading partners.  In return, the intermediary takes a cut on the transaction, but doesn’t really add a lot of value beyond collecting and dispersing funds.  Trading parties are then left to deal with the terms that the intermediary sets.  Using this model typically does not provide incentives for innovation, in fact it typically does the opposite and stifles it due to complacency and entitlement by large incumbent organizations.

What you will quickly discover with blockchain is that it is more about business than technology.  While technology plays a very significant role in blockchain, if your conversation starts off with technology, you are headed in the wrong direction.  With this in mind, I read Blockchain Revolution by Alex and Don Tapscott which really focuses on the art of the possible and identifying some modern-day scenarios that can benefit from blockchain.  While some of the content is very aspirational, it does set the tone for what blockchain could become.

Having completed the book, I decided to continue down the learning path.  I wanted to now focus on the technical path.  I am a firm believer that in order for me to truly understand something, I need to touch it.  By taking the Blockchain Developer course from B9Lab I was able to get some hands on experience with the technology.  As a person that spends a lot of time in the Microsoft ecosystem, this was a good learning opportunity to get back into Linux and more of the open source community as blockchain tools and platforms are pretty much all open source.  Another technical course that I took was the following course on Udemy.  The price point for this course is much lower, so it may be a good place to start without making a more significant financial investment in a longer course.

Next, I wanted to be able to apply some of my learnings.  I found the Future Commerce certificate course from MIT.  It was a three month course, all delivered online.  There were about 1000 students, worldwide, in the course and it was very structured and based upon a lot of group work.  I had a great group that I worked with on an Energy-based blockchain startup.  We had to come up with a business plan, pitch deck, solution architecture and go to market strategy, Having never been involved in a start-up at this level (I did work for MuleSoft, but they were at more than 300 people at the time), it was a great experience to work through this under the tutelage of MIT instructors. 

If you are interested in the future of finance, aka FinTech, I highly recommend this course.  There is a great mix of Finance, Technology, Entrepreneurs, Risk and Legal folks in this class you will learn a lot.

MIT

Gary Vaynerchuk

While some people feel that Twitter is losing its relevancy, I still get tremendous value out of the platform.  The following is just an example.  Someone I follow on Twitter is Dona Sarkar, from Microsoft, I had the opportunity to see her speak at the Microsoft World Partner Conference and quickly became a fan.  Back in October, she put out the following tweet, which required further investigation on my part.

Gary V

Dona’s talks, from the ones that I have seen, are very engaging and also entertaining at the same time.  If she is talking about “Gary Vee” in this manner, I am thinking there is something here.  So I start to digest some of his content.  I was very quickly impressed.  What I like about Gary is he has a bias for action.  Unfortunately, I don’t see this too often in Enterprise IT shops; we try to boil the ocean and watch initiatives fail because people have added so much baggage that the solution is unachievable or people have become disenfranchised.  I have also seen people being rewarded for building “strategies” without a clue how to actual implement them.  I find this really prevalent in Enterprise Architecture where some take pride in not getting into the details.  While you may not need to stay in the details for long, without understanding the mechanics, a strategy is just a document.  And a strategy that has not/cannot be executed is useless.

If you have not spent time listening to Gary, here are some of his quotes that really resonated with me.

  • Bet on your strengths and don’t give a f&%# about what you are not good at.
  • Educate…then Execute
  • You didn’t grow up driving, but somehow you figured it out.
  • Results are results are results
  • I am just not built, to have it dictate my one at-bat at life.
  • Document, Don’t Create.
  • We will have people who are romantic and hold onto the old world who die and we will have people that execute and story tell on the new platform who emerge as leaders in the new world.
  • I am built to get punched in the mouth, I am going spit my front tooth out and look right back at you and be like now what bitch.

image

If this sounds interesting to you, check out a few of his content clips that I have really enjoyed:

Looking Forward

I find it is harder and harder to do this.  The world is changing so fast, why would anyone want to tie themselves down to an arbitrary list? Looking back on my recap from last year, you won’t find blockchain or bots anywhere in that post, yet those are two of the transformative topics that really interested me in 2016.  But, there are some constants that I don’t see changing.  I will continue to be involved in the Microsoft Integration community, developing content, really focused on iPaaS and API Management.  IoT continues to be really important for us at work so I am sure I will continue to watch that space closely.  In fact, I will be speaking about IoT at the next Azure Meetup in Calgary on January 10th.  More details here.

I will also be focusing on blockchain and bots/artificial intelligence as I see a lot of potential in these spaces.  One thing you can bet on is that I will be watching the markets closely and looking for opportunities where I see a technology disrupting or transforming incumbent business models.

Also, it looks like I will be running a marathon again in 2017.  My training has begun and am just awaiting confirmation into the race.

Advertisements

2016 in Review: Reading and Writing Highlights

2016 in Review: Reading and Writing Highlights

2016 was a wild year for plenty of folks. Me too, I guess. The wildest part was joining Pivotal and signing up for a job I’d never done before. I kept busy in other ways in 2016, including teaching a couple of courses for Pluralsight, traveling around to speak at conferences, writing a bunch for InfoQ.com, and blogging here with semi-regularity. 2017 should be more of the same (minus a job change!), plus another kiddo on the way.

I tend to read a lot, and write a bit, so each year I like to reflect on my favorites.

I create stuff in a handful of locations—this blog, InfoQ.com, Pivotal blog—and here were the content pieces I liked the most.

[My Blog] Modern Open Source Messaging: Apache Kafka, RabbitMQ and NATS in Action. This was my most popular blog post this year, by far. Application integration and messaging are experience a renaissance in this age of cloud and microservices, and OSS software is leading the way. If you want to watch my conference presentation that sparked this blog post, head to the BizTalk360 site.

[My Blog] Trying out the “standard” and “enterprise” templates in Azure Logic Apps. Speaking of app integration, Microsoft turned a corner in 2016 and has its first clear direction in years. Logic Apps is a big part of that future, and I gave the new stuff a spin. FYI, since I wrote the original post, the Enterprise Integration Pack shipped with a slightly changed user experience.

[My Blog] Characteristics of great managers. I often looked at “management” as a necessary evil, but a good manager actually makes a big difference. Upon reflection, I listed some of the characteristics of my best managers.

[My Blog] Using Concourse to continuously deliver a Service Bus-powered Java app to Pivotal Cloud Foundry on Azure. 15 years. That’s how long it had been since I touched Java. When I joined Pivotal, the company behind the defacto Java framework called Spring, I committed to re-learning it. Blog posts like this, and my new Pluralsight course, demonstrated that I learned SOMETHING.

[InfoQ] Outside of my regular InfoQ contributions covering industry news, I ran a series on the topic of “cloud lock-in.” I wrote an article called “Everything is Lock-In: Focus on Switching Costs” and facilitated a rowdy expert roundtable.

[InfoQ] Wolfram Wants to Deliver “Computation Everywhere” with New Private Cloud. I purposely choose to write about things I’m not familiar with. How else am I supposed to learn? In this case, I dug into the Wolfram offerings a bit, and interviewed a delightful chap.

[Pivotal] Pivotal Conversations Podcast. You never know what may happen when you say “yes” to something. I agreed to be a guest on a podcast earlier this year, and as a result, my delightfully bearded work colleague Coté asked me to restart the Pivotal podcast with him. Every week we talk about the news, and some tech topic. It’s been one of my favorite things this year.

[Pivotal] Standing on the Shoulders of Giants: Supercharging Your Microservices with NetflixOSS and Spring Cloud. I volunteered to write a whitepaper about microservices scaffolding and Spring Cloud, and here’s the result. It was cool to see thousands of folks check it out.

[Pivotal blog] 250k Containers In Production: A Real Test For The Real World. Scale matters, and I enjoyed writing up the results of an impressive benchmark by the Cloud Foundry team. While I believe our industry is giving outsized attention to the topic of containers, the people who *should care* about them (i.e. platform builders) want tech they can trust at scale.

[Pivotal blog] To Avoid Getting Caught In The Developer Skills Gap, Do This. It’s hard to find good help these days. Apparently companies struggle to fill open developer positions, and I offered some advice for closing the skills gap.

I left my trusty Kindle 3 behind on an airplane this year, and replaced it with a new Kindle Paperwhite. Despite this hiccup, I still finished 31 books this year. Here are the best ones I read.

The Hike. I don’t read much fantasy-type stuff, but I love Drew’s writing and gave this a shot. Not disappointed. Funny, tense, and absurd tale that was one of my favorite books of the year. You’ll never look at crustaceans the same way again.

The Prey Series. I’m a sucker for mystery/thriller books and thought I’d dig into this  long-running series. Ended up reading the first six of them this year. Compelling protagonist, downright freaky villains.

The Last Policeman Trilogy. I’m not sure where I saw the recommendation for these books, but I’m glad I did. Just fantastic. I plowed through Book 1, Book 2, and Book 3 in about 10 days. It starts as a “cop solving a mystery even though the world is about to end” and carries onward with a riveting sense of urgency.

Rubicon: The Last Years of the Roman Republic. I really enjoyed this. Extremely engaging story about a turning point in human history. It was tough keeping all the characters straight after a while, but I have a new appreciation for the time period and the (literally) cutthroat politics.

The Great Bridge: The Epic Story of the Building of the Brooklyn Bridge. It’s easy to glamorize significant construction projects, but this story does a masterful job showing you the glory *and* pain. I was inspired reading it, so much so that I wrote up a blog post comparing software engineering to bridge-building.

The Path Between the Seas: The Creation of the Panama Canal, 1870-1914. You’ve gotta invest some serious time to get through McCullough’s books, but I’ve never regretted it. This one is about the tortured history of building the Panama Canal. Just an unbelievable level of effort and loss of life to make it happen. It’s definitely a lesson on preparedness and perseverance.

The Summer of 1787: The Men Who Invented the Constitution. Instead of only ingesting hot-takes about American history and the Founder’s intent, it’s good to take time to actually read about it! I seem to read an American history book each year, and this one was solid. Good pacing, great details.

The Liberator: One World War II Soldier’s 500-Day Odyssey from the Beaches of Sicily to the Gates of Dachau. I also seem to read a WWII book every year, and this one really stayed with me. I don’t believe in luck, but it’s hard to attribute this man’s survival to much else. Story of hope, stress, disaster, and bravery.

Navigating Genesis: A Scientist’s Journey through Genesis 1–11. Intriguing investigation into the overlap between the biblical account and scientific research into the origins of the universe.  Less conflict than you may think.

Boys Among Men: How the Prep-to-Pro Generation Redefined the NBA and Sparked a Basketball Revolution. I’m a hoops fan, but it’s easy to look at young basketball players as spoiled millionaires. That may be true, but it’s the result of a system that doesn’t set these athletes up for success. Sobering story that reveals how elusive that success really is.

Yes, My Accent Is Real: And Some Other Things I Haven’t Told You. This was such a charming set of autobiographical essays from The Big Bang Theory’s Kunal Nayyar. It’s an easy read, and one that provides a fun behind-the-scenes look at “making it” in Hollywood.

Eleven Rings: The Soul of Success. One of my former colleagues, Jim Newkirk, recommended this book from Phil Jackson. Jim said that Jackson’s philosophy influenced how he thinks about software teams. Part autobiography, part leadership guide, this book includes a lot of advice that’s applicable to managers in any profession.

Disrupted: My Misadventure in the Start-Up Bubble. I laughed, I cried, and then I panicked when I realized that I had just joined a startup myself. Fortunately, Pivotal bore no resemblance to the living caricature that is/was HubSpot. Read this book from Lyons before you jump ship from a meaningful company to a glossy startup.

Overcomplicated: Technology at the Limits of ComprehensionThe thesis of this book is that we’re building systems that cannot be totally understood. The author then goes into depth explaining how to approach complex systems, how to explore them when things go wrong, and how to use caution when unleashing this complexity on customers.

Pre-Suasion: A Revolutionary Way to Influence and Persuade. If you were completely shocked by the result of the US presidential election, then you might want to read this book. This election was about persuasion, not policy. The “godfather of persuasion” talks about psychological framing and using privileged moments to impact a person’s choice. Great read for anyone in sales and marketing.

Impossible to Ignore: Creating Memorable Content to Influence Decisions: Creating Memorable Content to Influence Decisions. How can you influence people’s memories and have them act on what you think is important? That’s what this book attempts to answer. Lots of practical info grounded in research studies. If you’re trying to land a message in a noisy marketplace, you’ll like this book.

Win Your Case: How to Present, Persuade, and Prevail–Every Place, Every Time. I apparently read a lot about persuasion this  year. This one is targeted at trial lawyers, but many of the same components of influence (e.g. trust, credibility) apply to other audiences.

The Challenger Customer: Selling to the Hidden Influencer Who Can Multiply Your Results. Thought-provoking stuff here. The author’s assertion is that the hard part of selling today isn’t about the supplier struggling to sell their product, but about the customer’s struggle to buy them. An average of 5.4 people are involved in purchasing decisions, and it’s about using “commercial insight” to help them create consensus early on.

The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. This is the new companion book to the DevOps classic, The Phoenix Project. It contains tons of advice for those trying to change culture and instill a delivery mindset within the organization. It’s full of case studies from companies small and large. Highly recommended.

Start and Scaling DevOps in the Enterprise. Working at Pivotal, this is one of the questions we hear most from Global 2000 companies: how do I scale agile/DevOps practices to my whole organization? This short book tackles that question with some practical guidance and relevant examples.

A sincere thanks to all of you for reading my blog, watching my Pluralsight courses, and engaging me on Twitter in 2016. I am such a better technologist and human because of these interactions with so many interesting people!

Advertisements



Categories: BizTalk, General Architecture, Microservices, Microsoft Azure, Pivotal

Using the XSLT API from Apigize to transform XML messages

Using the XSLT API from Apigize to transform XML messages

I’d like to build tools for other developers and in the past I created some small tools for BizTalk 2010 and BizTalk 2013 to test your maps. This time I wanted to create something else because nowadays I’m focusing more and more on functionality in Azure. Therefore I created an XSLT API that you can use to transform XML messages with XSLT. The API can also be used to execute BizTalk maps as long you don’t have Custom Functoids.

Prerequisites

The only thing what you have to do is to create account on Apigize.com. Then you get secure and optimized cloud storage for free that is only accessible by yourself. You can upload up to 10 XSLT files without any costs!

On the Register page enter your details and then click on the Register button.
Register an user in apigize.com
 
Copy the Primary key value from Your subscription details.
User Subscription Details
Note:
Every API request you make to the Web API must be authenticated by including an Authorization Header with your API Key.
 
Click on Files API in the Documentation menu to see the operations of the Files API.
API Reference on Apigize website
 

How to upload a XSLT map?

You can create a XSLT map by using the Visual Studio Enterprise Integration SDK or for example another XSLT tool like MapForce and XMLSpy from Altova.  When you are finished creating the XSLT transformation, you upload the XSLT file into your integration account.  
 
Use a tool like Postman to call the Files API.

Select POST and enter: https://free.apigize.io/v1/files in the Request URL.
In the Header enter “Ocp-Apim-Subscription-Key” as the key name and your Primary key as the value.
Postman - Headers for FILE API
 
In the Body select the form-data radio button and then select the file that you want to upload.
Click on the Send button to upload the file.
Postman - Upload a file to my account with Files API
 

 

How to create a XSLT transform?

With the XSLT API you can transform an .XML message.

Select POST and enter: https://free.apigize.io/v1/xslt/{XSLT filename} in the Request URL.
In the Header enter “Ocp-Apim-Subscription-Key” as the key name and your Primary key as the value. Also specify the Content-Type and what you want to Accept.
Postman - Headers for XSLT API
 
In the Body select the raw radio button and then enter the XML message that you want to transform.
Click on the Send button to transform the message.
Postman - Transform an XML message with XSLT API
 

Conclusion

In this blog post I only showed how to upload a file with the Files API and how to use the XSLT API. Of course there are more operations available on the API’s but these work similar. I want to add even more functionality to the API’s so stay tuned!

Handling Azure Alerts with Azure Logic Apps

Handling Azure Alerts with Azure Logic Apps

Luckily, you can configure your Azure Alerts to push to a webhook where you process the notifications and Azure Logic Apps is a perfect fit for this! By configuring all your Azure Alerts to push the events to your Azure Logic Apps, you decouple the processing from the notification medium and can easily change the way an event can be handle.

Today I will show you how you can push all your Azure Alerts to a dedicated Slack channel, but you can use other Logic App Connectors to fit your need as well!

Creating a basic Azure Alert Handler

For starters, we will create a new Logic App that will receive all the event notifications – In this example azure-alert-handler.

Note: It is a best practice to host it as close to your resources as possible so I provision it in West Europe.

Once it is provisioned, we can start by adding a new Request Triggerconnector. This trigger will expose a webhook that can be called on a dedicated URL. This URL is generated once you save it for the first time.

As you can see you can also define the schema of the events that will be received but more on this later on.

Now that our Logic App is ready to receive events, we can configure our Azure Alerts. In this scenario we will create an alert on an API, but you could do this on almost any Azure resource. Here we will configure it to get a notification once there are more than 100 HTTP Server Errors in 5 minutes.

To achieve this, navigate to your Web App and search for “Alerts”.

Click on “New Alert”, define what the alert should monitor and specify the URL of the Request Trigger in our Logic App.

We are good to go! This means that if our Alert will change its state, it will push a notification to our webhook inside our Logic Apps.

You can see all events coming in our Logic App by using the Trigger History and All Runs of the Logic App itself.

When you select a Run you can view the details and thus what it has sent. Based on this sample payload, I generated the schema with jsonschema.net and used that to define the schema of the webhook. 

Didn’t specify the schema? Don’t worry you can still change it!

While this is great, I don’t want to come back every 5 minutes to see whether or not there were new events.

Since we are using Slack internally this is a good fit consolidate all alerts in a dedicated channel so that we have everything in one place.

To push the notification to Slack, add a new Slack (Post Message) Action and authenticate with your Slack team.

Once you are authenticated, it is fairly simple to configure the connector – You need to tell it to what channel you want to push messages, what the message should look like or even other things like the name of the bot et al.

Here is an overview of all the settings of the Slack connector that you can use.

I used the @JSON function to parse the JSON input dynamically, later on we will have a look how we can simplify this.

"Alert *'@{JSON(string(trigger().outputs.body)).context.name}'* is currently *@{JSON(string(trigger().outputs.body)).status}* for *@{JSON(string(trigger().outputs.body)).context.resourceName}* in @{JSON(string(trigger().outputs.body)).context.resourceRegion}_(@{JSON(string(trigger().outputs.body)).context.portalLink})_"

Once our updated Logic App is triggered you should start receiving messages in Slack.

Tip – You can also Resubmit a previous run, this allows you to take the original input and re-run it again with that information.

Awesome! However, it tends to be a bit verbose because it mentions a lot of information, despite that it’s already resolved. Nothing we can’t fix with Logic Apps!

Sending specific messages based on the alert status

In Logic Apps you can add a Condition that allows you to execute certain logic if some criteria are met.

In our case we will create a more verbose message when a new Alert is ‘Activated’ while for other statuses we only want to give a brief update about that alert.

As you can see we are no longer parsing the JSON dynamically but rather using dynamic content, thanks to our Request Trigger Schema. This allows us to create more human-readable messages while omitting the complexity of the JSON input.

Once our new alerts are coming in it will now send customized messages based on the event!

Monitoring the Monitoring

The downside of centralizing the processing of something is that you create a single-point-of-failure. If our Logic App is unable to process events we won’t see Slack messages assuming that everything is fine, while it certainly isn’t. Because of that, we need to monitor the monitoring!

If you search for “Alerts” in your Logic App you will notice that you can create alerts for it as well. 
As you can see there are no alerts available by default so we will add one.

In our case we want to be notified if a certain amount of runs is failing. When that happens we want to receive an email. You could setup another webhook as well but I think emails are a good fit here.

Wrapping up

Thanks to this simple Logic App I can now easily customize the processing of our Azure Alerts without having to change any Alerts.

This approach also gives more flexibility in how we process them – If we have to process database alerts differently, we want to change Slack with SMS or another integration it is just a matter of changing the flow.

But don’t forget to monitor the monitoring!

Thanks for reading,

Tom Kerkhove.

PS – Thank you Glenn Colpaert for the jsonschema.net tip!

BTUG.be XL – BizTalk 2016 Slidedeck

Last week, I had a great time at BTUG.be, while presenting my session on BizTalk 2016.

I presented the new features in BizTalk Server 2016 RTM and a few takeaway from SQL Server 2016. More specifically, and in-depth, on SQL Server AlwaysOn support for BizTalk Server 2016 on-premise and in the Azure cloud, as well as an intro on the new Logic App adapter and how to install and connect it to your on-premise BizTalk Server.

As promised there, please find my slide deck below via SlideShare:

Contact me if you have any questions regarding the slides, I’d be happy to answer you.

The other speakers there were Glenn Colpaert (session about Azure Functions), Kristof Rennen (session on Building scalable and resilient solutions using messaging) and Nino Crudele (session on Holistic approaches to Integration).

As always, it was nice to talk to the people present. A big thank you to BTUG.be for having me again!

Enjoy the slide deck!

Pieter

Integration Magic – End-to-End Scenario part 2

Integration Magic – End-to-End Scenario part 2

In our previous post, I guided you through setting up a WCF service and protecting it using URL Authentication. Although a lengthy post you would have noticed that setting up url-authentication is actually quite simple and only involving a few steps.

Anyways, in this post we will be focusing on adding the integration magic, without adding a single line of custom code, using Azure Logic Apps.

The integration magic which we will be adding will take care of the following functionality within our end-to-end scenario.

A request will come in which will start our process which is to retrieve a list of customers.

The customer information to be retrieved combines the result from two sources; the first source being the WCF service we build in our previous post and the second source a public rest api. The data which is to be returned to the caller as such will consist of the base data originating from the wcf services enriched with data obtained from the public rest api.

Visualizing the flow

Before we start implementing the solution using Logic Apps it is always a good practice to work-out the actual process flow using a tool such as Microsoft Visio.

Having said that, let’s eat my own dogfood. Low and behold, see below the diagram depicting the process and an explanation of the process.

clip_image002[1]

The process kicks off whenever a http post requesting a list of customer data is being made to Logic Apps (1). Once received within logic apps a new message (soap request) has to be created (2). Once created this message is being offloaded to the custom WCF service (3), we created in the previous post. If the call is successful the webservice will return a list of customers (4). The information contained within the response contains the following data: customerId, FirstName, SurName and postcode.

The postcode value(s) contained within this response is subsequently used to retrieve detailed location information.

In order to retrieve this location information, logic apps will perform a loop over the response message (5), extract the postal code and invoke a custom rest API to do the location lookup (6). The response received contains the following data: Suburb name, postcode, state-name, state abbreviation, locality and the latitude and longitude of the locality.

This data and the basic customer data is then combined and temporarily persisted in DocumentDB (7).

//Reason, for leveraging this external persistence store is to make life easier for us, as we want //enrich all the customer data with additional information retrieved from the second api call and //return it in one go to the caller. Currently there is no easy way of doing this directly from within //logic-apps as, however, have no fear; in one of the next releases a feature to store session state //within a logic app will be implemented and thus we would no longer need to result to an //intermediate ‘session state’ store.

This process is then repeated for all customers and once we have iterated over all customer records we exit the loop and retrieve all ‘enriched’ documents stored in DocumentDB (8) which we then will return to the caller. The information returned to the caller will then contain the following data; FirstName, LastName and Location information consisting of Locality, State Name, SubUrb, Postcode and longitude and latitude (9).

Provision the logic App

At this point we have worked out the high-level flow and logic and as such we can now go-ahead and create the logic app, so let’s go ahead and do so

1. Login to the Azure Portal

2. Select the resource-group which you created in part-1, in which you deployed your custom wcf service. In my case this resource-group is called Demos

3. Once the resource-group blade is visible, click on the Add button

clip_image003[1]

4. A new blade will popup, within this blade search for Logic App and click on the Logic App artefact published by Microsoft and of the Category Web + Mobile

clip_image004[1]

5. Click on create

clip_image005[1]

6. Now fill out the details and once done click Create, after which your logic app will be created

clip_image006[1]

7. Once the logic app has been created, open it and you should be presented with a screen which allows you to create a new logic app using one of the pre-build templates. In our case we will choose the “Blank LogicApp”

clip_image008

Implement the ‘Blank LogicApp’

Once you’ve clicked on the blank logic app template, the designer will pop up. We will be using this designer to develop the below depicted flow which will be explained in the following sections. Well let’s get started.

clip_image010

Step 1: Request Trigger

Within this designer, you will be presented with a ‘card selector’. This card selector, being the first of many, contains so-called triggers. These triggers can best be explained as ‘event listeners’ which indicate when a logic app is to be instantiated.

In our scenario, we want to trigger our logic app by means of sending a request. So, in our case we would select the Request trigger. Now select this Request Trigger.

clip_image012

To dig up more information regarding the different triggers and actions you can click on the Help button, which will open up a Quick Start Guide blade containing links too more information.

clip_image014[1]

Configure

Once you’ve selected the trigger, the Request Trigger ‘Card’ will be expanded and will allow you to configure this trigger.

clip_image015[1]

1. This section is not customizable, but once the logic app is saved will contain the generated endpoint. This endpoint is to be used by clients who which to invoke the logic app.

clip_image017[1]

2. The request body JSON schema section, is an optional section, which allows us to add a schema describing what the inbound request message should look like.

clip_image019

You might be wondering why bother? Well if we bother by adding a schema we get the benefit of an ‘intellisense like’ experience from within the designer, which can help us down the road in case we want to easily access one of the properties of the request message in a follow up action.

So let’s go ahead and add a schema. In our case, we will only require one property to be send to our logic-app and this property is RequestId. We will be using the property further down the stream to uniquely identify the request and use it to store our ‘session state’.

As such our Json request can be represented as follows:

{

“RequestId”:”2245775543466″

}

Now that we know what the payload message looks like, we need to derive the Json schema. Well luckily for us, we can go to JSONSchema.net and generate a schema. J The generated schema, subsequently would be represented as

{

“type”: “object”,

“properties”: {

“RequestIds”: {

“type”: “string”

}

},

“required”: [

“RequestIds”

]

}

At this point we have all the information required to fill out the ‘Request Body JSON Schema’ section, so all we have to do is copy and paste it into that section.

clip_image020

3. At this point we are ready to proceed with our next step. Which according to our high-level design consists of an activity which composes a new message, which represents the request message (soap) which is to be send to the customer WCF service.

So, let’s proceed and click on the + New Step button
clip_image021

4. Now several options appear, but we are currently only interested in the option ‘Add an action’, so select this.

clip_image022[1]

Step 2: Compose SOAP request message

As part of our last step we clicked on the “new step” button and selected “Add an action”. Which subsequently would display the ‘card selector’ again, only this time displaying available actions to choose from.

Please note: typical actions to choose from would include

· connectors to SaaS services such as Dynamics CRM Online, on premise hosted Line of business applications such as SAP and connectors to existing logic-apps, azure functions and API’s hosted in API Management

· typical workflow actions which allow us to delay processing or even allow us to terminate further processing.

Looking back at our overall scenario which we are about to implement one of the initial actions would be retrieving a list of customers.

In order to retrieve this list of customers we would need to invoke our Customer WCF service, we build earlier. As our WCF service is SOAP based, it requires us to implement one additional step before we can actually invoke the service from within Logic Apps and this steps involves creating the SOAP request message, using a Compose Action.

So from within the ‘Card Selector’ select the compose Action.

clip_image023[1]

Please note: In the near future this additional step will no longer be required as API Management will be able to RESTify your soap endpoints which than can easily consumed from within logicapps (see roadmap). Besides having functionality in API Management, the chances are pretty good as well that a first-class SOAP connector will be added to logic apps in the future as it is ranked high on the logic apps functionality wishlist.

Configure

Once you’ve selected the compose action the following ‘Card’ will show up on in the designer which allows you to compose a Message, which in our case will be the SOAP Request message.

1. The input section allows us to construct the soap (xml) message, which will act as the request which we will be sending to our customer WCF service.

clip_image024[1]

So how would you determine what this message would look like. Well the easiest way would be by using a tool such as SOAPUI which can generate a sample request message. In the previous post, I’ve added a section which explains how to do this and in our scenario the soap request message looks as follow:

<?xml version=”1.0″ encoding=”UTF-8″?>

<Envelope xmlns=”http://schemas.xmlsoap.org/soap/envelope/”>

<Body>

<GetCustomers xmlns=”http://tempuri.org/” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” />

</Body>

</Envelope>

2. Once we have our sample SOAP request message, we simply copy and paste it into the input field.

clip_image025

Please note; once you click on the Inputs section a windows will appear which will allow you to select ‘dynamic content, used within this flow’. This is the ‘intellisense like’ experience I referred to earlier in this post. Anyways we will be ignoring this for now, but in future steps we will be using this.

3. At this point we are ready to proceed with our next step. Which will actually call our customer WCF service.

So, let’s proceed and click on the + New Step button
clip_image021[1]

4. Once again several options appear and once again select the option ‘Add an action’.clip_image022[2]

Step 3: Invoke our Customer WCF Service

After completing step 2 we are now able to actually implement calling our customer WCF service. In order to do so all, we need to do is select the ‘HTTP’ Action from within the ‘Card Selector’

clip_image026[1]

Configure

Once you’ve selected the HTTP action the following ‘Card’ will show up on in the designer which allows you to configure the HTTP request in order to receive the customer information.

clip_image027[1]

As you might remember the custom WCF Service which we are about to invoke uses URL Authorization using Azure Active Directory (see previous post) and as such requires any (POST) request to be authenticated. Long story short; One of the nice things of the HTTP action is that it makes it a breeze invoking web-services even if they require authentication, all we need to do is configure the action correctly and this is done by expanding the advanced options of the HTTP Card, which will allow us to do so.

1. The Method which we need to select is ‘POST’ as we will be posting the soap request to the customer WCF service.

clip_image028[1]

2. The Uri sections allows us to enter the Request URL of the web-service. In our case that would be https://demo-apis.azurewebsites.net/Customers.svc

clip_image029

3. The Headers sections will be used to add both the SOAP Action which needs to be invoked as well as the Content-Type of the actual request message.

The easiest way to retrieve the SOAP Action would be by means of SOAPUI as well. So from within SOAPUI open the request and then select WS-A (bottom menu-bar), and then copy and paste the Action

clip_image030[1]

The header information needs to be passed in as a Json string, and looks as follows

{

“Content-Type”:”text/xml”,

“SOAPAction”:”http://tempuri.org/ICustomers/GetCustomers”

}

clip_image031[1]

4. The body section will contain the message which we composed in the previous step. As such once you click in this section, additional information will displayed on the desiger which allows you to select ‘dynamic content’. (this is the ‘intellisense like’ experience I referred to earlier). From this menu, select the variable ‘’ This variable contains the message which we composed earlier.

clip_image032[1]

5. Now click on the Show Advanced Options, which will allow us to fill out the required authentication information.

clip_image033[1]

6. From the dropdown select Active Directory OAuth

clip_image034[1]

7. For Active Directory OAuth we will require to fill out the Tenant, Audience, Client ID and Secret. This information is to be retrieved as follows

a. In the Azure Portal, go to Azure Active Directory Blade and click on APP Registrations

clip_image036

b. Select the application in question (see previous blog-post) which you registered for the WCF Customer service. In my case demo-apis

clip_image037

c. Now on the settings blade click on Properties and make a note of the following:
Application ID – This is the equivalent of the Client ID

App ID Uri – This is the equivalent of the Audience

clip_image038[1]

d. GO back to the settings blade, click on Keys

clip_image040

e. Now it is time to generate the secret. In order to do this, add a description and select how long the secret should be valid. Once done save the entry and make a note of the value (this is the secret)

clip_image042[1]

f. Now on the portal page, click on the Help Icon and select ‘Show diagnostics’

clip_image043

g. In the window, which pops up, search for tenants. Find your tenant (most likely the one which states ‘isSignedInTenant = true’ and note down the Tenant ID

clip_image044

h. At this point we have all the information in order to fill out the required information

clip_image046

Test

Now that we’ve implemented the call, it would be a good time to go ahead and test the logic app. Luckily for us, this is quite simple.

1. Click on the Save button to save your logic app

2. Now Click on the run button.

clip_image048

3. Wait a few seconds and you should see a debug output. If everything went Ok, it should look similar to the image below.

clip_image050[1]

4. Now click on the HTTP – GetCustomers shape. Which allows you to look at the debug / tracking information. It will show you the input as well as the output information.

clip_image051

5. Now go to the OUTPUTS section and copy and paste the Body section. We will be needing this in Step 4 J

clip_image052

Step 4: Loop over the customer result

Our last step resulted in the fact that we configured our HTTP Action which was responsible for invoking our customer wcf service and returning us a list of customers.

Now in this step we will need to loop over the returned customer list, such that we can enrich each individual record with localization information obtained from a different API.

In order to do so we will have to select a for-each action. This action can be selected by clicking on the “+ New Step button”. Several options will appear of which we need to select the ‘more’ followed with the ‘add a for each’ action.

clip_image053

Configure

1. Once the for-each step has been selected it is being dropped on the designer. The designer than offers us a section in which we can add in input over which we want to loop.

clip_image054

2. If our WCF service would have returned an Json Array object, we would have been able to simply select this output using the ‘Dynamic Content’ selection process (aka intellisense). However in our case the output over which we want to loop is a customer resultset formatted in XML. So, in our case we will need to help the the logic-apps engine a bit, and they way to do this, is by adding a custom expression. Which in our case is a Xpath expression, pointing to the node over which we want to loop.

The xpath expression in our case would be:

/*[local-name()=”Envelope”]/*[local-name()=”Body”]/*[local-name()=”GetCustomersResponse”]/*[local-name()=”GetCustomersResult”]/*

Easiest way to test this xpath expression, would be by using the response message we extracted when we tested our logic app earlier and subsequently use an online tool to test the xpath expression.

Now that we have our xpath expression, we can use it in the following Logic App Expression

@xpath(xml(body(‘Replace with the name of action of which we want to use the response’,’Xpath Expression’)

In my scenario the expression would be as follows

@xpath(xml(body(‘HTTP_-_GetCustomers’)), ‘/*[local-name()=”Envelope”]/*[local-name()=”Body”]/*[local-name()=”GetCustomersResponse”]/*[local-name()=”GetCustomersResult”]/*’)

clip_image056[1]

Step 5: Extract individual customer info

In our previous step we instantiated our for-each loop which will loop over our xml result set. Now our next step is to extract the individual customer info and store it in a intermediate json format which we will be using in subsequent actions.

So from within our for-each action, select the Add an action.

clip_image058

From within the ‘Card Selector’ select the compose Action.

clip_image023[2]

Configure

Once you’ve selected the compose action the following ‘Card’ will show up on in the designer which allows you to compose a Message, which in our case will be a custom Json message which holds the individual customer information consisting of CustomerId, FirstName, LastName and PostCode

Note: As in Step 4 when configuring the for-each iteration path. We will be leveraging xpath expressions in order to extract the individual customer data. Alternatively, I could have leveraged an Azure Function to convert the received XML Customer response into JSON or I could have leveraged API Management which by means of policies can perform conversion from xml to json out of the box. In my next post (part 3 of this series) I will be using this.

.

1. The input section allows us to construct our custom Json message which holds the individual customer information consisting of CustomerId, FirstName, LastName and PostCode

clip_image024[2]

2. In order to extract the required fields from the xml will be leveraging the following xpath queries

a. customerId extraction:

string(/*[local-name()=”CustomerData”]/*[local-name()=”CustomerId”])’)

b. FirstName extraction:

string(/*[local-name()=”CustomerData”]/*[local-name()=”FirstName ”])’)

c. SurName extraction:

string(/*[local-name()=”CustomerData”]/*[local-name()=”SurName”])’)

d. PostCode extraction:

string(/*[local-name()=”CustomerData”]/*[local-name()=”PostCode”])’)

the logic app expression which we will be leveraging to extract a value using xpath will be

@{xpath(xml(decodeBase64(item().$content)), ‘Xpath Expression‘) where item() refers to the current item (customer record) in the loop and $content represents the content (customer record xml part)

Combined in a Json construct the complete message construction would look like (note that we escape using )

{

“CustomerId”: “@{xpath(xml(decodeBase64(item().$content)), ‘string(/*[local-name()=”CustomerData”]/*[local-name()=”CustomerId”])’)}”,

“FirstName”: “@{xpath(xml(decodeBase64(item().$content)), ‘string(/*[local-name()=”CustomerData”]/*[local-name()=”FirstName”])’)}”,

“LastName”: “@{xpath(xml(decodeBase64(item().$content)), ‘string(/*[local-name()=”CustomerData”]/*[local-name()=”SurName”])’)}”,

“PostCode”: “@{xpath(xml(decodeBase64(item().$content)), ‘string(/*[local-name()=”CustomerData”]/*[local-name()=”PostCode”])’)}”

}

clip_image059

Test

Now that we’ve implemented the xml extraction within the for-each, it would be a good time to go ahead and test the logic app, and see if everything works accordingly.

1. Click on the Save button to save your logic app

2. Now Click on the run button.

clip_image048[1]

3. Wait a few seconds and you should see a debug output. If everything went Ok, it should look similar to the image below.

clip_image060[1]

4. As you can see the last item in the flow, contains a Json output depicting the customer values extracted.

Step 6: Invoke the postcodeapi

Now that we have extracted our customer data and stored it in a json format. We can proceed with the next step, which invokes invoking a public postcode api. In order to do so we will once again select the HTTP Action within the ‘Card Selector’

clip_image026[2]

Configure

Once you’ve selected the HTTP action the following ‘Card’ will show up on in the designer which allows you to configure the HTTP request in order to receive localization information based on a postal code.

clip_image027[2]

1. The Method which we need to select is ‘GET as we will be retrieving data from a rest endpoint.

clip_image061[1]

2. The Uri sections allows us to enter the Request URL of the web-service. In our case that would be http://v0.postcodeapi.com.au/suburbs.json?postcode=XXXXX where XXXX is a dynamic parameter, to be more specific; we will be using the PostCode field which we extracted in step 5. In order to use this PostCode value we will

a. Enter the value http://v0.postcodeapi.com.au/suburbs.json?postcode= in the Uri field.

b. Select the dynamic content ‘Outputs’ from the Extracted xml

We are currently not able to directly access the PostCode field from within the designer as the designer currently is not aware of this property. It is only aware of the fact that the ‘compose step – Extracted xml’ has a output which is a ‘message’ and as such we can only select the complete message.

Note: In a future release of logic-apps this experience will be improved and additional magic will be added such that the designer can ‘auto-discover’ these message properties. How this will be implemented is not 100% clear, but one of the possibilities would be; that we would manually add a ‘description’ of the output (Json schema, for example) to the compose action or any other action which returns / creates an object.

clip_image062

3. In order to select the PostCode field from the Outputs, we will be needing to switch to Code View.

clip_image064

4. Once in code view, find the code block which contains the http://v0.postcodeapi.com.au/suburbs.json?postcode= url. Once found we simple modify the code from
http://v0.postcodeapi.com.au/suburbs.json?postcode=@{outputs(‘Extracted_xml’)}

to

http://v0.postcodeapi.com.au/suburbs.json?postcode=@{outputs(‘Extracted_xml’).PostCode}

clip_image066

5. Now go back to the designer

clip_image068[1]

6. And behold the designer now states “http://v0.postcodeapi.com.au/suburbs.json?postcode={} PostCode”

clip_image070

Test

Now that we’ve implemented the postcode api call, it would be a good time to go ahead and test the logic app.

1. Click on the Save button to save your logic app

2. Now Click on the run button.

clip_image071[1]

3. Wait a few seconds and you should see a debug output. If everything went Ok, it should look similar to the image below. If you expand the HTTP action, you wull notice that the URI now is composed using the extracted PostCode value

clip_image072[1]

Step 7: Compose an enriched customer message

Now that we have invoked the postcode API it is time to combine both the original customer data and the postcode data. In order to do this, we will be composing a new Json message using the Compose Action.

From within the ‘Card Selector’ select the compose Action.

clip_image023[3]

Configure

Once you’ve selected the compose action the following ‘Card’ will show up on in the designer which allows you to compose a Message, which in our case will be a new Json message which holds both the customer data as well as the location data retrieved from the PostCode lookup.

1. The input section allows us to construct our custom Json message which will hold all the combined data

clip_image024[3]

2. Now copy and paste the below ‘sample’ json message into the input section

This message will be of the following structure:

{

“FirstName”: “## FirstName from the WCF Customer web service##”,

“LastName”: “## LastName from the WCF Customer web service##”,

“Location”: {

“Latitude”: “## Latitude obtained from the postal api##”,

“Locality”: “## Locality obtained from the postal api##”,

“Longitude”: “## Longitude obtained from the postal api##”,

“PostCode”: “# PostCode from the ‘extract_xml’ message##”,

“State”: “## State obtained from the postal api##”,

“Suburb”: “## Suburb obtained from the postal api##”

},

“RequestId”: “## Obtained from the request trigger##”,

“id”: “# CustomerId from the ‘extract_xml’ message##”

}

clip_image073

3. Now go to code view

clip_image074

4. Once in code view, find the code block which represents the Json message which we just copied and pasted in the input section.

Note: In a future release of logic-apps this experience will be improved and additional magic will be added such that the designer can ‘auto-discover’ these message properties, which we will now add manually. How this will be implemented is not 100% clear, but one of the possibilities would be; that we would manually add a ‘description’ of the output (Json schema, for example) to the compose action or any other action which returns / creates an object.

clip_image076

5. Now replace the json such that it looks like depicted below

“Enrich_with_postal_code”: {

“inputs”: {

“FirstName”: “@{outputs(‘Extracted_xml’).FirstName}”,

“LastName”: “@{outputs(‘Extracted_xml’).LastName}”,

“Location”: {

“Latitude”: “@{body(‘HTTP’)[0].latitude}”,

“Locality”: “@{body(‘HTTP’)[0].locality}”,

“Longitude”: “@{body(‘HTTP’)[0].longitude}”,

“PostCode”: “@{outputs(‘Extracted_xml’).PostCode}”,

“State”: “@{body(‘HTTP’)[0].state.name}”,

“Suburb”: “@{body(‘HTTP’)[0].name}”

},

“RequestId”: “@{triggerBody()[‘RequestId’]}”,

“id”: “@{outputs(‘Extracted_xml’).CustomerId}”

},

“runAfter”: {

“HTTP”: [

“Succeeded”

]

},

“type”: “Compose”

},

clip_image078

Test

Now that we’ve composed a message containing both the WCF and PostCode API data, it would be another good time to go ahead and test if everything works and this time we will be testing our logic app using Fiddler

1. Download Fiddler, if you already have.

2. Go to you logic app and expand the Request Trigger and press on the “Copy Icon”, this will copy the logic app endpoint to your clipboard.

clip_image079

3. Open fiddler, and select the composer tab

clip_image080

4. In the composer

a. Set the HTTP Action to POST

b. Copy and Paste the uri in the Uri field

c. In the header section add

i. Content-Type:application/json

d. In the body section add the following json

{

“RequestId”:”20161220″

}

e. Click on the Execute button

clip_image082

5. Now go back to your Logic App

6. In the run history, select the last entry

clip_image084

7. If everything went Ok, it should look similar to the image below.

clip_image085

Step 8: Store Session State in DocumentDB

At this point we have implemented functionality which

· allows us to iterate over all the customer records

· retrieve localization data from the postal code api using the postal code extracted from the customer record.

· Compose a new message which contains all the data.

The functionality which is left to implement at this point in time consists of; combining all the composed new messages, containing the customer and localization data, in one document and returning it to the caller.

Note: Currently there is no easy way of doing this directly from within logic-apps as logic apps currently does not contain the functionality which would allow us to ‘combine the data’ in memory. But have no fear; in one of the next releases of Logic Apps will have support for storing session state and once this is available we will no longer require this additional step, which is explained below.

Configure

As Logic Apps currently has no means of storing session state, we will be resorting to an external session state store. In our case, the most obvious choice would be DocumentDB.

So before we proceed, let’s go and create a DocumentDB service.

1. Go to the Azure Portal and click on the New Icon

clip_image086

2. Search for DocumentDB

clip_image088

3. Select DocumentDB from Publisher Microsoft

clip_image090

4. Fill out the required information and once done create the DocumentDB instance

clip_image092

5. After creation has completed, open the DocumentDB Instance.

clip_image094

6. Now Add a Collection

clip_image096

7. Fill out the required information for the Collection Creation, and press OK once done

clip_image098

8. Go back to the main DocumentDB Blade, and click on Keys

clip_image100

9. From within the Keys, Copy and Paste the Primary or Secondary Key

clip_image102

10. Now go back to our logic app, and open it in the designer

11. In the Logic App, Click on the Add New Item

12. Now search for DocumentDB Actions and select “Azure DocumentDB – Create or update document”

clip_image103

13. The connector will now be displayed and will require some configuration

clip_image104

14. Fill out the required information. For which it has to be noted that for Database Account Nam is the actual name of the documentDB. In my case docdb-playground.clip_image106

15. Once filled out the information should look similar to the one depicted below in the image

clip_image107

16. At this point the connection has been created, and we can now proceed with the actual configuration in which we will

a. select the correct Database ID from the dropdown

b. select the collection to use

c. add the dynamic content (message) which we want to store

d. we set the value to True for IsUpsert

clip_image109

Step 9: Exit Loop, Retrieve and return Stored Data

Our last step resulted in the fact that we persisted all documents into DocumentDB. Now before we proceed, let’s have a look at Step 7 in which we composed the following message, which eventually was stored in DocumentDB.

clip_image111

Have a good look at the field: RequestId. This field is actually passed in whenever we invoke our LogicApp. (see step 7, the test section).

There was a reason why we added this field and have it stored in DocumentDB. The reason? Well this way we are able to select all documents stored in DocumentDB belonging to the specific ID of the current Request and return them to the caller.

Configure

1. Select the Add an action button located just below the for-each scope.

2. Now search for DocumentDB Actions and select “Azure DocumentDB – Query documents”

clip_image112

3. The Document DB Query Documents connector, can now be configured as follows

a. Select the correct database ID from the dropdown in our case ProcessingState

b. Select the applicable collection from the dropdown in our case LogicApp

c. Now add a query, which will return all documents stored in the collection which have the same request id.

SELECT c.Id as CustomerId, c.FirstName,c.LastName,c.Location FROM c where c.RequestId = …..

d. Where c.RequestId = “ SELECT REQUEST ID from the Dynamic Content window”

clip_image114

4. At this point we have completed the action which will retrieve all the applicable stored documents. So the only thing which is left to do is, returning this list of document back to the caller. In order to do this, we add one more action. This action is called Response

clip_image115

5. The Response action, can can now be configured as follows

a. Enter 200 for the return status code, this indicates the HTTP Status code ‘OK’

b. In the response header we will need to set the content-type. We will do this by adding the following piece of json

{ “Content-Type”:”application/json” }

c. In the body we will add the dynamic content which relates to documents which were returned from document DB

clip_image117

Test

Well now that we have implemented the complete flow, it is time to do our final test and once again we will be using Fiddler to perform this test.

1. Open fiddler, and select the composer tab

clip_image080[1]

2. In the composer

a. Set the HTTP Action to POST

b. Copy and Paste the uri in the Uri field

c. In the header section add

i. Content-Type:application/json

d. In the body section add the following json

{

“RequestId”:”20161221″

}

e. Click on the Execute button

clip_image082[1]

3. Now open the result and you should see a response similar to the one below

clip_image119

4. No go back to your logic app and in the run history, select the last entry

clip_image120

5. If everything went Ok, it should look similar to the image below.

clip_image122

Conclusion

This post has guided you through setting up a logic app which calls two api’s a, combines the data and returns the aggregated result back to the caller.

In my next post I will introduce API Management into the mix which will be using to expose the two api mentioned and apply some api management magic which further simplify our logic app implementation.

So until next time, stay tuned.

Cheers

René

BizTalk 2016 – SFTP Adapter – How to set it up properly

Cutting a long story short I was exploring migrating a BizTalk 2013 solution to BizTalk 2016.  The solution uses SFTP and I wanted to migrate from the open source Blogical SFTP adapter to the out of the box SFTP adapter which has had some enhancements.  The documentation around this enhancement and in particular the setup is lacking.  It will not work out of the box from a BizTalk install unless you additionally setup WinSCP.  The problems are:

  1. The BizTalk documentation lists the wrong version on the dependencies, you need 5.7.7 not 5.7.5
  2. The WinSCP pages cover lots of other versions and it can be confusing because while their documentation is good and detailed, it talks about using WinSCP with the GAC which you will assume you need for BizTalk.  This is a red herring so dont read this bit unless you need to
  3. There is WinSCP and WinSCP .net library.  I got confused between the two and didnt initially realise I need both

To get this up and running the steps you need to follow are:

  1. Download WinSCP and the .net Library making sure you get the right versions
  2. Copy the .exe and .dll to the BizTalk installation folder
  3. DO NOT gac anything.  If you GAC the .net library it will not work because it expects WinSCP.exe to be in the same path so thats why they both go in the BizTalk installation folder

This is pretty simple enough in the end but I felt the easiest way to ensure you get the right version is to use NuGet to get version 5.7.7 and to make your life even simpler just use the powershell script below once you have setup your BizTalk environment and it will do it for you.  You just need to set the path for your BizTalk install folder and also where you want to download NuGet stuff to.

#Parameters
$downloadNoGetTo = “F:SoftwareWinSCP”
$bizTalkInstallFolder = “F:Program Files (x86)Microsoft BizTalk Server 2016”

#Download NuGet
Write-Host “Downloading Nuget”
$sourceNugetExe = “https://dist.nuget.org/win-x86-commandline/latest/nuget.exe”
$targetNugetExe = “$downloadNoGetTonuget.exe”
Invoke-WebRequest $sourceNugetExe -OutFile $targetNugetExe

#Download the right version of WinSCP
Write-Host “Downloading WinSCP from NuGet”
Invoke-Expression “$targetNugetExe Install WinSCP -Version 5.7.7 -OutputDirectory $downloadNoGetTo”

#Copy WinSCP items to BizTalk Folder
Write-Host “Copying WinSCP Nuget to BizTalk Folder”
Copy-Item “$downloadNoGetToWinSCP.5.7.7contentWinSCP.exe” $bizTalkInstallFolder
Copy-Item “$downloadNoGetToWinSCP.5.7.7libWinSCPnet.dll” $bizTalkInstallFolder

CS0006 Error – Metadata file could not be found

CS0006 Error – Metadata file could not be found

In the last 2 days, I struggled against this error and it was a real nightmare and, because the solution is really wired and complicate to find I decided to keep note about that in my blog and hopefully help some other people.
One day and without any real specific reason I was not able to build my solution anymore, hundreds of CS0006 errors, so my first and usual actions were:

  1. Clean the solution, nothing…
  2. Restart VS, nothing…
  3. Clean and restart the solution, nothing…

At that point I started looking in the network and I found so many articles, tips and hints like, remove all the references and re add again, check the build settings in the configuration manager, fix precedencies but nothing and, to be honest, with 90 projects in the solution, was not a feasible option for me.

Two main assumptions drive me to the solution:

  1. Previously the solution was able to build and I didn’t touch any code
  2. I copied the solution in another machine and, with my surprise, it was able to build.

I used process explorer to check what Visual Studio does during the compilation and I noticed a quite interesting number of temporary files related to the nugget packages were created.
The length of the path generated was interesting so I decided for moving my project folder in the C: ROOT and the problem has been solved.

The biggest problem I think was because I added a new nugget package to a project and VS was not able to generate specific temp files and unfortunately, I didn’t receive any exception or warning about that.

So, in case of CS0006 error the first test to do is trying to copy the solution in a shorter path and hopefully is going to fix the issue.

Author: Nino Crudele

Nino has a deep knowledge and experience delivering world-class integration solutions using all Microsoft Azure stacks, Microsoft BizTalk Server and he has delivered world class Integration solutions using and integrating many different technologies as AS2, EDI, RosettaNet, HL7, RFID, SWIFT.