Azure Functions to validate XML against DTD

Azure Functions to validate XML against DTD

by | Feb 14, 2024 | BizTalk Community Blogs via Syndication

After the release of a set of Azure Functions that will help us minimize or completely remove the need for an Integration Account:

Today, I’m going to release a new function – validate XML against DTD – that will bring additional capabilities to Logic App Consumption and Standard since this functionality is not currently supported in either of the tiers nor with the support of the Integration Account.

DTD? What is a DTD?

Yes, this is probably old school, which is not often used nowadays. But DTD, which stands for Document Type Definition, allows you to define the structure and the legal elements and attributes of an XML document.

This is a sample of a DTD file:

And this is a sample of an XML message with a reference to a DTD:




   My stock
   nine
   (099) 999-9999

The DOCTYPE declaration above contains a reference to a DTD file.

Although the use of DTDs is not very frequent to see these days, it is still very common to encounter this in RosettaNet PIPs.

Validate XML against DTD

A Document Type Definition (DTD) is a document that describes the structure of an XML document, what elements and attributes it contains, and what values it may have. DTDs form part of the W3C’s XML Standard but are typically considered to be a separate schema technology and are not typically used in conjunction with other schema formats like XSD and so on.

A DTD document can be embedded within an XML file or can exist on its own. When it is not embedded, normally, there are two ways to reference the DTD:

  • Using the PUBLIC keyword: This format is generally used to declare publicly available DTDs, standard character sets, and commonly used notations
  • Or using the SYSTEM keyword: These entities are not assumed to be known to a receiving system. Thus, such entities require a full declaration of system identification (path, etc.) when they are exchanged.
    • The SYSTEM identifier specifies the location of the DTD file. Since it does not
      start with a prefix like http:/or file:/, the path is relative to the location of
      the XML document.

This Azure Function allows you to perform XML validations against a DTD file. The function only accepts DTDs defined using the SYSTEM keyword.

To trigger this function, you need to:

  • In the Body, the XML payload that you want to be validated.
  • You should specify the following mandatory headers:
    • Content-Type as text/xml (or application/xml).
    • DTDFileName with the name of the DTD file present in the storage account.

The response will be a:

  • 200 OK – Validation successful. If it’s a valid message.
  • Or 400 Bad Request with a list of errors if there is something invalid.

Where can I download it?

You can download the complete Azure Functions source code here:

Hope you find this helpful! So, if you liked the content or found it useful and want to help me write more, you can buy (or help me buy) my son a Star Wars Lego! 

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.

He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.
View all posts by Sandro Pereira

Azure Function to Apply XML Validation (Advanced)

Azure Function to Apply XML Validation (Advanced)

by | Feb 13, 2024 | BizTalk Community Blogs via Syndication

After the release of our previous XML Validation Functions: 

It is now time to release our last Azure Function under the same context: an Azure Function to Apply XML Validation (Advanced).

As I explained in my previous posts, all the out-of-the-box Azure Integration Services capabilities to validate XMLs have a huge limitation: they don’t allow us to have a chain of XML Schemas! As I also mentioned, this is a common feature present in many Enterprise XML Schemas definitions. EDI or RosettaNet Schemas may have 2 or more schemas that define the overall structures of the messages. Blocking this way many enterprise scenarios that we need to address in our Azure integration solutions.

Our previous Azure Function allows us to solve many of those scenarios, and it can be used inside Logic Apps Consumption or Standard or even inside API Management. However, it also has a limitation: it only allows first-level chain support for XML Schemas (meaning that it will only take into consideration all the import schemas of the main XML Schema).

However, for example, in RosettaNet Schemas, it is very common that our main schema imports a “child” XML Schema and that child schema imports or includes other XML Schemas itself.

Apply XML Validation (Advanced)

What does this Azure Function do?

This Azure Function allows you to perform XML validations against an XML Schema, including support for all chains of XML Schemas. That means that it will take into consideration all depth of importation for a specific type of message. It will recursively include or import all XML Schemas, supporting this way all types of XML message validation.

To trigger this function, you need to:

  • In the Body, the XML payload that you want to be validated.
  • You should specify the following mandatory headers:
    • Content-Type as text/xml (or application/xml).
    • SchemaFileName with the name of the XML Schema (XSD) file present in the storage account.

The response will be a:

  • 200 OK – Validation successful. If it’s a valid message.
  • Or 400 Bad Request with a list of errors if there is something invalid.

Notice that by default, if you send a message to be validated against a schema that doesn’t have the same target namespace and root node, the response will be an OK. This function also validates that type of message, so if you send a message that doesn’t correspond to that schema, you will get a 400 Bad Request.

Where can I download it?

You can download the complete Azure Functions source code here:

Hope you find this helpful! So, if you liked the content or found it useful and want to help me write more, you can buy (or help me buy) my son a Star Wars Lego! 

Thanks to my team member Luís Rigueira for helping me realize and implement this idea.

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.

He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.
View all posts by Sandro Pereira

Azure Function to Apply XML Validation (Intermedium)

Azure Function to Apply XML Validation (Intermedium)

by | Feb 12, 2024 | BizTalk Community Blogs via Syndication

After the release of our Azure Function to Apply XML Validation (Basic), it is now time for another Azure Function under the same context: an Azure Function to Apply XML Validation (Intermedium).

You may be wondering why a new Azure Function is required to archive the same? And why not only one with all capabilities?

Both are good questions that I will be happy to respond to. First of all, in my last blog post, I mentioned that, in the next few days, we will be releasing two additional versions of this function with more functionalities/capabilities. And to answer the second question, the main reason why I decided to do 3 different versions is performance. The basic function has fewer capabilities, but it will have better performance. Of course, the advanced function will have all the capabilities, but it will have a small overhead in the overall performance.

The previous Azure Function is great for basic validations, similar to what we can archive using the default out-of-the-box capabilities inside:

  • Logic Apps Standard;
  • Logic Apps Consumption using the Integration Account;
  • Or in API Management

Of course, the basic XML Validation Function it is a good approach to replace the dependency of the Integration Account in Logic App Consumption.

But all of them have a huge limitation! None of those services allow a chain of XML Schemas!

What do you mean by a chain of XML Schemas?

XML Schema provides mechanisms to include or import other XML Schema documents, enabling the reuse and extension of schema definitions across multiple files. This capability is essential for managing complex schemas in a modular and maintainable manner.

  1. Include: The include element is used when you want to incorporate definitions from another schema that is in the same target namespace. By using include, you can split your schema definitions into separate, smaller files for better manageability and readability while treating them as part of a single schema during validation. The included schema essentially becomes a subset of the including schema, allowing for the extension or redefinition of elements and types within the same namespace.
  2. Import: The import element is used to incorporate definitions from another schema that is in a different target namespace or from no namespace into the current schema. This allows you to reference and use types and elements defined in an external schema within your current schema document. Importing is crucial when you need to integrate or reference types defined in a completely separate schema, possibly managed by a different organization or standard body.

Both include and import mechanisms facilitate the construction of complex XML schemas from modular components, promoting reuse and simplifying the management of schema definitions. They enable schema designers to build upon existing standards and to organize their schema definitions logically and efficiently.

This a common feature present in many Enterprise XML Schemas definitions. EDI or RosettaNet Schemas may have 2 or more schemas that define the overall structures of the messages.

Apply XML Validation (Intermedium)

What does this Azure Function do?

This Azure Function allows you to perform XML validations against an XML Schema, including first-level chain support for XML Schemas. That means that it will take into consideration all the import schemas of the main XML Schema

To trigger this function, you need to:

  • In the Body, the XML payload that you want to be validated.
  • You should specify the following mandatory headers:
    • Content-Type as text/xml (or application/xml).
    • SchemaFileName with the name of the XML Schema (XSD) file present in the storage account.

The response will be a:

  • 200 OK – Validation successful. If it’s a valid message.
  • Or 400 Bad Request with a list of errors if there is something invalid.

Notice that by default, if you send a message to be validated against a schema that doesn’t have the same target namespace and root node, the response will be an OK. This function also validates that type of message, so if you send a message that doesn’t correspond to that schema, you will get a 400 Bad Request.

Where can I download it?

You can download the complete Azure Functions source code here:

Hope you find this helpful! So, if you liked the content or found it useful and want to help me write more, you can buy (or help me buy) my son a Star Wars Lego! 

Thanks to my team member Luís Rigueira for helping me realize and implement this idea.

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.

He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.
View all posts by Sandro Pereira

Azure Function to Apply XML Validation (Basic)

Azure Function to Apply XML Validation (Basic)

by | Feb 11, 2024 | BizTalk Community Blogs via Syndication

After the release of our two previous Azure Functions that will help us minimize or completely remove the need for an Integration Account:

Today is the correct time to release another Azure Function that will replace another Integration Account functionality: an Azure Function to Apply XML Validation.

Of course, you can apply out-of-the-box this functionality in:

  • Logic App Standard (without the need for an Integration Account)

However, Logic App Consumption requires an Integration Account to provide those same capabilities out of the box.

Once again, our main objective in creating this specific version of this function was to use it inside Logic Apps Consumption to avoid needing an Integration Account. But that we will address later on in another blog post.

If you are wondering what I mean by this specific version of this function, well, in the next few days, we will be releasing two additional versions of this function with more functionalities. But, once again, that we will address later on in another blog post.

Apply XML Validation (Basic)

XML (Extensible Markup Language) validation is the process of checking an XML document against a set of rules to ensure its structure and content adhere to a specific format or standard. This process is crucial for ensuring that the XML document is both well-formed and valid.

  1. Well-formed XML: This means that the XML document follows the basic syntax rules laid out by the XML specification. These rules include proper nesting of elements, correct use of opening and closing tags, attribute value quoting, and more. A well-formed XML document is one that can be correctly parsed and understood by an XML parser.
  2. Valid XML: Beyond being well-formed, a valid XML document also adheres to a specific schema or Document Type Definition (DTD) that defines the structure, content, and relationships within the document. Validation against a schema or DTD ensures that the XML document contains the expected elements, attributes, and data types, and that these components are organized in a defined way.

There are several schema languages used for XML validation, with the most common being:

  • DTD (Document Type Definition): An older schema language that defines the structure and allowed content within an XML document. – Not supported in the Azure Function.
  • XML Schema (also known as XSD): A more powerful and expressive schema language that allows for more detailed specifications of the content and structure, including data types and namespace support.

XML validation is performed using XML parsers or validation tools, which can programmatically check a document against its DTD or XSD to ensure compliance. This is a critical step in many data exchange, configuration management, and content authoring workflows, ensuring that the data is correctly structured and interpretable by receiving systems or applications.

What does this Azure Function do?

This Azure Function allows you to perform basic XML validations against an XML Schema.

To trigger this function, you need to:

  • In the Body, the XML payload that you want to be validated.
  • You should specify the following mandatory headers:
    • Content-Type as text/xml (or application/xml).
    • SchemaFileName with the name of the XML Schema (XSD) file present in the storage account.

The response will be a:

  • 200 OK – Validation successful. If it’s a valid message.
  • Or 400 Bad Request with a list of errors if there is something invalid.

Notice that by default, if you send a message to be validated against a schema that doesn’t have the same target namespace and root node, the response will be an OK. This function also validates that type of message, so if you send a message that doesn’t correspond to that schema, you will get a 400 Bad Request.

Where can I download it?

You can download the complete Azure Functions source code here:

Hope you find this helpful! So, if you liked the content or found it useful and want to help me write more, you can buy (or help me buy) my son a Star Wars Lego! 

Thanks to my team member Luís Rigueira for helping me realize and implement this idea.

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.

He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.
View all posts by Sandro Pereira

API Management Best Practices, Tips, and Tricks: #1 How to validate if a Header is an empty string

API Management Best Practices, Tips, and Tricks: #1 How to validate if a Header is an empty string

by | Oct 23, 2023 | BizTalk Community Blogs via Syndication

If you follow my blog posts, you will not be surprised that I love to speak about Best Practices, Tips, and Tricks. I wrote in the past Power Automate Best Practices, BizTalk Server, and often about Logic Apps Best Practices, Tips, and Tricks. So it’s not surprising that I would address other Azure Integration Services sooner or later, this time Azure API Management!

Let’s embark on a new journey as we begin a series of blog posts dedicated to API Management Best Practices, Tips, and Tricks. To start this series, I chose a topic that I found very useful in many situations: How to validate if a Header is an empty string?

#1 How to validate if a Header is an empty string?

Verifying whether the header contains a specific value(s) is straightforward. In fact, Microsoft’s documentation covers this as the very first topic on their page. To accomplish that, we can use the check-header policy to enforce that a request has a specified HTTP header.


	application/json
	plain/text

In this case, our request needs to have the Content-Type header, and the accepted values are:

  • application/json
  • or plain/text

If you exclude this header or opt for an alternative value for it, you’ll trigger a 401 error due to the missing Content-Type header:

{
	"statusCode": 401,
	"message": "Missing Content-Type header"
}

Now, checking only if it exists without any value restriction is poorly explained in MSFT documentation. However, it is still very simple to implement since values are options on the check-header policy. To accomplish that, we just need a simple instruction line:

What you may find less commonplace is the technique for confirming the existence of a Header that isn’t empty. To clarify:

  • The header needs to exist;
  • The header can contain any value except being null, an empty string, or a string containing only blank spaces.

Well, of course, you may find multiple ways to accomplish this task, even using the check-header policy to validate the existence of the header and another strategy to validate if the value was not empty. I decided in my approach to suppress the use of the check-header policy, and instead, I will use the following approach:

  • Read the Header using the context.Request.Headers.GetValueOrDefault(“my-header”) expression.
  • And then using a condition to see if the value is empty or not

You can accomplish that by using the following policy:



	<when condition="@(context.Variables.GetValueOrDefault("myHeaderValue") == null 
			|| string.IsNullOrEmpty(context.Variables.GetValueOrDefault("myHeaderValue")) 
			|| context.Variables.GetValueOrDefault("myHeaderValue").Contains("undefined"))"> 
		
			
			
				application/json
			
			{
				"error": {
					"code": "401",
					"message": "My-Header header is missing or empty"
				}
			}

I hope you enjoy this tip and stay tuned for the following Azure API Management Best practices, Tips, and Tricks.

If you liked the content or found it helpful and want to help me write more content, you can buy (or help buying) my son a Star Wars Lego! 

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.

He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.
View all posts by Sandro Pereira

Checking for the existence of a property by Thomas Canter

Checking for the existence of a property by Thomas Canter

by | Sep 4, 2023 | BizTalk Community Blogs via Syndication

Today, I will bring back to life another old BizTalk Server blog post by an old friend of mine, Thomas Canter, with his permission, that I find pretty interesting and useful: Checking for the existence of a property. This was initially published on http://geekswithblogs.net/ThomasCanter, now retired.

NOTE:

  • The variable PropExists as bool has already been created
  • The Property of interest is BTS.RetryCount
  • The Message is Message_In

The list from Using Operators in Expressions (https://learn.microsoft.com/en-us/biztalk/core/using-operators-in-expressions) has the typical list of stuff that you expect in C#, multiplication, bit operations (shift left and right), and Boolean operators, but a couple of extremely useful constructs are available that are unique to BizTalk.

The most important of these (in my humble opinion) is the exists operator.

As you are all aware, to even check whether a property exists in an expression throws an exception… as in the following case:

PropExists = (Message_In(BTS.RetryCount) != null && Message_In(BTS.RetryCount) != “”);

If BTS.RetryCount does not exist in the message context, then the MissingPropertyException (Windows Event Log Event ID 10019) is thrown.

Without having to resort to a scope shape and exception handler, the exists operator allows you to check if a property exists in a message and is used in the following format:

PropExists = BTS.RetryCount exists Message_In;

OR

if (BTS.RetryCount exists Message_In)
{
     …;
}

Conclusion

Using the XLANG/s exists operator in your orchestration allows you to test for the existence of a property in a message without resorting to a scope shape and exception handler.

Below are a few more XLANG/s functions that can provide some value to your Orchestrations:

Operator Description Example
checked() raise error on arithmetic overflow checked(x = y * 1000)
unchecked() ignore arithmetic overflow unchecked(x = y * 1000)
succeeded() test for successful completion of transactional scope or orchestration succeeded()
exists test for the existence of a message context property BTS.RetryCount exists Message_In

Hope you find this helpful! So, if you liked the content or found it helpful and want to help me write more content, you can buy (or help buy) my son a Star Wars Lego! 

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.

He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.
View all posts by Sandro Pereira

CSV Structure Validation Pipeline Component

CSV Structure Validation Pipeline Component

by | Jan 22, 2023 | BizTalk Community Blogs via Syndication

I’m back to another blog post about BizTalk Server! I know that my latest post has been about Azure Logic Apps, and you may count on seeing many more in the future. And the reason is that I work both on-premise with BizTalk Server and in the cloud with Azure Integration Services… but relax, I will continue to post many things about BizTalk Server. BizTalk Server is not dead. It is well alive, contrary to what many think!

Today I’m returning to one of my pet projects: the BizTalk Pipeline Components Extensions Utility Pack project that is available on GitHub!

For those who aren’t familiar with it, the BizTalk Pipeline Components Extensions Utility Pack project is a set of custom pipeline components (libraries) with several custom pipeline components that can be used in receive and sent pipelines. Those pipeline components provide extensions of BizTalk’s out-of-the-box pipeline capabilities.

CSV Structure Validation Pipeline Component

The CSV Structure Validation Pipeline Component is a pipeline component that can be used to validate the structure of a basic CSV or flat file component before being converted to an XML message. Of course, this same strategy can be used in more complex scenarios.

This is the list of properties that you can set up on the CSV Structure Validation pipeline component:

Property Name Description Sample Values
DelimiterChar Define what is the delimiter char inside a line. ;
NumberFieldsPerLine Number of fields expected per line 3

Note that this component takes as granted that the line delimiter is the CRLF (Carriage ReturnLine Feed).

If we take this example:

one;two;t
1;2;2

Then we need to configure the port in the following way:

If we receive an invalid file, then the component will raise an error suspending the message in the BizTalk Server Administration Console. For example, with the following error message:

  • Invalid format data in the document. Line number 3 has 2 fields, and it should be expected 3 fields

If you are wondering why create a Pipeline component to validate the structure of the CSV or flat-file document? Can we use instead the Flat-File Schema to do this structure validation?

And the answer is Yes and No! In many cases and with many requirements, we don’t need to create a custom pipeline component. Using a Flat-File schema can be used to address the goals but in other scenarios doing a CSV Validation with the Flat-File schema is not enough. However, I will leave that “discussion” to my next BizTalk Server Best practices, tips and tricks.

How to install it

As always, you just need to add these DLLs on the Pipeline Components folder that in BizTalk Server 2020 is by default:

  • C:Program Files (x86)Microsoft BizTalk ServerPipeline Components

In this particular component, we need to have this  DLL:

  • BizTalk.PipelineComponent.CSVFlatFileStructureValidation.dll

How to use it

Like all previous, to use the pipeline component, I recommend you to create generic or several generic pipelines that can be reused by all your applications and add the Message Archive Pipeline Component in the stage you desire. The component can be used in a stage of the receive and send pipelines.

Download

THIS COMPONENT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND.

You can download CSV Structure Validation Pipeline Component from GitHub here:

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.

He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.
View all posts by Sandro Pereira