Let’s see one of the projects I’ve made with the Business Rule Engine from the Microsoft BizTalk tool set.
There are the EDI data in”very flexible format”. The source partners fill up the fields of these documents by different applications and peoples. As a result the data are usually spread through the several fields if they filling application has a limit to the field size.
For example:
Good data:
N1*CN*YOUNG BAY INTERNATIONAL INC
N3*2655 ST-JACQUES#101*MONTREAL QC H3J 1H8 CANADA
N3*TEL:514-8121887 514-9313157
Bad data:
N1*CN*AFS -VISA INTERNATIONAL (HK) LTD SUITE NO.1 29/F SKYLINE T
N3*OWER 39 WANG KWONG ROAD, KOWLO
N3*ON BAY, KOWLOON, HONG KONG
The data are in the wrong fields; the data are spread through the several fields; etc.
I need to map the data to the into the local database. For example:
we’ve got the source data: “YOUNG BAY INTERNATIONAL INC”, which can be mapped to several “YOUNG BAY”-like companies, depends of other source data. In our data base we have:
“YOUNG BAY INC”
“YOUNG BAY INT’L CO”
“YOUNG-BAY INTERNATIONAL “
…
plus the variation in the other source data as City, State, CountryCode, PostalCode, Address etc.
There are the EDI data in”very flexible format”. The source partners fill up the fields of these documents by different applications and peoples. As a result the data are usually spread through the several fields if they filling application has a limit to the field size.
For example:
Good data:
N1*CN*YOUNG BAY INTERNATIONAL INC
N3*2655 ST-JACQUES#101*MONTREAL QC H3J 1H8 CANADA
N3*TEL:514-8121887 514-9313157
Bad data:
N1*CN*AFS -VISA INTERNATIONAL (HK) LTD SUITE NO.1 29/F SKYLINE T
N3*OWER 39 WANG KWONG ROAD, KOWLO
N3*ON BAY, KOWLOON, HONG KONG
The data are in the wrong fields; the data are spread through the several fields; etc.
I need to map the data to the into the local database. For example:
we’ve got the source data: “YOUNG BAY INTERNATIONAL INC”, which can be mapped to several “YOUNG BAY”-like companies, depends of other source data. In our data base we have:
“YOUNG BAY INC”
“YOUNG BAY INT’L CO”
“YOUNG-BAY INTERNATIONAL “
…
plus the variation in the other source data as City, State, CountryCode, PostalCode, Address etc.
This project is the good candidate to use BRE. The data in the input set has different relations (rules). Those rules tend to be constantly improved and modified. It is the ideal candidate to use BRE, isn’t it?
We can process the data by BRE. We can easily develop and modify the sets of rules.
The project was implemented in two stages.
First was the “Unification of the data”. It unified the punctuation, “Incorp” –> “”Inc”, “Logictic” -> “Logistics”, “HONGKONG” -> “HK”, “U S A” -> “US”, “H3J 1H8” -> “H3J1H8”, etc. etc. etc.
The second stage was the “Company resolving”. It took the unified data from the input documents and looked through the corporate data base to map the input data to the corporate data. The example is above.
This part was implemented with help of the BRE .
The Rule Sets were implemented. There were about forty rules.
We can process the data by BRE. We can easily develop and modify the sets of rules.
The project was implemented in two stages.
First was the “Unification of the data”. It unified the punctuation, “Incorp” –> “”Inc”, “Logictic” -> “Logistics”, “HONGKONG” -> “HK”, “U S A” -> “US”, “H3J 1H8” -> “H3J1H8”, etc. etc. etc.
The second stage was the “Company resolving”. It took the unified data from the input documents and looked through the corporate data base to map the input data to the corporate data. The example is above.
This part was implemented with help of the BRE .
The Rule Sets were implemented. There were about forty rules.
And from the start there were the big issues with performance.
Why? The most of the rules did not have the conditions with the simple boolean comparisons like “<Name_Compny> == “value1” but the resource consuming query to the data base.
In the BRE the MOST of the rule conditions are tested (invoked) at the start. (For explanation there is a great article by Charles Young http://geekswithblogs.net/cyoung/articles/79500.aspx) In my case near ALL forty data base queries hit the performance the system.
I did not need in all queries at the start. Then I separated the rules to stages. If the first set of rules did not get the result, the second set comes to the work, etc.
Sounds good? Let see.
One of the BRE technology advantage is a Rete networks (a Rete algorithm) (see http://en.wikipedia.org/wiki/Rete_algorithm). It gives us the capability to efficiently process the big or huge sets of data. The BRE takes care about the order of the tests in the rule set conditions. Moreover the BRE cached the fields of an XML document or the columns of a database table and when these data are accessed for the second time or later within the policy, the values are usually retrieved from the cache. However, when the members of a .NET object are accessed for the second time or later, the values are retrieved from the .NET object, and not from the cache. That means if I invoke the same “test” method from several rules, BRE do not cache the result of this method but each time re invoke the method. It was unfortunately my case.
User has few techniques to manage the order and I used the simplest. I added to each test (or stage) a flag which goes up after this test is ended. The conditions checked the flags. The down flags in the conditions prevented the heavy part of the condition from the start. Now the conditions were started in managed order.
Looks weird.
First of all, the BRE is used to separate the business logic to the independent layer to make it easy (easy to creating, modification, maintaining).
Did I gather the business logic in the rules? No. The main part of rules was remained in the database queries and the classes which wrapped those queries. The rules did only manage the invoking of these queries. If these layers were the same size then the layering give me improvement. The most of the logic was out of BRE Rules layer and easily could be implemented by .NET code. Maintaining the BRE infrastructure for such easy work as invoking the methods in my case was not a good choice.
Second. The BRE takes care about the order of the tests in the rule set conditions. It SHOULD take care of it, otherwise we throw away of this advantage of the Business Rule Engine. If we manage the rule set condition order, we stopped at the same point where we started. I mean now I used BRE, it was cool, but all the rule order processing was mine responsibility. And the most of my issues was to avoid the BRE most amazing feature.
If the BRE was managed the rule set condition order, the performance was bad.
If I managed the rule set condition order, why do I need the BRE?
After several modifications in the rules I’ve decides to redesign the project. I’ve created the simple class to manage the rules and got rid of the BRE artifacts.
Ufff… It was cool but in my case it was something different.
Now all staff is pretty simple and manageable.
You can tell me “Now you cannot use the BRE Composer as a User interface to modifying and improving the business layer logic.” That’s true. Now I cannot delegate the “rule-modification” job to the third party. If this is a requirement then the BRE and the Composer would be a good choice.
Conclusions (“Donts”):
1. If your rules have the resource-consuming conditions (predicates) be aware of non-efficient performance.
2. If your rules have the well-defined processing order, seems the BRE is not your choice. If you don’t care about the rule processing order, the BRE could be useful.
3. If your rules have the values in the conditions retrieved from the .NET object be aware of non-efficient performance.
4. Rules should cover the biggest part of the business logic. If the rules have a lot of calls to the .NET classes think of redesign this project.
Why? The most of the rules did not have the conditions with the simple boolean comparisons like “<Name_Compny> == “value1” but the resource consuming query to the data base.
In the BRE the MOST of the rule conditions are tested (invoked) at the start. (For explanation there is a great article by Charles Young http://geekswithblogs.net/cyoung/articles/79500.aspx) In my case near ALL forty data base queries hit the performance the system.
I did not need in all queries at the start. Then I separated the rules to stages. If the first set of rules did not get the result, the second set comes to the work, etc.
Sounds good? Let see.
One of the BRE technology advantage is a Rete networks (a Rete algorithm) (see http://en.wikipedia.org/wiki/Rete_algorithm). It gives us the capability to efficiently process the big or huge sets of data. The BRE takes care about the order of the tests in the rule set conditions. Moreover the BRE cached the fields of an XML document or the columns of a database table and when these data are accessed for the second time or later within the policy, the values are usually retrieved from the cache. However, when the members of a .NET object are accessed for the second time or later, the values are retrieved from the .NET object, and not from the cache. That means if I invoke the same “test” method from several rules, BRE do not cache the result of this method but each time re invoke the method. It was unfortunately my case.
User has few techniques to manage the order and I used the simplest. I added to each test (or stage) a flag which goes up after this test is ended. The conditions checked the flags. The down flags in the conditions prevented the heavy part of the condition from the start. Now the conditions were started in managed order.
Looks weird.
First of all, the BRE is used to separate the business logic to the independent layer to make it easy (easy to creating, modification, maintaining).
Did I gather the business logic in the rules? No. The main part of rules was remained in the database queries and the classes which wrapped those queries. The rules did only manage the invoking of these queries. If these layers were the same size then the layering give me improvement. The most of the logic was out of BRE Rules layer and easily could be implemented by .NET code. Maintaining the BRE infrastructure for such easy work as invoking the methods in my case was not a good choice.
Second. The BRE takes care about the order of the tests in the rule set conditions. It SHOULD take care of it, otherwise we throw away of this advantage of the Business Rule Engine. If we manage the rule set condition order, we stopped at the same point where we started. I mean now I used BRE, it was cool, but all the rule order processing was mine responsibility. And the most of my issues was to avoid the BRE most amazing feature.
If the BRE was managed the rule set condition order, the performance was bad.
If I managed the rule set condition order, why do I need the BRE?
After several modifications in the rules I’ve decides to redesign the project. I’ve created the simple class to manage the rules and got rid of the BRE artifacts.
Ufff… It was cool but in my case it was something different.
Now all staff is pretty simple and manageable.
You can tell me “Now you cannot use the BRE Composer as a User interface to modifying and improving the business layer logic.” That’s true. Now I cannot delegate the “rule-modification” job to the third party. If this is a requirement then the BRE and the Composer would be a good choice.
Conclusions (“Donts”):
1. If your rules have the resource-consuming conditions (predicates) be aware of non-efficient performance.
2. If your rules have the well-defined processing order, seems the BRE is not your choice. If you don’t care about the rule processing order, the BRE could be useful.
3. If your rules have the values in the conditions retrieved from the .NET object be aware of non-efficient performance.
4. Rules should cover the biggest part of the business logic. If the rules have a lot of calls to the .NET classes think of redesign this project.
====================================================================================
Be aware of the BRE:
It is the tool for processing the specific data set with specific conditions. Not all data and not all business logic can be implemented efficiently with BRE.
Maybe it is not what exactly we can estimate from the product with such name. Now it is hardly a tool for the Business Analyst. It is the tool for software developers. And this is great tool!
I’m sure the next version would be much better from a developers point of view.
=================================================
Notes:
* The other simple method to stage the rules is to separate the rules to different policies and chain these policies. This approach did not work well for my case because I mostly had to manage the order of the rules, not the stages, what leads to pack one rule to one policy. Each policy creates the persistence point in the orchestration. Maybe it would not hit the performance too hard in my case, I didn’t try this.
* We can force the caching of the .NET objects but only programmatic. The Business Rule Composer tool does not expose this possibility.
Be aware of the BRE:
It is the tool for processing the specific data set with specific conditions. Not all data and not all business logic can be implemented efficiently with BRE.
Maybe it is not what exactly we can estimate from the product with such name. Now it is hardly a tool for the Business Analyst. It is the tool for software developers. And this is great tool!
I’m sure the next version would be much better from a developers point of view.
=================================================
Notes:
* The other simple method to stage the rules is to separate the rules to different policies and chain these policies. This approach did not work well for my case because I mostly had to manage the order of the rules, not the stages, what leads to pack one rule to one policy. Each policy creates the persistence point in the orchestration. Maybe it would not hit the performance too hard in my case, I didn’t try this.
* We can force the caching of the .NET objects but only programmatic. The Business Rule Composer tool does not expose this possibility.
—