Windows Azure tables: partitioning guid row keys | Windows Azure tables: partionnement de clefs de type GUID

by | May 18, 2012 | BizTalk Community Blogs via Syndication

 

 

This post is about using a partition key while the rowkey is a GUID in Windows Azure Table Storage. You may read more on Windows Azure Tables partitioning in this MSDN article. Ce billet est %u00e0 propos de l’utilisation d’une clef de partition alors que la clef est un GUID lors de l’utilisation des tables non relationnelles de Windows Azure. On trouvera des informations compl%u00e9mentaires sur le partitionnement des tables Windows Azure dans cet article MSDN.
Here is some context on when this sample code comes from. Voici un peu de contexte sur l’origine de cet exemple de code.
I’m currently developing some sample code that works as a canvas application in Facebook. As this type of app runs in a iFrame, it restricts the use of cookies. Still, I wanted to keep some authentication token from page to page (the Facebook signed_request) without displaying it explicitly in the URL (it may also be sent to HTTP referer headers to external sites). So i decided to store the signed_request in a session. ASP.NET sessions can have their ID stored in the URL instead of cookies but ASP.NET pipelines does not provide the session yet while authenticating. So I created my own storage for the authentication and authorization session (Auth Session for short). I did it in Windows Azure tables so that it can easily scale out. Je d%u00e9veloppe actuellement un exemple d’application Facebook de type Canvas. Comme ces types d’applications s’ex%u00e9cutent dans une IFrame, les cookies sont difficilement utilisables. Cependant je devais tout de m%u00eame pouvoir garder un jeton d’authentification de page en page (le champ signed_request de Facebook) sans l’afficher explicitement dans l’URL (il pourrait entre autres %u00eatre envoy%u00e9 dans l’ent%u00eate referer HTTP vers des sites externes). J’ai donc choisi de stocker le signed_request dans la session. ASP.NET propose de stocker l’ID de session dans l’URL plut%u00f4t que dans des cookies mais l’ordre des %u00e9v%u00e9nements ASP.NET fait que lors de l’authentification la session n’est pas encore disponible. J’ai donc cr%u00e9%u00e9 mon propre stockage pour la session d’authentification et d’autorisation (en abr%u00e9g%u00e9: Auth Session). Je l’ai fait en m’appuyant sur les tables Windows Azure de fa%u00e7on %u00e0 b%u00e9n%u00e9ficier simplement de la mont%u00e9e en charge horizontale.
The functional key is a GUID (I don’t want user X to guess user Y’s authSessionKey). The key is passed from page to page as a query parameter (typically, app urls look like this: https://myhost/somepath?authSessionKey=3479D7A2-5D1A-41A8-B8FF-4F62EB1A07BB. La clef fonctionnelle est un GUID (je ne veux pas que l’utilisateur X puisse deviner la clef de session d’un utilisateur Y). Cette clef est pass%u00e9e de page en page sous la forme d’un param%u00e8tre de la requ%u00eate (typiquement les urls de l’app. ressemblent %u00e0 ceci: https://myhost/somepath?authSessionKey=3479D7A2-5D1A-41A8-B8FF-4F62EB1A07BB.
Still, in order to have this scaling horizontally I need to have a partition key. Here is the code I used: Il reste que pour pouvoir monter en charge horizontalement, il faut une clef de partition. Voici le code:

   

internal class AuthSessionDataSource
{
//...
        public const int nbPartitions = 15;
// ...

public static class AuthSessionState
{
//...
    private static string PartitionKeyForGuid(Guid guid)
    {
        int sumOfBytes = 0;
        foreach (var x in guid.ToByteArray())
        {
            sumOfBytes += x;
        }
        int partitionNumber = sumOfBytes % AuthSessionDataSource.nbPartitions;
        return partitionNumber.ToString();
    }
//...
The principle is to get the remainder of the sum of all bytes participating in the GUID divided by the number of partitions as the partition number. Le principe est de prendre comme num%u00e9ro de partition le reste de la division de la somme des octets qui consituent le GUID par le nombre de partitions.
In order to have a rough idea of what it provides, here is a small console application (code, then sample execution result). De fa%u00e7on %u00e0 avoir une id%u00e9e rapide de ce que cela donne, on peut mettre le code dans une application console et l’ex%u00e9cuter. 
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace ConsoleApplication2
{
    class Program
    {
        static void Main(string[] args)
        {
            for (int i = 0; i < 36; i++)
            {
                Guid g = Guid.NewGuid();
                Console.WriteLine("{0}, {1}", g, PartitionKeyForGuid(g));
            }
            Console.ReadLine();


        }


        private static string PartitionKeyForGuid(Guid guid)
        {
            const int nbPartitions = 12;

            int sumOfBytes = 0;
            foreach (var x in guid.ToByteArray())
            {
                sumOfBytes += x;
            }
            int partitionNumber = sumOfBytes % nbPartitions;
            return partitionNumber.ToString();
        }

    }
}

 

The advantage is that the partition numbers should be distributed quite regularly and that you can get calculate the partition from the rowkey as long as the number of partitions doesn’t change. L’avantage est que les num%u00e9ros de partition devraient %u00eatre distribu%u00e9s assez r%u00e9guli%u00e8rement et que l’on peut calculer ce num%u00e9ro de partition %u00e0 partir de la clef (le GUID) tant que le nombre de partitions ne change pas.
Should I change and have more partitions as the number of users grow, I could store new users’ sessions to a new table where the number of partitions is higher while keeping already active users to the old table. Auth sessions don’t live very long so changing the number of partitions can be quite simple. Et si on devait changer le nombre de partitions parce que le nombre d’utilisateurs augmente? Dans mon cas, je peux stocker les session des nouveaux utilisateurs dans une nouvelle table avec un plus grand nombre de partitions tout en gardant les utilisateurs d%u00e9j%u00e0 actifs dans l’ancienne table. Les Auth Sessions n’ont de toutes fa%u00e7ons pas une dur%u00e9e de vie tr%u00e8s longue et changer le nombre de partitions devrait %u00eatre assez simple.

 

public const int nbDaysForOldSessions = 3;
//...
internal void RemoveOldValues()
{
    DateTime oldDate = DateTime.UtcNow.AddDays(-1 * nbDaysForOldSessions);

    for(int p=0; p<nbPartitions; p++)
    {
        string partitionKey = p.ToString();
        var query = (from c in context.AuthSessionStateTable
                     where c.PartitionKey == partitionKey
                     && c.Timestamp <= oldDate
                     select c)
                    .AsTableServiceQuery<AuthSessionStateEntity>();
        var result = query.Execute();
        int i = 0;
        foreach (var x in result)
        {
            i++;
            this.context.DeleteObject(x);
            if (i >= 100)
            {
                this.context.SaveChangesWithRetries();
                i = 0;
            }
        }
        this.context.SaveChangesWithRetries();
    }
}

 

Why not using the rowkey as a partition key? Well having several rows in the same partition allows batching which is also good for performance. For instance, I have to remove old sessions. As batch can only happen in a same partition and as no more than 100 rows can be batched together, here is the code to purge old Auth sessions: Pourquoi ne pas utiliser la clef (le GUID) en tant que clef de partition %u00e9galement? En fait, avoir plusieurs rang%u00e9es dans la m%u00eame partition permet de grouper des requ%u00eates ce qui am%u00e9liore aussi les performances. Par exemple, j’ai besoin de supprimer les vieilles sessions. Comme les regroupements ne peuvent se faire que dans la m%u00eame partition et que 100 enregistrements par 100 enregistrements voici le code de purge des vieilles sessions:
In my case, having this way of partitioning data, seems to be a good fit. Dans mon cas, partitionner les donn%u00e9es de cette fa%u00e7on semble assez adapt%u00e9.

 

Benjamin

Blog Post by: Benjamin GUINEBERTIERE

BizTalk Innovation Event, Milan, Italy | May 24th

by | May 17, 2012 | BizTalk Community Blogs via Syndication

Overnet Education in collaboration with Microsoft Italy and User Group Italiano Connected System has organaised one of the biggest BizTalk event in Italy.

All thanks goes to Nino Crudele (fellow Italian BizTalk Server MVP) for taking the effort to organise the event. It’s not a easy task by no means organising a community activity (FREE) of this magnitude bringing international speakers. Big thanks to Nino for organising this.

If you are somewhere near by (Sweden, Norway, UK or any European destination), it will be a worthy while day to network with some of the great guys in BizTalk community.

The agenda for the event:

BizTalk Host thresholds and automatic throttling
by Tord Glad Nordahl;

BizTalk360
by Saravana Kumar (MVP BizTalk Server)

Adapter Pack Integration Capabilities (BizTalk)
by Steef-Jan Wiggers (MVP BizTalk Server)

Introduction to the Azure Service Bus EAI/EDI features
by Sandro Pereira (MVP BizTalk Server)

BizTalk Innovation Event Wrap up
by Nino Crudele (MVP BizTalk Server)

Click here to ensure your registration or to get more information?s about this event.

Nandri!

Saravana Kumar

BizTalk Message Based Routing: why is my file not transfered to the location of the Send Port

by | May 17, 2012 | BizTalk Community Blogs via Syndication

Inspired by some tweets I had with Sam Vanhoutte (@SamVanhoutte), Mikael Sand (@MikaelSand) and Dean Robertson (@DeanRobertson), I decided to write down a number of our experiences of what can go wrong, or what you could forget, in case of a simple message based routing scenario.

So basically this article contains a checklist, which can be used as a reminder in case your message based routing scenario does not work, like you expected.

Scenario
Imagine you have a message based routing solution in its most simple form.
Your Receive Location listens to a local folder and your Send Port is subscribed to the Receive Port from the Receive Location. The URI of the Send Port is another local folder.
Although this is a very simple solution, still there is enough that can prevent the messages from flowing from A to B.

1. The Receive Location listens to the wrong folder or contains a typo
To be absolutely sure you have configured the correct folder, copy the path from the address bar in Windows Explorer and paste it at the URI field in the Receive Location.

2. The Receive Location is not enabled
If the Receive Location is not enabled, it won’t poll the configured folder, so you must enable the Receive Location.

3. BizTalk is not authorized to access the file location of the Receive Location
Make sure that the identity which is configured at the Host which runs the Receive Location, has enough permissions (read/write) on the folder of the Receive Location. Failing to do so, results in the Receive Location disabling itself (after some retries).

4. The Host Instance which runs the Receive Location is not started
Host Instances take care of the actual processing of your Receive Locations, Send Ports and Orchestrations. So if the Host Instance from the Receive Location is not started, no files will be picked up, although your Receive Location is enabled, so start the designated Host Instance.

5. The filter on the Send Port contains the wrong Receive Port or contains a typo
Correct this by copying the name from the Receive Port and paste it in the filter of the Send Port.

6. The name of the Receive Port in the filter of the Send Port is surrounded with quotes
Quotes are not needed, so remove them.

7. The Send Port is not enlisted and started
If the Send Port is not enlisted, there is no active subscription for the Send Port. Further the Send Port must be started to send any files, coming from the Receive Port to the outgoing folder. So Enlist and Start the Send Port.

8. The Host Instance which runs the Send Port is not started
Host Instances take care of the actual processing of your Receive Locations, Send Ports and Orchestrations. So if the Host Instance from the Send Port is not started, the file will be picked up by the Receive Location, stay in the MessageBox, but won’t be delivered to the outgoing folder, so start the designated Host Instance.

9. BizTalk is not authorized to access the file location from the Send Port
Make sure that the identity which is configured at the Host which runs the Send Port, has enough permissions (read/write) on the folder of the Send Port. Failing to do so, will result in suspended messages.

10. Mismatch of the file mask on the Receive Location
The file that you drop on the folder of the Receive Location does not match the file mask from the Receive Location. If you are dropping a .XML file while the Receive Location expects only *.TXT files, your .XML file won’t be picked up by the Receive Location.

11. Tricked by Windows Explorer
You think you are dropping a .XML file, and the file mask expects .XML files, but still the file is not picked up. Perhaps you are tricked by Windows Explorer. Windows Explorer has an option to hide familiar extensions of files. So if you have a file called order.xml.txt, Windows Explorer hides the .txt extension, perhaps giving you the idea that you are dealing with a .XML file, while in fact you are dealing with a .TXT file. In Windows Explorer turn off the option to hide familiar extensions, making sure that you will always deal with the full file name.

BizTalk Training – Mapping – How to implement multi-level Muenchian grouping in BizTalk Maps

BizTalk Training – Mapping – How to implement multi-level Muenchian grouping in BizTalk Maps

by | May 15, 2012 | BizTalk Community Blogs via Syndication

I already talked about Muenchian Grouping in BizTalk Maps in the past: BizTalk Training – Mapping – Muenchian Grouping and Sorting in BizTalk Maps without losing Map functionalities Inspired by a question in BizTalk Server Forums: XSLT Mapping Question (Summing Values) I decided to solved this problem and publish this sample to help this user […]
Blog Post by: Sandro Pereira

Do you want to win a book?

by | May 15, 2012 | BizTalk Community Blogs via Syndication

So, I have this little raffle within my company, where I run the local BizTalk User Group called BugLoSE. The subject of the raffle/competition is “upcoming subjects for BugLoSE” and I thought I would invite “non Logicans” as well.

So send me suggestions and you might be the lucky one to pick from one of the books behind this link. (Note that the link displays page one of three). You can e-mail me directly at: firstname(dot)lastname(at)logica(dot)com.

The books are from Packt Publishing, a publisher that has risen to a very high spot when it comes to books about integration and Microsoft technologies. Also (if you do not win), they have a “Brighten your May at Packt’s Microsoft Carnival” at the moment.

Blog Post by: Mikael Sand

Use our $400 Discount for TechEd North America 2012

by | May 14, 2012 | BizTalk Community Blogs via Syndication

As an exhibitor at TechEd, QuickLearn Training is able to offer you a $400 discount to Microsoft TechEd North America this year.

Technology is changing faster than ever, and TechEd is the best chance you’ll get all year to learn from the world’s most qualified experts. You’ll even have the opportunity to see QuickLearn’s MVP Anthony Borton present his session, “The Accidental Team Foundation Server Admin” on Wednesday at 5PM.

To get the discount, just use this code when you register: ATEPQUICK

Visit our booth for a chance to win your next class for free!

How to obtain a list of “BizTalk Software Inventory” installed with PowerShell

How to obtain a list of “BizTalk Software Inventory” installed with PowerShell

by | May 14, 2012 | BizTalk Community Blogs via Syndication

Just play around with PowerShell and some BizTalk administrative and maintenance tasks It’s always good to know what software is installed in our environment. Sometimes we need to know what version of BizTalk is installed or what version of the Adapter Pack, x86 or x64? And preferably be able to get this list in an […]
Blog Post by: Sandro Pereira