How To Create a Base Disk or Image in Windows Azure Virtual Machines and Share It Across Accounts

by stephen-w-thomas | Oct 22, 2012 | Stephen's BizTalk and Integration Blog

The Windows Azure Storage API now have the ability to do cross account, asynchronous blob transfers. This means it is no longer required to download the blob in order to upload it to a different Windows Azure account. You can read all about this here.

Over the past few weeks I have been working with the Windows Azure Virtual Machine feature that is currently in preview mode. I talked in a past blog post and video about using the Gallery to create a BizTalk 2010 R2 CTP Virtual Machine.

The HUGE value I see here is we now have the ability to quickly and easily create Virtual Machines using a given Image or Virtual Hard Drive (VHD). We can even create our own custom Images that can be quickly distributed out to a large development team. VHDs created in Windows Azure can be downloaded and used locally and local VHDs can be uploaded to run inside Window Azure Virtual Machines.  Related to that, keep in mind the bandwidth is charged for downloads and that it can take 3 hours or more to download a 30 GB VHD. 

I have worked on several large scale development projects in the past and we have had full teams of people who in the beginning were responsible for creating and maintaining the developer image not to mention the internal IT resources required to support, for a limited amount of time, sometimes hundreds of new virtual machines. Add in the “corporate red tape” that is involved with getting new hardware and setting up a new Virtual Machine resulting in massive time (i.e. money) costs to projects.

Some of that time and money can now be saved with Windows Azure Virtual Machines!

Below I am going to walk through the steps in creating a custom, standalone VHD to be shared with developers for local, isolated development.  I am creating a VHD and not an Image since this is an isolated Virtual Machine I do not need to join a Domain. 

Fist off, why create an Image vs. a Disk (VHD).  Shan McArthur has a great blog post on the difference between an Image and a Disk.  To sum it up:

Windows Azure Image – is a syspreped version of Windows ready to be setup like a brand new install.  You get to create a new Administrator password and you get a new machine name.  This would be OK to connect to a domain but ensure the image does not have Machine Name specific items configured, like BizTalk.  Creating an Image would require more work for the end user once they got it.

Details on how to create a custom image can be found here.

CRITICAL – I have found that once you sysprep and Capture an Image from an existing VHD, that VHD is now unusable. Kind of makes sense, since you did just sysprep it. You can always re-create a new Virtual Machine by using the new Image you just created.

Windows Azure Disk – is a an exact copy of a Virtual Hard Drive.  The Administrator password is the same and so is the machine name.  This works well for machines that will run as Stand Alone, not connected to a domain.  This would work well to share an already running BizTalk VHD that is fully configured and ready for development.

If we wanted something connected to a domain we would go the Image route since they are syspreped. In this case, I just want to give someone a quick, working version of BizTalk 2010 ready to go so I am going with a Windows Azure Disk (VHD) route.

CRITICAL – The only reliable and repeatable way I have found to copy an Image or VHD is to ensure it does not have any existing leases.  A lease is held when you create a Virtual Machine Image or Azure Disk from a VHD in Cloud Storage.  Ensure you delete these Images and Disks before copying the blob – do not delete the underlying blob storage.  Sometimes, these will get orphaned and you need to remove them using PowerShell.  A assume these are all issues related to being a Preview Feature and will be resolved soon.

Below are the steps to create and transfer a VHD to another user. For simplicity, the user will be in the same data center. Cross data center copies are supported, they just take longer.

To help with the cross account copy, I am using my custom tool available here: Windows Azure Virtual Hard Disk (blob storage) Cross Account Copy, Lease Break, and Management Tool

Step 1:  Create your Master VHD.

Start with using a custom or prebuild Image to create a new Virtual Machine.  Once the Machine is setup like you wish, shut it down and delete the Virtual Machine.  Then, inside the Portal Delete the Disk.  Ensure you select the “Retain The Associated VHD” option to keep the underlying blob storage of the VHD.

Step 2:  Set up the Source and Destination Windows Azure Storage Account Details in the Helper Tool.  Download the tool here.

To enable more advanced features, set EnableRESTApi to True and set the Certificate Path and Source Windows Azure Subscription ID.

<appSettings>

  <!-- Enter the Source Storage Account Details here. -->

  <add key="SourceStorageConnection" value="DefaultEndpointsProtocol=https;AccountName=NAME;AccountKey=KEY" />

  <!-- Uncomment and Enter the Destination Storage Account Details here. To work with only one account leave commented out.  

      To copy to the same store set this to the same as Source Account above. -->

  <add key="DestinationStorageConnection" value="DefaultEndpointsProtocol=https;AccountName=NAME;AccountKey=KEY" />

  <!-- Select to enable REST API Advanced Features - Subscription ID and Certificate are Required for this feature.  true | false -->

  <add key='EnableRESTApi' value='true' />

  <!-- Enter the full path to the -->

  <add key='CertificatePath' value='C:\DemoFolder\AzureCerts\Cert.cer'/>

  <!-- Your Windows Azure Subscription ID -->

  <add key='SourceSubscriptionID' value='GUID' />

</appSettings>

.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }

Step 3:  Select the VHD to Copy to the Destination Storage Account.

Launch the Helper Tool.  Click on START QUERY on the top left.  Select the VHD you want to copy.  Note that only Available (i.e. blobs without a lease) can be copied with the tool.  Rename the destination VHD if desired.  Click on Copy. 

If your VHD is Leased, that means it is used by an Azure Disk and could be used by a running Virtual Machine.  Or it could be a known issue that leaves the VHD in a stuck state.  If you know for sure the VHD is not used by an Azure Disk or Virtual Machine, you can select the VHD blob and click Break VHD Lease.  You also have the ability to Delete or Download an VHD if you wish all using this Helper Tool.

Step 4:  See the results in the Destination Storage Account

At this point, if the copy was within the same datacenter it should take only a few seconds.  A cross datacenter copy takes much longer.  Once you have the VHD, create an Azure OS Disk inside the Portal and use that Disk as the OS Disk of a new Virtual Machine.  Just make sure you know the log in account details for the VHD.

Step 5:  Enjoy using your new VHD.

If you gave someone else your Azure Storage Key information make sure you regenerate it for security reasons.

Below is the code that is used to make the copy.  This is using 1.7.1 of the Azure Storage APIs.  At present, I am still working though checking the status of Cross Data Center copies.  The method below returns the Copy ID, not the Request ID.  The Request ID can easily be used with the API to check the status of a request.  Still working on what to do with the Copy ID.

// This is the method that copies the selected VHD

// Account details are read from the App.config file.

// Source and Destination VHD are inputs on the Form.

// Container name is hard coded to vhds.

// If you want to copy to/from other containers just change this and remove the text validation on the Destination Blob Name.

internal void CopyVHD()

{

    // Reads Configuration details from the App.config file.  Needs a reference to Microsoft.WindowsAzure.Configuration.dll.

    // Watch out of issues when using Copy and Paste from the portal.  I pasted into Notepad first and that seemed to mess something up.

    CloudStorageAccount sourceStorageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("SourceStorageConnection"));

    CloudStorageAccount destinationStorageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("DestinationStorageConnection"));

    // Create the blob client using the Accounts above

    CloudBlobClient sourceBlobClient = sourceStorageAccount.CreateCloudBlobClient();

    CloudBlobClient destinationBlobClient = destinationStorageAccount.CreateCloudBlobClient();

    // Retrieve reference to a previously created container

    // Rename "vhds" as needed.  Can be used to read from any container.

    CloudBlobContainer sourceContainer = sourceBlobClient.GetContainerReference(containerName);

    CloudBlob sourceBlob = sourceContainer.GetBlobReference(lstBoxVHDSSource.SelectedItem.ToString());

    CloudBlobContainer destinationContainer = destinationBlobClient.GetContainerReference(containerName);

    // Created the container if it does not exist

    CreateContainerIfNeeded(destinationContainer);

    CloudBlob destinationBlob = destinationContainer.GetBlobReference(txtNewVHDName.Text);

    // Logic added to check if a security token has already been created

    if (!hasSecurityToken)

    {

        // Create a permission policy, consisting of a container-level access policy and a public access setting, and store it on the container. 

        BlobContainerPermissions destinationContainerPermissions = new BlobContainerPermissions();

        // The container-level access policy provides read/write access to the container for 1 day.

        destinationContainerPermissions.SharedAccessPolicies.Add("imageCopyPolicy", new SharedAccessBlobPolicy()

        {

            SharedAccessExpiryTime = DateTime.UtcNow.AddDays(1),

            Permissions = SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Read

        });

        destinationContainerPermissions.PublicAccess = BlobContainerPublicAccessType.Off;

        sourceContainer.SetPermissions(destinationContainerPermissions);

        // This gets the token needed for cross-account copies.  This is not needed to copy into the same account.

        globalSecurityToken = sourceContainer.GetSharedAccessSignature(new SharedAccessBlobPolicy(), "imageCopyPolicy");

        // Set the Global Variable

        hasSecurityToken = true;

    }

    // This does the copy using the StartCopyFromBlog method in the 1.7.1 SDK.            

    string copyID = destinationBlob.StartCopyFromBlob(new Uri(sourceBlob.Uri.AbsoluteUri + globalSecurityToken));

}

.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }