Integrating Integrators – BizTalk, Windows Azure, Windows Workflow, and Beyond

Windows Azure Virtual Hard Disk (blob storage) Cross Account Copy, Lease Break, and Management Tool

BizTalk Blogs

This group is for blogs related to BizTalk Server. This includes Community Syndicated blogs and Stephen W. Thomas’s blog.

Stephen W. Thomas BizTalk MVP

Insight and guidance on Microsoft BizTalk Server, Windows Azure, Infrastructure as a Service, and related Microsoft technologies.

Windows Azure Virtual Hard Disk (blob storage) Cross Account Copy, Lease Break, and Management Tool

Rate This
  • Comments 13

Recently the Windows Azure Storage API’s were updated to include the ability to do cross account, asynchronous blob transfers. This means it was no longer necessary to download the blob in order to upload it to a different Windows Azure account. You can read all about this here.

Over the past few weeks I have been working with the Windows Azure Virtual Machine feature that is currently in preview mode. I talked in a past blog post and video about using the Gallery to create a BizTalk 2010 R2 CTP Virtual Machine.

I am working on proving out and understanding the scenario of creating a Master Image (or Master VHD for non-domain Virtual Machines) and sharing it with a large team of developers. Why? Because on past projects we have spent considerable time and resources (i.e. money) creating, maintaining, and provisioning developer images. For example on a recent project an urgent issue arose and we needed a dedicated BizTalk environment to test a cumulative update release. It was requested on Friday morning. Still today on Wednesday it was not created (4 business days later). With Windows Azure Virtual Machines and a pre-built image this would literally be a few minute task and something I could do myself without the involvement of 3 to 5 other people. In total, so much time (i.e. money) can now be saved with Windows Azure Virtual Machines!

This type of example leads me to my tool and why I created it. In my scenario, a Windows Azure Virtual Machine guru would create a master image or vhd for distribution. With the new asynchronous cross account copy, sending an Image or VHD to a new developer is a snap… as long as you know the “rules”. I created, copied, deleted, and destroyed over 100 Virtual Machines at a total cost of around $5.00 across multiple Azure accounts to fully understand how to successfully copy an Image and VHD from one account to another without issues. I put all that into a simple tool that anyone can use to manage vhds and copy them from one account to another. See below for known issues, features of the tool, and quick how to’s.

I welcome all feedback! Please note this tool is an excellent example of why I am NOT a Windows Form guy and in no way should anything done in this tool be considered a best practice. This tool is more about getting the job done and trying to make things easy to understand.

Download: Windows Azure VHD Helper Tool

Basic Steps for Cross Account VHD Copy:

  1. Shut down your Virtual Machine.
  2. Delete your Virtual Machine.
  3. Delete the associated Image or Disk inside the Virtual Machine Portal. This does not remove the underlying blob of the vhd.
  4. Copy the blob from one storage account to another.
  5. Add your vhd back and recreate the virtual machine.

Issues (per my research) with VHD Copying as a Preview Feature– make sure you understand these issues and limitations. I assume most are related to preview issues and this process will get simpler with time.

  • If you copy a vhd that is Leased (configured as an Image or Disk inside the Virtual Machine Portal) the Source Disk will become unusable.
  • If you copy a vhd that is Leased (configured as an Image or Disk inside the Virtual Machine Portal) the Destination Disk will likely not work. I would say about 75% of the time they do not work. I just re-tested this tonight and they worked.
  • Copying from one data center to another seems to be as slow as downloading the file.
  • When downloading the file, make sure you do not close the form or the download will stop.
  • Copy Status does not seem to be working. I either get NULL or Pending all the time. Probably a user error on my part…. The best way to find out if the copy is complete is to check the file size on the destination. It is 0 MB until the copy is complete.

Features:

  • Visual view of vhds contained inside a Storage Account
  • Simple, one click Delete of vhds
  • Simple, one click Lease Break of vhds (for orphaned vhds – see more here)
  • One click download of a vhd, tested to work with Hyper-V (downloads at a rate of about 3 MB/sec)
  • Support for easily making a copy of a VHD within the same Azure account
  • Support for making a vhd copy across Window Azure Accounts within the same region within a few seconds
  • Support for making a vhd copy to any account in any region (slow when out of region)
  • Prevents you from making a copy of a vhd with an existing Lease
  • Uses the Windows Azure Storage API version 1.7.1 (github)

Advanced Features (with REST API Management Certificate):

  • Visual view of all vhds contained inside a single Windows Azure Account
  • See a “***” by vhds inside the Source storage account
  • List all custom user images inside a single Windows Azure Account

Missing Features (i.e. items I hope to add later):

  • Ability to upload a vhd
  • Ability to get details on an Image 

 

Quick How-To’s:

How to work with just one Storage Account:

Set just a Source Configuration inside the app.config file. I also have a check box for “source only” that will try to auto determine what you want to do. The default is Source Only.

How to Copy a vhd into the same account:

Inside the app.config file for the tool, set the Source and Destination to the same Storage Account.

How to Copy a vhd into another account in the same region (inter-account or cross-account):

Inside the app.config file for the tool, set the Source to the Storage Account that has the source vhd and set Destination to Storage Account that you want for the Destination. The destination container vhds does not need to exist. The tool will create it for you. Click “Copy”. The Destination list should refresh and the copy already be compete. You will know it is complete when the size of the vhd is not 0.

How to Copy a vhd cross region:

Inside the app.config file for the tool, set the Source to the Storage Account that has the source VHD and set Destination to Storage Account that you want for the Destination. The destination container vhds does not need to exist. The tool will create it for you. Select a vhd in the List box that is not lease. Change the vhd name if you wish. Click “Copy”. Click on “List All VHDS” on the top right until you see the file size of the Destination change from 0 to the size of the VHD. This will take a long time, up to 8 hours. I would not close the form until I have confirmed it was completed.

How to Download a Source vhd:

Inside the app.config file for the tool, set the Source to the Storage Account that has the source VHD. Click on “List All VHDS”. Select the VHD you want to download inside the List Box. Click on “Download”. You will be prompted for the download location. Make sure you have enough disk space. Leave the tool open until you see a Completed notice on the top right of the tool. This will take up to 8 hours. Do not close the form or the download will stop.

How to Break a Lease or Delete a vhd on the Source or Destination:

Inside the app.config file for the tool, set the Source and Destination to the Storage Accounts that has the VHDS. Click on “List All VHDS”. Using the List Box, select a single vhd from the list. Click on “Break VHD Lease” to break the existing lease or “Delete VHD” to delete the vhd. Note: this operations are not undoable. Using these incorrectly can result in loss of data or vhds.

I would like to thank the following references:
- How not to throw an exception using CreateIfNotExists on a container - http://blog.smarx.com/posts/testing-existence-of-a-windows-azure-blob
- How to do an Azure REST API Service call - http://msdn.microsoft.com/en-us/library/windowsazure/ee460782.aspx
- How to have a 2nd thread update the main windows form - http://stackoverflow.com/questions/10775367/cross-thread-operation-not-valid-control-textbox1-accessed-from-a-thread-othe
- Async across copy with SDK 1.7.1 - http://blogs.msdn.com/b/windowsazurestorage/archive/2012/06/12/introducing-asynchronous-cross-account-copy-blob.aspx


Download: Windows Azure VHD Helper Tool

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Post
  • This is a great great awsome tool. Only thing that worked for me

  • Seems like a great idea and tool. It`s a little unclear to me what kinda value is needed and where to put it in this config line:    

    <!--Enter the Source Storage Account Details here.-->

       <add key="SourceStorageConnection" value="DefaultEndpointsProtocol=https;AccountName=value;;AccountKey=value" />

  • Okey. I manage to figure this out myself :)

    Example:

    <add key="SourceStorageConnection" value="DefaultEndpointsProtocol=https;AccountName=[STORAGE ACCOUNT NAME];AccountKey=[ONE OF THE KEYS TO YOUR STORAGE ACCOUNT]" />

  • If you want to make a copy of a VHD, set the SourceStorageConnection to your storage account.  Just update the AccountName="" to be the name of your account and the AccountKey to your key.  You can get those values by going to your storage account and clicking the Key button on the bottom middle.

    To copy to the same location, set the DestinationStorageConnection to the same value.  Otherwise, set it to something different.  Note if you copy to a different Data Center, it will take a few hours.  Same data center, takes seconds.

    Hope this helps.

  • This was exactly what I needed, Stephen.  Thanks for the fantastic tool!

  • Windows Azure provides a set of gallery images for us to use when provisioning virtual machines, but

  • I am trying to utilize you executable tool to break a lease I have on a blob so I can then delete the Storage Disk.  However I am getting an error every time I try running the tool.  It is "Unhandled exception has occurred in your application. If you click Continue the application will ignore this error and attempt to continue. If you click Quit the application will close immediately."

  • Make sure you have a container called vhds in the container you are trying to connect to.  It’s a hard coded value in the code.  If you want to break a lease inside a different container name, you’ll need to change this value.

    Also, make sure Use REST API in the Config file is set to No.  And ensure you are not setting a destination account.  

    I have a new version of the tool here: www.biztalkgurus.com/.../VHDandVMTool.zip

    That might help as well.

    If all else fails, I’m happy to take a deeper look drop me an email.  In the new tool, I have a Support button.

  • Again, you absolutely nailed it. Thank you so much for the tool.

  • Is it still true that copying a leased vhd asynchronously aross accounts renders the destination disk unusable? I just did this by mistake on a linux machine and it booted up but I am afraid of errors. Thanks.

  • Thaler – great question.  I believe this is the case only if the VM is running and using the disk.  If the VM is Leased by the Disk and not in use I think it is ok.  

    That said, I’ve never tried to copy any VHD that had any type of Lease on it.  Just to be safe.

    Please post if you run into any issues.

  • You are the god, man, just the god :) Saved my day. Thanks!

    One question though - why should we write tools to fix Microsoft bugs ;)

  • Happy that is worked for you!  I’ve not updated the tool in a while so good to know it is still working.

    That is a good question Sergey!  

    I don’t look at it as fixing a Microsoft bug.  With Azure we are forced to take all of the platform, the good and bad and anything else that comes with that.  I look at this tool as just making life on the Azure Platform easer and at the end of the day making less work for me.  That was the main motivation to build the tool.  

    Hope that makes sense.

Page 1 of 1 (13 items)
Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Post