Windows Azure HDInsight is the simplest option to get a Hadoop cluster up and running very quickly in a Windows Azure environment. Among other numerous advantages, this service allows to use Windows Azure blob storage (ASV or Azure Storage Vault for short) exactly as HDFS (Hadoop distributed file system). In its public beta version, Windows Azure HDInsight uses ASV as its default file system. I put some screen shots of what you can easily do (in a matter of minutes) with such a cluster in this post: TUTO – Hadoop arrive dans le portail Windows Azure. Lan%u00e7ons des jobs Java, PIG et HIVE pour voir !.

Still, you may want to install a custom distribution, run custom components which are not available in the HDInsight distribution, or have Hadoop running on Linux instead of Windows. This post shows how to install a Linux distribution on Windows Azure virtual machines.

While it is possible to install several distributions like MapR, CDH (Cloudera) or HortonWorks (HDP) on different Linux OSs like CentOS, SUSE, or Ubuntu, this post takes HDP 1.2 on CentOS as an example. The documentation I want to be able to follow here in order to install the cluster is the Hortonworks’ HDP 1.2.2 installation with Ambari on CentOS.

In this post, I suppose you already have a Windows Azure account. The web site can provide you with the information you need to get one. I also suppose readers are quite advanced users so I don’t give all the details when documentation also exists elsewhere.


This blog post shows one way to install a Linux Hadoop cluster. It may not respect all the best practices, particularly in terms of security. The goal of the post is to show how this kind of environment can be hosted in Windows Azure. In this post, I chose to use Windows (DNS Server, Web Browser, scripting environment, ) because this is what is simple for me, but this is possible to install the cluster without using Windows at all. I give some hints on how to do that. Also, I use the portal and PowerShell (scripting) because I find it more easy to understand that way, but I’m pretty confident everything could be done with scripting.


Choosing the Windows Azure environment for the Linux cluster

There are several ways to have a local network in Windows Azure. One of them is to create a virtual network.

I also want to be able to use a browser against the cluster and have a DNS I can easily setup and manage. In this scenario, I install a Windows Server which plays those roles. This is because I’m more a Windows guy ! Note that you may prefer installing a Linux based DNS server in the virtual Network and browse the cluster thru an SSH tunnel.

The following table contains the machines I want to instanciate and their roles:

Server Name Server role subnet
n124dns DNS
(Windows Server)
n124m master node subnet2
n124w1 to n124w3 worker nodes subnet3

In this sample, all machines are in the following DNS domain name :

At the end, I’ll have those machines:

n124dns will be a Windows Server machine, and it could be a Linux machine.

All other will be Linux machines.


Configure the Virtual Network

You can find details on Windows Azure virtual networks here:

I create a virtual network like explained at

The parameters I enter are detailed in the XML file below (this file can be obtained by exporting the virtual network configuration from the Windows Azure Management portal).


<NetworkConfiguration xmlns:xsd="" xmlns:xsi="" xmlns="">
        <DnsServer name="n124dns" IPAddress="" />
      <VirtualNetworkSite name="n124" AffinityGroup="NorthEurope2Aff">
          <Subnet name="Subnet-1">
          <Subnet name="Subnet-2">
          <Subnet name="Subnet-3">
          <DnsServerRef name="n124dns" />

from the portal, the virtual network looks like this:


Install the DNS Server

I install the DNS server as a Windows Server 2012 VM. It could also be a Linux machine, but I find it simpler to use a Windows Server which will also serve as a desktop environment for administrative tasks like browsing services like Ambari Server or Hadoop dashboards from a machine that has local access to the whole cluster.

I create a new Windows Server 2012 VM. For that, I go to the Windows Azure Management portal, and select NEW at the bottom left of the web page. Then,

Then I can connect to this machine from the portal:



from the Server Manager which can be started with this icon at the bootom left of the Desktop, I do the following:


I ignore the following warning:



After the installation, I can use the DNS configuration console that can be started that way:

I create a zone called ( is a domain I own) and its associated reverse lookup zone. Here is how it looks:




NB: you may also want to turn off the IE Enhanced Security Configuration here:


Create a Linux image

HDP documentation explains how to install on RPM based Linux OSs. CentOS is one of them. So let’s install CentOS virtual machines.

For that, I go to the Windows Azure Management portal, and select NEW at the bootom left of the web page. Then,




I’ll follow the Hortonworks’ documentation for a CentOS 6 OS. In particular, chapter 1.5 explains how to prepare the environment. I’ll give URLs in the documentation so that you can have the context, as well as the main steps I follow in my environment.



(keep defaults)


sudo su


mkdir .ssh

cat /home/benjguin/.ssh/ >> .ssh/authorized_keys

chmod 700 .ssh

chmod 640 .ssh/authorized_keys

# cf

restorecon -R -v /root/.ssh



ssh root@localhost should work without asking for a password

sudo -s

setenforce 0

vi /etc/selinux/config

sudo -s

chkconfig iptables off

/etc/init.d/iptables stop

sudo -s

vi /etc/yum/pluginconf.d/refresh-packagekit.conf


sudo shutdown -h now



Now, I have my own Linux image. I’ll use it to create the different Linux VMs I need in my cluster.


Instanciate Linux virtual machines



sudo -s

vi /etc/hosts



sudo -s

fdisk -l

grep SCSI /var/log/messages


fdisk /dev/sdc

mkfs.ext3 /dev/sdc1

mkdir /mnt/datadrive

mount /dev/sdc1 /mnt/datadrive

vi /etc/fstab

add the following line at the end of the file:

/dev/sdc1 /mnt/datadrive ext3 defaults 1 2


The other machines in the cluster can be created thru the portal too. But that can also be done thru a script. There are two main flavors of automation scripts in Windows Azure: the Windows Azure PowerShell module can be used from Windows machines, the Command Line Interface (CLI for short, more information) which is based on Node.js can be used from Windows, Mac and Linux. They can be downloaded from

As I’m a Windows guy, I will use PowerShell here. The details on how to start with the Windows Azure PowerShell cmdlets is available here.

Here is the script I use to create n124w1, n124w2 and n124w3:



Import-Module azure


#select default subscription and storage account

$subscription = ‘Azure bengui’

Set-AzureSubscription -SubscriptionName $subscription -CurrentStorageAccount ‘northeurope2affstorage’

Set-AzureSubscription -DefaultSubscription $subscription



+#region secret



#create an empty collection of VMConfigs

$vms = @()


#loop on the three VM we want to create

for($i=1; $i -le 3; $i++)


    Write-Host "creating n124w${i}OS"

    $sshPort = 52200 + $i


    #create a new VM Config

    $newVM = `

        New-AzureVMConfig -ImageName myCentOSImage -InstanceSize Medium -Name "n124w$i" `

            -AvailabilitySetName "n124wAvailabilitySet" -DiskLabel "n124w${i}os" `

            -HostCaching ReadWrite -Label "n124w$i" |

        Add-AzureProvisioningConfig -Linux -LinuxUser "benjguin" -Password $adminPassword |

        Add-AzureDataDisk -CreateNew -DiskLabel n124w${i}data1 -DiskSizeInGB 100 -LUN 0 |

        Add-AzureEndpoint -LocalPort 22 -Name "SSH$i" -Protocol tcp -PublicPort $sshPort |

        Set-AzureSubnet ‘Subnet-3’


    #add the VM config to the collection

    $vms += ,$newVM



#show the collection

$vms | format-table


#create the VM

New-AzureVM -ServiceName ‘n124’ -VMs $vms -VNetName ‘n124’


This generates the following output:

creating n124w1OS

creating n124w2OS

creating n124w3OS

AvailabilitySetName   ConfigurationSets     DataVirtualHardDisks  Label                 OSVirtualHardDisk     RoleName              RoleSize             RoleType           
——————-   —————–     ——————–  —–                 —————–     ——–              ——–             ——–           
n124wAvailabilitySet  {n124w1, Microsoft… {}                    bjEyNHcx              Microsoft.Samples…. n124w1                Medium               PersistentVMRole   
n124wAvailabilitySet  {n124w2, Microsoft… {}                    bjEyNHcy              Microsoft.Samples…. n124w2                Medium               PersistentVMRole   
n124wAvailabilitySet  {n124w3, Microsoft… {}                    bjEyNHcz              Microsoft.Samples…. n124w3                Medium               PersistentVMRole   

WARNING: VNetName, DnsSettings, DeploymentLabel or DeploymentName Name can only be specified on new deployments.

OperationDescription                                      OperationId                                               OperationStatus                                         
——————–                                      ———–                                               —————                                         
New-AzureVM – Create VM n124w1                            652a8988-df51-4777-94ae-bc7764296b0b                      Succeeded                                               
New-AzureVM – Create VM n124w2                            5b165d43-dcc7-4f5d-a1fc-ae77c6fa62dd                      Succeeded                                               
New-AzureVM – Create VM n124w3                            70750433-9f19-493c-9fdd-a8aad69fa193                      Succeeded                                               


and that can be seen in the Windows Azure management portal:


The DNS server can be manually updated in order to obtain the following configuration:

Then, from n124m, I can connect to other machines like this:

From there, I can update /etc/hosts in order to have hostname and hostname -f having the right names.

I can also format the data disks.

These are the same steps as before for n124w*. To summarize, here a few screen shots for n124w2:


mkfs.ext3 /dev/sdc1

mkdir /mnt/datadrive

mount /dev/sdc1 /mnt/datadrive

vi /etc/fstab

add the following line at the end of the file:

/dev/sdc1 /mnt/datadrive ext3 defaults 1 2

(same for other n124w* machines).


Now I have a whole set of machines in the same network, and they are ready to have Hadoop installed on them.

Install Hadoop distribution

I can now start at step 2 of the installation in Hortonworks’ documentation.

connect to n124m

sudo su

rpm -Uvh


yum install epel-release


yum install ambari-server




ambari-server setup

(I use default answers during the setup)


ambari-server start

I am now at step 3 of the Hortonworks’ documentation states to connect to http://{main.install.hostname}:8080. I do that from the Windows Server machine which is in the local network where the Linux machines are.

NB: in order to connect to n124dns, one simple way is to select this VM in the Windows Azure management portal and click Connect:

In a remote desktop on the n124dns machine, I do the following:

I connect with admin/admin

Next screen asks for the .ssh/id_rsa key. The simplest way is to copy it from an ssh session ( in my case) and paste it in the browser.


in the customize services web page, I have

I choose to remove /mnt/resource folders because it corresponds to a disk that lives with the VM and which is not persisted in the Windows Azure storage folder. This disk is destroyed when one destroys the VM. HDFS would support it, but I want to be able to stop my whole cluster without loosing HDFS data. So I change to the following:

I do the same in tab MapReduce

I also enter the required passwords and click Next

the detail is the following


Admin Name : admin

Cluster Name : n124hdp

Total Hosts : 4 (4 new)

Local Repository : No



      NameNode :
      SecondaryNameNode :
      DataNodes : 3 hosts

      JobTracker :
      TaskTrackers : 3 hosts

      Server :
      Administrator : nagiosadmin / (

      Server :
    Hive + HCatalog

      Hive Metastore :
      Database : MySQL (New Database)

      Master :
      Region Servers : 3 hosts

      Server :

      Servers : 3 hosts






Let’s now test HDFS, PIG and HIVE by ourselves in this cluster.

I open a new SSH connection to the master node (n124m, available at

let’s copy /etc/paswd file to /hwork/in HDFS folder

[benjguin@n124m ~]$ cp test.pig /tmp

[benjguin@n124m ~]$ cd /tmp

[benjguin@n124m tmp]$ cat test.pig

A = load ‘/hwork/in’ using PigStorage(‘:’);

B = foreach A generate \$0 as id;

store B into ‘/hwork/out1’;

[benjguin@n124m tmp]$ sudo -u hdfs pig test.pig

2013-04-05 10:54:08,586 [main] INFO  org.apache.pig.Main – Apache Pig version (rexported) compiled Mar 28 2013, 12:20:36

2013-04-05 10:54:08,587 [main] INFO  org.apache.pig.Main – Logging error messages to: /tmp/pig_1365159248577.log

2013-04-05 10:54:09,141 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to hadoop file system at: hdfs://

2013-04-05 10:54:09,318 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to map-reduce job tracker at:

2013-04-05 10:54:10,104 [main] INFO – Pig features used in the script: UNKNOWN

2013-04-05 10:54:10,294 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler – File concatenation threshold: 100 optimistic? false

2013-04-05 10:54:10,323 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer – MR plan size before optimization: 1

2013-04-05 10:54:10,323 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer – MR plan size after optimization: 1

2013-04-05 10:54:10,431 [main] INFO – Pig script settings are added to the job

2013-04-05 10:54:10,469 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3

2013-04-05 10:54:10,471 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – creating jar file Job5927476079760658662.jar

2013-04-05 10:54:14,037 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – jar file Job5927476079760658662.jar created

2013-04-05 10:54:14,061 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – Setting up single store job

2013-04-05 10:54:14,101 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 1 map-reduce job(s) waiting for submission.

2013-04-05 10:54:14,603 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 0% complete

2013-04-05 10:54:15,036 [Thread-7] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat – Total input paths to process : 1

2013-04-05 10:54:15,036 [Thread-7] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths to process : 1

2013-04-05 10:54:15,052 [Thread-7] WARN – Snappy native library is available

2013-04-05 10:54:15,053 [Thread-7] INFO  org.apache.hadoop.util.NativeCodeLoader – Loaded the native-hadoop library

2013-04-05 10:54:15,053 [Thread-7] INFO – Snappy native library loaded

2013-04-05 10:54:15,056 [Thread-7] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths (combined) to process : 1

2013-04-05 10:54:16,311 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – HadoopJobId: job_201304050850_0005

2013-04-05 10:54:16,311 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – More information at:

2013-04-05 10:54:26,873 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 50% complete

2013-04-05 10:54:30,936 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 100% complete

2013-04-05 10:54:30,938 [main] INFO – Script Statistics:

HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features       hdfs    2013-04-05 10:54:10     2013-04-05 10:54:30     UNKNOWN


Job Stats (time in seconds):

JobId   Maps    Reduces MaxMapTime      MinMapTIme      AvgMapTime      MaxReduceTime   MinReduceTime   AvgReduceTime  Alias    Feature Outputs

job_201304050850_0005   1       0       3       3       3       0       0       0       A,B     MAP_ONLY        /hwork/out1,


Successfully read 45 records (2588 bytes) from: "/hwork/in"


Successfully stored 45 records (304 bytes) in: "/hwork/out1"


Total records written : 45

Total bytes written : 304

Spillable Memory Manager spill count : 0

Total bags proactively spilled: 0

Total records proactively spilled: 0

Job DAG:


2013-04-05 10:54:30,954 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – Success!

[benjguin@n124m tmp]$

so PIG works!


let’s now do the same thing with HIVE:

that works too!



Automate the cluster [de]provisioning

Once I have the VMs setup, I want to be able to stop paying for compute by shutting down and removing the VMs (while still keeping the virtyual hard disks VHD).

Il also want to be able to restart the whole cluster quite quickly.

As seen before in this post, it is possible to automate by scripting. Here are the scripts I use to stop and start this cluster.

NB: I took the OS disk names and data disk names from the Windows Azure portal


To shut up the cluster, I use


#region init

Import-Module ‘c:\Program Files (x86)\Microsoft SDKs\Windows Azure\PowerShell\Azure\Azure.psd1’


$subscription = ‘Azure bengui’

Set-AzureSubscription -SubscriptionName $subscription -CurrentStorageAccount ‘northeurope2affstorage’

Set-AzureSubscription -DefaultSubscription $subscription


$cloudSvcName = ‘n124’



#region shutdown and delete

echo ‘will shut down and remove the following’

#$vms = Get-AzureVM -ServiceName $cloudSvcName | where { !($ -eq ‘n124dns’) }

$vms = Get-AzureVM -ServiceName $cloudSvcName

$vms | select name

$vms | Stop-AzureVM

$vms | Remove-AzureVM






To startup the cluster, I use:



Import-Module azure


$subscription = ‘Azure bengui’

Set-AzureSubscription -SubscriptionName $subscription -CurrentStorageAccount ‘northeurope2affstorage’

Set-AzureSubscription -DefaultSubscription $subscription


$cloudSvcName = ‘n124’

$vNetName = ‘n124’


$vms = @()



$n124dns = New-AzureVMConfig -DiskName ‘n124dns-n124dns-0-20130117132807’ -InstanceSize Small -Name $vmName -Label $vmName |

    Set-AzureSubnet ‘Subnet-1’ |

    Add-AzureEndpoint -LocalPort 3389 -Name ‘RDP’ -Protocol tcp -PublicPort 3389

$vms += ,$n124dns



$vm1 = New-AzureVMConfig -DiskName ‘n124-n124m-2013-04-03’ -InstanceSize Medium -Name $vmName -Label $vmName |

    Add-AzureDataDisk -DiskName ‘n124-n124m-0-201304031626300873’ -Import -LUN 0 |

    Add-AzureEndpoint -LocalPort 22 -Name ‘SSH’ -Protocol tcp -PublicPort 22 |

    Set-AzureSubnet ‘Subnet-2’


$vms += ,$vm1


$osDiskNames = @(‘n124-n124w1-0-201304041529430938’, `

    ‘n124-n124w2-0-201304041530370691’, `



$dataDiskNames = @(‘n124-n124w1-0-201304041529470297’, `

    ‘n124-n124w2-0-201304041530410363’, `



#loop on the three VM we want to create

for($i=1; $i -le 3; $i++)


    Write-Host "creating n124w${i}OS"

    $sshPort = 52200 + $i


    $j = $i 1


    #create a new VM Config

    $newVM = `

        New-AzureVMConfig -DiskName $osDiskNames[$j] -InstanceSize Medium -Name "n124w$i" `

            -AvailabilitySetName "n124wAvailabilitySet" -Label "n124w$i" |

        Add-AzureDataDisk -DiskName $dataDiskNames[$j] -Import -LUN 0 |

        Add-AzureEndpoint -LocalPort 22 -Name "SSH$i" -Protocol tcp -PublicPort $sshPort |

        Set-AzureSubnet ‘Subnet-3’


    #add the VM config to the collection

    $vms += ,$newVM



#show the collection

$vms | format-table


#create the VM

New-AzureVM -ServiceName ‘n124’ -VMs $vms -VNetName ‘n124’


So I now have a Linux cluster with Hadoop installed and I can start, and stop it (without loosing HDFS data) with scripts.



Blog Post by: Benjamin GUINEBERTIERE