BizTalk Admin only installation bug : Failed to create WCF-SQL Transport Component

Recently we received a support call from one of our BizTalk360 customer saying, they were able to control all the send port/receive locations using BizTalk360 but whenever they try to start/stop send port or enable/disable receive location that uses WCF-SQL adapter it throws the following exception. Failed to create ‘WCF-SQL’ Transport Component Microsoft.BizTalk.ExplorerOM.BtsException: Failed to […]

The post BizTalk Admin only installation bug : Failed to create WCF-SQL Transport Component appeared first on BizTalk360 Blog.

Blog Post by: Saravana Kumar

How to install Hadoop on Windows Azure Linux virtual machines

Windows Azure HDInsight is the simplest option to get a Hadoop cluster up and running very quickly in a Windows Azure environment. Among other numerous advantages, this service allows to use Windows Azure blob storage (ASV or Azure Storage Vault for short) exactly as HDFS (Hadoop distributed file system). In its public beta version, Windows Azure HDInsight uses ASV as its default file system. I put some screen shots of what you can easily do (in a matter of minutes) with such a cluster in this post: TUTO – Hadoop arrive dans le portail Windows Azure. Lan%u00e7ons des jobs Java, PIG et HIVE pour voir !.

Still, you may want to install a custom distribution, run custom components which are not available in the HDInsight distribution, or have Hadoop running on Linux instead of Windows. This post shows how to install a Linux distribution on Windows Azure virtual machines.

While it is possible to install several distributions like MapR, CDH (Cloudera) or HortonWorks (HDP) on different Linux OSs like CentOS, SUSE, or Ubuntu, this post takes HDP 1.2 on CentOS as an example. The documentation I want to be able to follow here in order to install the cluster is the Hortonworks’ HDP 1.2.2 installation with Ambari on CentOS.

In this post, I suppose you already have a Windows Azure account. The windowsazure.com web site can provide you with the information you need to get one. I also suppose readers are quite advanced users so I don’t give all the details when documentation also exists elsewhere.

Scope

This blog post shows one way to install a Linux Hadoop cluster. It may not respect all the best practices, particularly in terms of security. The goal of the post is to show how this kind of environment can be hosted in Windows Azure. In this post, I chose to use Windows (DNS Server, Web Browser, scripting environment, ) because this is what is simple for me, but this is possible to install the cluster without using Windows at all. I give some hints on how to do that. Also, I use the portal and PowerShell (scripting) because I find it more easy to understand that way, but I’m pretty confident everything could be done with scripting.

 

Choosing the Windows Azure environment for the Linux cluster

There are several ways to have a local network in Windows Azure. One of them is to create a virtual network.

I also want to be able to use a browser against the cluster and have a DNS I can easily setup and manage. In this scenario, I install a Windows Server which plays those roles. This is because I’m more a Windows guy ! Note that you may prefer installing a Linux based DNS server in the virtual Network and browse the cluster thru an SSH tunnel.

The following table contains the machines I want to instanciate and their roles:

Server Name Server role subnet
n124dns DNS
Browser
(Windows Server)
subnet1
n124m master node subnet2
n124w1 to n124w3 worker nodes subnet3

In this sample, all machines are in the following DNS domain name : n124.benjguin.com

At the end, I’ll have those machines:

n124dns will be a Windows Server machine, and it could be a Linux machine.

All other will be Linux machines.

 

Configure the Virtual Network

You can find details on Windows Azure virtual networks here: http://www.windowsazure.com/en-us/manage/services/networking/.

I create a virtual network like explained at http://www.windowsazure.com/en-us/manage/services/networking/create-a-virtual-network/.

The parameters I enter are detailed in the XML file below (this file can be obtained by exporting the virtual network configuration from the Windows Azure Management portal).

 

<NetworkConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/ServiceHosting/2011/07/NetworkConfiguration">
  <VirtualNetworkConfiguration>
    <Dns>
      <DnsServers>
        <DnsServer name="n124dns" IPAddress="10.124.1.4" />
      </DnsServers>
    </Dns>
    <VirtualNetworkSites>
      <VirtualNetworkSite name="n124" AffinityGroup="NorthEurope2Aff">
        <AddressSpace>
          <AddressPrefix>10.0.0.0/8</AddressPrefix>
        </AddressSpace>
        <Subnets>
          <Subnet name="Subnet-1">
            <AddressPrefix>10.124.1.0/24</AddressPrefix>
          </Subnet>
          <Subnet name="Subnet-2">
            <AddressPrefix>10.124.2.0/24</AddressPrefix>
          </Subnet>
          <Subnet name="Subnet-3">
            <AddressPrefix>10.124.3.0/24</AddressPrefix>
          </Subnet>
        </Subnets>
        <DnsServersRef>
          <DnsServerRef name="n124dns" />
        </DnsServersRef>
      </VirtualNetworkSite>
    </VirtualNetworkSites>
  </VirtualNetworkConfiguration>
</NetworkConfiguration>

from the portal, the virtual network looks like this:

 

Install the DNS Server

I install the DNS server as a Windows Server 2012 VM. It could also be a Linux machine, but I find it simpler to use a Windows Server which will also serve as a desktop environment for administrative tasks like browsing services like Ambari Server or Hadoop dashboards from a machine that has local access to the whole cluster.

I create a new Windows Server 2012 VM. For that, I go to the Windows Azure Management portal, and select NEW at the bottom left of the web page. Then,

Then I can connect to this machine from the portal:

 

 

from the Server Manager which can be started with this icon at the bootom left of the Desktop, I do the following:

 

I ignore the following warning:

 

 

After the installation, I can use the DNS configuration console that can be started that way:

I create a zone called n124.benjguin.com (benjguin.com is a domain I own) and its associated reverse lookup zone. Here is how it looks:

 

 

 

NB: you may also want to turn off the IE Enhanced Security Configuration here:

 

Create a Linux image

HDP documentation explains how to install on RPM based Linux OSs. CentOS is one of them. So let’s install CentOS virtual machines.

For that, I go to the Windows Azure Management portal, and select NEW at the bootom left of the web page. Then,

 

Yes

 

I’ll follow the Hortonworks’ documentation for a CentOS 6 OS. In particular, chapter 1.5 explains how to prepare the environment. I’ll give URLs in the documentation so that you can have the context, as well as the main steps I follow in my environment.

 

http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.2/bk_using_Ambari_book/content/ambari-chap1-5-2.html

 

ssh-keygen

(keep defaults)

 

sudo su

cd

mkdir .ssh

cat /home/benjguin/.ssh/id_rsa.pub >> .ssh/authorized_keys

chmod 700 .ssh

chmod 640 .ssh/authorized_keys

# cf  http://stackoverflow.com/questions/7268788/centos-6-sshd-server-refused-our-key

restorecon -R -v /root/.ssh

CTRL-D

 

ssh root@localhost should work without asking for a password

 

 

http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.2/bk_using_Ambari_book/content/ambari-chap1-5-5.html

sudo -s

setenforce 0

vi /etc/selinux/config

 

http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.2/bk_using_Ambari_book/content/ambari-chap1-5-6.html

sudo -s

chkconfig iptables off

/etc/init.d/iptables stop

http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.2/bk_using_Ambari_book/content/ambari-chap1-5-7.html

sudo -s

vi /etc/yum/pluginconf.d/refresh-packagekit.conf

 

sudo shutdown -h now

()

 

Now, I have my own Linux image. I’ll use it to create the different Linux VMs I need in my cluster.

 

Instanciate Linux virtual machines

 

 

sudo -s

vi /etc/hosts

 

 

sudo -s

fdisk -l

grep SCSI /var/log/messages

 

fdisk /dev/sdc

mkfs.ext3 /dev/sdc1

mkdir /mnt/datadrive

mount /dev/sdc1 /mnt/datadrive

vi /etc/fstab

add the following line at the end of the file:

/dev/sdc1 /mnt/datadrive ext3 defaults 1 2

 

The other machines in the cluster can be created thru the portal too. But that can also be done thru a script. There are two main flavors of automation scripts in Windows Azure: the Windows Azure PowerShell module can be used from Windows machines, the Command Line Interface (CLI for short, more information) which is based on Node.js can be used from Windows, Mac and Linux. They can be downloaded from http://www.windowsazure.com/en-us/downloads/.

As I’m a Windows guy, I will use PowerShell here. The details on how to start with the Windows Azure PowerShell cmdlets is available here.

Here is the script I use to create n124w1, n124w2 and n124w3:

 

 

Import-Module azure

 

#select default subscription and storage account

$subscription = ‘Azure bengui’

Set-AzureSubscription -SubscriptionName $subscription -CurrentStorageAccount ‘northeurope2affstorage’

Set-AzureSubscription -DefaultSubscription $subscription

 

#$adminPassword="******obfuscated*****"

+#region secret

 #endregion

 

#create an empty collection of VMConfigs

$vms = @()

 

#loop on the three VM we want to create

for($i=1; $i -le 3; $i++)

{

    Write-Host "creating n124w${i}OS"

    $sshPort = 52200 + $i

 

    #create a new VM Config

    $newVM = `

        New-AzureVMConfig -ImageName myCentOSImage -InstanceSize Medium -Name "n124w$i" `

            -AvailabilitySetName "n124wAvailabilitySet" -DiskLabel "n124w${i}os" `

            -HostCaching ReadWrite -Label "n124w$i" |

        Add-AzureProvisioningConfig -Linux -LinuxUser "benjguin" -Password $adminPassword |

        Add-AzureDataDisk -CreateNew -DiskLabel n124w${i}data1 -DiskSizeInGB 100 -LUN 0 |

        Add-AzureEndpoint -LocalPort 22 -Name "SSH$i" -Protocol tcp -PublicPort $sshPort |

        Set-AzureSubnet ‘Subnet-3’

   

    #add the VM config to the collection

    $vms += ,$newVM

}

 

#show the collection

$vms | format-table

 

#create the VM

New-AzureVM -ServiceName ‘n124’ -VMs $vms -VNetName ‘n124’

 

This generates the following output:

creating n124w1OS

creating n124w2OS

creating n124w3OS

AvailabilitySetName   ConfigurationSets     DataVirtualHardDisks  Label                 OSVirtualHardDisk     RoleName              RoleSize             RoleType           
——————-   —————–     ——————–  —–                 —————–     ——–              ——–             ——–           
n124wAvailabilitySet  {n124w1, Microsoft… {}                    bjEyNHcx              Microsoft.Samples…. n124w1                Medium               PersistentVMRole   
n124wAvailabilitySet  {n124w2, Microsoft… {}                    bjEyNHcy              Microsoft.Samples…. n124w2                Medium               PersistentVMRole   
n124wAvailabilitySet  {n124w3, Microsoft… {}                    bjEyNHcz              Microsoft.Samples…. n124w3                Medium               PersistentVMRole   


WARNING: VNetName, DnsSettings, DeploymentLabel or DeploymentName Name can only be specified on new deployments.

OperationDescription                                      OperationId                                               OperationStatus                                         
——————–                                      ———–                                               —————                                         
New-AzureVM – Create VM n124w1                            652a8988-df51-4777-94ae-bc7764296b0b                      Succeeded                                               
New-AzureVM – Create VM n124w2                            5b165d43-dcc7-4f5d-a1fc-ae77c6fa62dd                      Succeeded                                               
New-AzureVM – Create VM n124w3                            70750433-9f19-493c-9fdd-a8aad69fa193                      Succeeded                                               

 

and that can be seen in the Windows Azure management portal:

 

The DNS server can be manually updated in order to obtain the following configuration:

Then, from n124m, I can connect to other machines like this:

From there, I can update /etc/hosts in order to have hostname and hostname -f having the right names.

I can also format the data disks.

These are the same steps as before for n124w*. To summarize, here a few screen shots for n124w2:

 

mkfs.ext3 /dev/sdc1

mkdir /mnt/datadrive

mount /dev/sdc1 /mnt/datadrive

vi /etc/fstab

add the following line at the end of the file:

/dev/sdc1 /mnt/datadrive ext3 defaults 1 2

(same for other n124w* machines).

 

Now I have a whole set of machines in the same network, and they are ready to have Hadoop installed on them.

Install Hadoop distribution

I can now start at step 2 of the installation in Hortonworks’ documentation.

connect to n124m

sudo su

rpm -Uvh http://public-repo-1.hortonworks.com/ambari/centos6/1.x/GA/ambari-1.x-1.el6.noarch.rpm

from http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.2/bk_using_Ambari_book/content/ambari-chap2.1.2.html,

yum install epel-release

y

yum install ambari-server

y

()

from http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.2/bk_using_Ambari_book/content/ambari-chap2-2.html

ambari-server setup

(I use default answers during the setup)

from http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.2/bk_using_Ambari_book/content/ambari-chap2-3.html

ambari-server start

I am now at step 3 of the Hortonworks’ documentation

http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.2/bk_using_Ambari_book/content/ambari-chap3-1.html states to connect to http://{main.install.hostname}:8080. I do that from the Windows Server machine which is in the local network where the Linux machines are.

NB: in order to connect to n124dns, one simple way is to select this VM in the Windows Azure management portal and click Connect:

In a remote desktop on the n124dns machine, I do the following:

I connect with admin/admin

Next screen asks for the .ssh/id_rsa key. The simplest way is to copy it from an ssh session ([email protected]:22 in my case) and paste it in the browser.

 

in the customize services web page, I have

I choose to remove /mnt/resource folders because it corresponds to a disk that lives with the VM and which is not persisted in the Windows Azure storage folder. This disk is destroyed when one destroys the VM. HDFS would support it, but I want to be able to stop my whole cluster without loosing HDFS data. So I change to the following:

I do the same in tab MapReduce

I also enter the required passwords and click Next

the detail is the following

 

Admin Name : admin

Cluster Name : n124hdp

Total Hosts : 4 (4 new)

Local Repository : No

Services

    HDFS

      NameNode : n124m.n124.benjguin.com
      SecondaryNameNode : n124w1.n124.benjguin.com
      DataNodes : 3 hosts
    MapReduce

      JobTracker : n124m.n124.benjguin.com
      TaskTrackers : 3 hosts
    Nagios

      Server : n124m.n124.benjguin.com
    Ganglia

      Server : n124m.n124.benjguin.com
    Hive + HCatalog

      Hive Metastore : n124m.n124.benjguin.com
      Database : MySQL (New Database)
    HBase

      Master : n124m.n124.benjguin.com
      Region Servers : 3 hosts
    Oozie

      Server : n124m.n124.benjguin.com
    ZooKeeper

      Servers : 3 hosts

 

 

 

 

Run

Let’s now test HDFS, PIG and HIVE by ourselves in this cluster.

I open a new SSH connection to the master node (n124m, available at n124.cloudapp.net:22)

let’s copy /etc/paswd file to /hwork/in HDFS folder

[benjguin@n124m ~]$ cp test.pig /tmp

[benjguin@n124m ~]$ cd /tmp

[benjguin@n124m tmp]$ cat test.pig

A = load ‘/hwork/in’ using PigStorage(‘:’);

B = foreach A generate \$0 as id;

store B into ‘/hwork/out1’;

[benjguin@n124m tmp]$ sudo -u hdfs pig test.pig

2013-04-05 10:54:08,586 [main] INFO  org.apache.pig.Main – Apache Pig version 0.10.1.23 (rexported) compiled Mar 28 2013, 12:20:36

2013-04-05 10:54:08,587 [main] INFO  org.apache.pig.Main – Logging error messages to: /tmp/pig_1365159248577.log

2013-04-05 10:54:09,141 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to hadoop file system at: hdfs://n124m.n124.benjguin.com:8020

2013-04-05 10:54:09,318 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to map-reduce job tracker at: n124m.n124.benjguin.com:50300

2013-04-05 10:54:10,104 [main] INFO  org.apache.pig.tools.pigstats.ScriptState – Pig features used in the script: UNKNOWN

2013-04-05 10:54:10,294 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler – File concatenation threshold: 100 optimistic? false

2013-04-05 10:54:10,323 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer – MR plan size before optimization: 1

2013-04-05 10:54:10,323 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer – MR plan size after optimization: 1

2013-04-05 10:54:10,431 [main] INFO  org.apache.pig.tools.pigstats.ScriptState – Pig script settings are added to the job

2013-04-05 10:54:10,469 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3

2013-04-05 10:54:10,471 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – creating jar file Job5927476079760658662.jar

2013-04-05 10:54:14,037 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – jar file Job5927476079760658662.jar created

2013-04-05 10:54:14,061 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – Setting up single store job

2013-04-05 10:54:14,101 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 1 map-reduce job(s) waiting for submission.

2013-04-05 10:54:14,603 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 0% complete

2013-04-05 10:54:15,036 [Thread-7] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat – Total input paths to process : 1

2013-04-05 10:54:15,036 [Thread-7] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths to process : 1

2013-04-05 10:54:15,052 [Thread-7] WARN  org.apache.hadoop.io.compress.snappy.LoadSnappy – Snappy native library is available

2013-04-05 10:54:15,053 [Thread-7] INFO  org.apache.hadoop.util.NativeCodeLoader – Loaded the native-hadoop library

2013-04-05 10:54:15,053 [Thread-7] INFO  org.apache.hadoop.io.compress.snappy.LoadSnappy – Snappy native library loaded

2013-04-05 10:54:15,056 [Thread-7] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths (combined) to process : 1

2013-04-05 10:54:16,311 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – HadoopJobId: job_201304050850_0005

2013-04-05 10:54:16,311 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – More information at: http://n124m.n124.benjguin.com:50030/jobdetails.jsp?jobid=job_201304050850_0005

2013-04-05 10:54:26,873 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 50% complete

2013-04-05 10:54:30,936 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 100% complete

2013-04-05 10:54:30,938 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats – Script Statistics:

HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features

1.1.2.23        0.10.1.23       hdfs    2013-04-05 10:54:10     2013-04-05 10:54:30     UNKNOWN

Success!

Job Stats (time in seconds):

JobId   Maps    Reduces MaxMapTime      MinMapTIme      AvgMapTime      MaxReduceTime   MinReduceTime   AvgReduceTime  Alias    Feature Outputs

job_201304050850_0005   1       0       3       3       3       0       0       0       A,B     MAP_ONLY        /hwork/out1,

Input(s):

Successfully read 45 records (2588 bytes) from: "/hwork/in"

Output(s):

Successfully stored 45 records (304 bytes) in: "/hwork/out1"

Counters:

Total records written : 45

Total bytes written : 304

Spillable Memory Manager spill count : 0

Total bags proactively spilled: 0

Total records proactively spilled: 0

Job DAG:

job_201304050850_0005


2013-04-05 10:54:30,954 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – Success!

[benjguin@n124m tmp]$

so PIG works!

 

let’s now do the same thing with HIVE:

that works too!

 

 

Automate the cluster [de]provisioning

Once I have the VMs setup, I want to be able to stop paying for compute by shutting down and removing the VMs (while still keeping the virtyual hard disks VHD).

Il also want to be able to restart the whole cluster quite quickly.

As seen before in this post, it is possible to automate by scripting. Here are the scripts I use to stop and start this cluster.

NB: I took the OS disk names and data disk names from the Windows Azure portal

 

To shut up the cluster, I use

 

#region init

Import-Module ‘c:\Program Files (x86)\Microsoft SDKs\Windows Azure\PowerShell\Azure\Azure.psd1’

 

$subscription = ‘Azure bengui’

Set-AzureSubscription -SubscriptionName $subscription -CurrentStorageAccount ‘northeurope2affstorage’

Set-AzureSubscription -DefaultSubscription $subscription

 

$cloudSvcName = ‘n124’

#endregion

 

#region shutdown and delete

echo ‘will shut down and remove the following’

#$vms = Get-AzureVM -ServiceName $cloudSvcName | where { !($_.name -eq ‘n124dns’) }

$vms = Get-AzureVM -ServiceName $cloudSvcName

$vms | select name

$vms | Stop-AzureVM

$vms | Remove-AzureVM

#endregion

 

 

 

 

To startup the cluster, I use:

 

 

Import-Module azure

 

$subscription = ‘Azure bengui’

Set-AzureSubscription -SubscriptionName $subscription -CurrentStorageAccount ‘northeurope2affstorage’

Set-AzureSubscription -DefaultSubscription $subscription

 

$cloudSvcName = ‘n124’

$vNetName = ‘n124’

 

$vms = @()

 

$vmName=‘n124dns’

$n124dns = New-AzureVMConfig -DiskName ‘n124dns-n124dns-0-20130117132807’ -InstanceSize Small -Name $vmName -Label $vmName |

    Set-AzureSubnet ‘Subnet-1’ |

    Add-AzureEndpoint -LocalPort 3389 -Name ‘RDP’ -Protocol tcp -PublicPort 3389

$vms += ,$n124dns

 

$vmName=‘n124m’

$vm1 = New-AzureVMConfig -DiskName ‘n124-n124m-2013-04-03’ -InstanceSize Medium -Name $vmName -Label $vmName |

    Add-AzureDataDisk -DiskName ‘n124-n124m-0-201304031626300873’ -Import -LUN 0 |

    Add-AzureEndpoint -LocalPort 22 -Name ‘SSH’ -Protocol tcp -PublicPort 22 |

    Set-AzureSubnet ‘Subnet-2’

 

$vms += ,$vm1

 

$osDiskNames = @(‘n124-n124w1-0-201304041529430938’, `

    ‘n124-n124w2-0-201304041530370691’, `

    ‘n124-n124w3-0-201304041532060509’)

 

$dataDiskNames = @(‘n124-n124w1-0-201304041529470297’, `

    ‘n124-n124w2-0-201304041530410363’, `

    ‘n124-n124w3-0-201304041532090540’)

 

#loop on the three VM we want to create

for($i=1; $i -le 3; $i++)

{

    Write-Host "creating n124w${i}OS"

    $sshPort = 52200 + $i

 

    $j = $i 1

 

    #create a new VM Config

    $newVM = `

        New-AzureVMConfig -DiskName $osDiskNames[$j] -InstanceSize Medium -Name "n124w$i" `

            -AvailabilitySetName "n124wAvailabilitySet" -Label "n124w$i" |

        Add-AzureDataDisk -DiskName $dataDiskNames[$j] -Import -LUN 0 |

        Add-AzureEndpoint -LocalPort 22 -Name "SSH$i" -Protocol tcp -PublicPort $sshPort |

        Set-AzureSubnet ‘Subnet-3’

   

    #add the VM config to the collection

    $vms += ,$newVM

}

 

#show the collection

$vms | format-table

 

#create the VM

New-AzureVM -ServiceName ‘n124’ -VMs $vms -VNetName ‘n124’

 

So I now have a Linux cluster with Hadoop installed and I can start, and stop it (without loosing HDFS data) with scripts.

 

Benjamin

Blog Post by: Benjamin GUINEBERTIERE

Moving the BizTalk databases (part 2)

Skipping ahead a bit

After some feedback about the last article (thanks everyone) you all seem to know how to follow instructions from Microsoft. You all told me to skip to the part about the instructions being wrong. So I will.

Make sure to read the entire article.

32 vs 64 bit

Ever heard of that issue? Ever run into trouble because of that? Of course you have! Now you might be able to avoid it.

When you reach step 11 in the article “How to restore your databases” you run into your first mistake by the author at Microsoft. It clearly states that if you run a 64 bit environment you should run the script using a 64 bit command prompt. That is just plain wrong! Here is why:

Reading registry settings the wrong way

Open the file C:\Program Files (x86)\Microsoft BizTalk Server 2010\Bins32\Schema\Restore\UpdateRegistry.vbs and take a look at line 23. It reads: bKey = WshShell.RegRead("HKLM\SOFTWARE\Microsoft\BizTalk Server\3.0\Administration\MgmtDBServer").

If you run this script in a 64-bit command prompt the script will actually look for this key: HKLM\SOFTWARE\Wow6432Node\Microsoft\BizTalk Server\3.0\Administration\MgmtDBServer. Now in this row there is nothing wrong with that as it will find a key.

The trouble comes a couple of rows down: WshShell.RegWrite will only update the 64-bit registry settings(!) and all your 32-bit hosts will fail.

So in order for the script to work you have to run it using a normal administrator command window (32-bit).

In order to make sure the UpdatedRegistry-script updated all the settings I put together this script. Download and change the extension to vbs. Run it and see that all the settings were updated correctly.

Wrong kind of error handling

Go back to line 24. It says if (err = 0) which seems like a good way to handle errors. It the setting is not found, an error is handled. Wrong! A failed read does not set an error code. It simply returns an empty value. So instead you should handle it using this: if (bKey).

Another obvious mistake

Now go to line 131. It says to look for an attribute called EDI. If it finds this attribute, the EDI settings will be updated. However step 11 in the guide says you should update your SampleUpdateInfo.xml to include an attribute called MsEDIAS2. Update the line in the script to: set node = configObj.selectSingleNode("/UpdateConfiguration/OtherDatabases/Database[@Name='MsEDIAS2']").

The last(?) obvious mistake

Now this is scary, because it effects the SSO database. It does not update any registry setting about the SSO database, but it uses the program called Ssoconfig to change the database name. This is done by using the %path%-parameter called “CommonProgramFiles”, which is awesome. However, dead wrong as it points to the 32-bit common program files (if you use a 64-bit cmd) and Ssoconfig is not installed there. The effect is that the SSO database is not moved and the script does not even throw an error.

This is very easy to test:

  1. Open a 32-bit command-prompt and type in the following: cd "%CommonProgramFiles%.
  2. This will change the directory to “c:\ProgramFiles\Common Files”, which is where Ssoconfig is located.
  3. Now try the same on a 64-bit command prompt (located at C:\Windows\SysWOW64\cmd.exe) and run the same command.
  4. Result C:\Program Files (x86)\Common Files. That is NOT the right path.

The solution is clear. Run the UpdateRegistry.vbs under a 32-bit command prompt.

Now to make it easier for you, I have made a new version of the script available here (change the extension after download).

The UpdateDatabase-script

This contains very few strange things but once again run it under a 32-bit cmd as keys are read. I also commented out everything about Hws. If you use it (strange person) then leave it in.

BAM databases and other BAM components

If you are using BAM in your environment, good for you. Be sure to follow this article to the letter after moving the databases and running the scripts. http://msdn.microsoft.com/en-us/library/aa561586.aspx It basically points everything (the BAM portal and BAM WS) to the new database.

Finishing up

Ok so there are a couple of more pointers. The jobs you scripted for BizTalk, some have to be updated by hand (no biggie).

  1. BizTalk Backup has a database server name in it.
  2. Operations_OperateOnInstances_OnMaster_BizTalkMsgBoxDb also has a server name in it.
  3. Rules_Database_Cleanup_BizTalkRuleEngineDb.sql run this when/if the Rule engine DB has been created on the new server.
  4. TrackedMessages_Copy_BizTalkMsgBoxDb.sql also has a server name in it.

Closing words

I would love feedback on this! Please use the contact form and send me an email.

Remember; This is not really all that hard and if you practice you can move an entire production environment in under an hour.

Blog Post by: Mikael Sand

How to configure SQL Server Network Protocols in the SQL Server that houses BizTalk Server databases with PowerShell

How to configure SQL Server Network Protocols in the SQL Server that houses BizTalk Server databases with PowerShell

Today is a special day for me, it’s my birthday, and I will continue the tradition of giving something to the community on this day (I started this tradition last year) In the past I wrote a series of posts describing step by step how to install and configure BizTalk Server 2010. One of this […]
Blog Post by: Sandro Pereira

BizTalk Monitoring Solutions (SCOM vs BizTalk360) white paper by Kent Weare

BizTalk Server being a middleware platform, most of your important business transactions will start to flow through your BizTalk environments. It will soon become a no brainer, to start monitoring the health and availability of your BizTalk environments. There are few options available for you to monitor your BizTalk environments. Most of the popular enterprise […]

The post BizTalk Monitoring Solutions (SCOM vs BizTalk360) white paper by Kent Weare appeared first on BizTalk360 Blog.

Blog Post by: Saravana Kumar

Windows Azure IaaS and Creating a New Azure Virtual Machine Video

I have put together a quick 10 minute video that introduced Windows Azure IaaS and walks you through creating your first Windows Azure Virtual Machine! 

I put a lot of time into adding cool animations and fun dialog, or so I think.

[View:http://www.youtube.com/watch?v=zeNb6LDvVzQ]

Download this video from BizTalkGurus.com or view on YouTube.

Enjoy and let me know what you think!

BizTalk Monitoring Whitepaper is now available

For the past month and half, in my spare time, I have been working on a whitepaper that compares and contrasts System Center Operations Manager (SCOM) and BizTalk360. 

I have used both products extensively in my professional life and in order to remain objective within the paper I have driven my evaluation off of requirements.  These requirements are based upon my experience of managing the 24×7 operations of a BizTalk environment at a large Power Distribution company in Canada.

The paper can be downloaded from here and feel free to leave any comments that you may have below.

I would also like to thank Steef-Jan Wiggers for being a technical reviewer on this paper.  He provided me with some valuable feedback that increased the quality of the paper.