Defining a Whirr Cluster
After generating an SSH key pair, the only task left to do before using Whirr is to define a cluster by creating a properties file. You can name the properties file whatever you like. The example properties file used in these instructions is named hadoop.properties. Save the properties file in your home directory. After defining a cluster in the properties file, you will be ready to launch a cluster and run MapReduce jobs.
MRv1 Cluster
The following file defines a cluster with a single machine for the NameNode and JobTracker, and another machine for a DataNode and TaskTracker.
whirr.cluster-name=myhadoopcluster whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker whirr.provider=aws-ec2 whirr.identity=<cloud-provider-identity> whirr.credential=<cloud-provider-credential> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub whirr.env.repo=cdh5 whirr.hadoop-install-function=install_cdh_hadoop whirr.hadoop-configure-function=configure_cdh_hadoop whirr.hardware-id=m1.large whirr.image-id=us-east-1/ami-ccb35ea5 whirr.location-id=us-east-1
YARN Cluster
The following configuration provides the essentials for a YARN cluster. Change the number of instances for hadoop-datanode+yarn-nodemanager from 2 to a larger number if you need to.
whirr.cluster-name=myhadoopcluster whirr.instance-templates=1 hadoop-namenode+yarn-resourcemanager+mapreduce-historyserver,2 hadoop-datanode+yarn-nodemanager whirr.provider=aws-ec2 whirr.identity=<cloud-provider-identity> whirr.credential=<cloud-provider-credential> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub whirr.env.mapreduce_version=2 whirr.env.repo=cdh5 whirr.hadoop.install-function=install_cdh_hadoop whirr.hadoop.configure-function=configure_cdh_hadoop whirr.mr_jobhistory.start-function=start_cdh_mr_jobhistory whirr.yarn.configure-function=configure_cdh_yarn whirr.yarn.start-function=start_cdh_yarn whirr.hardware-id=m1.large whirr.image-id=us-east-1/ami-ccb35ea5 whirr.location-id=us-east-1