Configuring Oozie for Falcon
Falcon uses HCatalog for data availability notification when Hive tables are replicated. Make the following configuration changes to Oozie to ensure Hive table replication in Falcon:
Stop the Oozie service on all Falcon clusters. Run the following commands on the Oozie host machine.
su - $OOZIE_USER
/usr/hdp/current/oozie-server/bin/oozie-stop.sh
where $OOZIE_USER is the Oozie user. For example, oozie.
Copy each cluster's hadoop conf directory to a different location. For example, if you have two clusters, copy one to
/etc/hadoop/conf-1
and the other to/etc/hadoop/conf-2
.For each
oozie-site.xml
file, modify the oozie.service.HadoopAccessorService.hadoop.configurations property, specifying clusters, the RPC ports of the NameNodes, and HostManagers accordingly. For example, if Falcon connects to three clusters, specify:<property> <name>oozie.service.HadoopAccessorService.hadoop.configurations</name> <value>*=/etc/hadoop/conf,$NameNode:$rpcPortNN=$hadoopConfDir1,$ResourceManager1:$rpcPortRM=$hadoopConfDir1,$NameNode2=$hadoopConfDir2,$ResourceManager2:$rpcPortRM=$hadoopConfDir2,$NameNode3 :$rpcPortNN =$hadoopConfDir3,$ResourceManager3 :$rpcPortRM =$hadoopConfDir3</value> <description> Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is used when there is no exact match for an authority. The HADOOP_CONF_DIR contains the relevant Hadoop *-site.xml files. If the path is relative is looked within the Oozie configuration directory; though the path can be absolute (i.e. to point to Hadoop client conf/ directories in the local filesystem. </description> </property>
Add the following properties to the
/etc/oozie/conf/oozie-site.xml
file:<property> <name>oozie.service.ProxyUserService.proxyuser.falcon.hosts</name> <value>*</value> </property> <property> <name>oozie.service.ProxyUserService.proxyuser.falcon.groups</name> <value>*</value> </property> <property> <name>oozie.service.URIHandlerService.uri.handlers</name> <value>org.apache.oozie.dependency.FSURIHandler, org.apache.oozie.dependency.HCatURIHandler</value> </property> <property> <name>oozie.services.ext</name> <value>org.apache.oozie.service.JMSAccessorService, org.apache.oozie.service.PartitionDependencyManagerService, org.apache.oozie.service.HCatAccessorService</value> </property> <!-- Coord EL Functions Properties --> <property> <name>oozie.service.ELService.ext.functions.coord-job-submit-instances</name> <value>now=org.apache.oozie.extensions.OozieELExtensions#ph1_now_echo, today=org.apache.oozie.extensions.OozieELExtensions#ph1_today_echo, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph1_yesterday_echo, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_currentMonth_echo, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_lastMonth_echo, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph1_currentYear_echo, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph1_lastYear_echo, formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo </value> </property> <property> <name>oozie.service.ELService.ext.functions.coord-action-create-inst</name> <value>now=org.apache.oozie.extensions.OozieELExtensions#ph2_now_inst, today=org.apache.oozie.extensions.OozieELExtensions#ph2_today_inst, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday_inst, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth_inst, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth_inst, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear_inst, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear_inst, latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo, formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user </value> </property> <property> <name>oozie.service.ELService.ext.functions.coord-action-create</name> <value> now=org.apache.oozie.extensions.OozieELExtensions#ph2_now, today=org.apache.oozie.extensions.OozieELExtensions#ph2_today, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_currentWeek, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_lastWeek, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear, latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo, formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user </value> </property> <property> <name>oozie.service.ELService.ext.functions.coord-job-submit-data</name> <value> now=org.apache.oozie.extensions.OozieELExtensions#ph1_now_echo, today=org.apache.oozie.extensions.OozieELExtensions#ph1_today_echo, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph1_yesterday_echo, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph1_currentWeek_echo, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph1_lastWeek_echo, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_currentMonth_echo, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_lastMonth_echo, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph1_currentYear_echo, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph1_lastYear_echo, dataIn=org.apache.oozie.extensions.OozieELExtensions#ph1_dataIn_echo, instanceTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_wrap, formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo, user=org.apache.oozie.coord.CoordELFunctions#coord_user </value> </property> <property> <name>oozie.service.ELService.ext.functions.coord-action-start</name> <value> now=org.apache.oozie.extensions.OozieELExtensions#ph2_now, today=org.apache.oozie.extensions.OozieELExtensions#ph2_today, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear, latest=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latest, future=org.apache.oozie.coord.CoordELFunctions#ph3_coord_future, dataIn=org.apache.oozie.extensions.OozieELExtensions#ph3_dataIn, instanceTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_nominalTime, dateOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateOffset, formatTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_formatTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user </value> </property> <property> <name>oozie.service.ELService.ext.functions.coord-sla-submit</name> <value> instanceTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_fixed, user=org.apache.oozie.coord.CoordELFunctions#coord_user </value> </property> <property> <name>oozie.service.ELService.ext.functions.coord-sla-create</name> <value> instanceTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_nominalTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user </value> </property>
Copy the existing Oozie WAR file to
/usr/hdp/current/oozie/oozie.war.
This will ensure that all existing items in the WAR file are still present after the current update.su - root
cp $CATALINA_BASE/webapps/oozie.war /usr/hdp/current/oozie-server/oozie.war
where $CATALINA_BASE is the path for the Oozie web app. By default, $CATALINA_BASE is:
/usr/hdp/2.3.6.0-<$version>/oozie/oozie-server
.Add the Falcon EL extensions to Oozie.
Copy the extension JAR files provided with the Falcon Server to a temporary directory on the Oozie server. For example, if your standalone Falcon Server is on the same machine as your Oozie server, you can just copy the JAR files.
mkdir /tmp/falcon-oozie-jars
cp /usr/hdp/current/falcon-server/oozie/ext/falcon-oozie-el-extension-<$version>.jar /tmp/falcon-oozie-jars
cp /tmp/falcon-oozie-jars/falcon-oozie-el-extension-<$version>.jar /usr/hdp/2.3.6.0-<$version>/oozie/libext
Package the Oozie WAR file as the Oozie user
su - $OOZIE_USER
cd /usr/hdp/current/oozie-server/bin
./oozie-setup.sh prepare-war
Where $OOZIE_USER is the Oozie user. For example, oozie.
Start the Oozie service on all Falcon clusters. Run these commands on the Oozie host machine.
su - $OOZIE_USER
/usr/hdp/current/oozie-server/bin/oozie-start.sh
Where $OOZIE_USER is the Oozie user. For example, oozie.