Chapter 5. Upgrade Ambari

This topic describes how to prepare for, and then perform upgrading Ambari. You must upgrade Ambari to version 2.5.2 before upgrading the Ambari-managed IOP cluster to an Ambari-managed HDP-2.6.2 cluster. The Ambari version used for the IOP-HDP upgrade is 2.5.2.

This document describes the procedure for upgrading Ambari, before upgrading the Ambari-managed IOP cluster to an Ambari-managed HDP cluster. The target Ambari version used for the upgrade is 2.5.2.

Preparing to Upgrade Ambari

Be sure to review the Ambari 2.5.2.0 Release Notes for Known Issues and Behavioral Changes.
You must have root, administrative, or root-equivalent authorization on the Ambari server host and all servers in the cluster.
You must backup the Ambari Server database.
You must make a safe copy of the Ambari Server configuration file found at /etc/ambari-server/conf/ambari.properties.
Plan to upgrade the Ambari Metrics service:
- Record the location of the Metrics Collector component before you begin the upgrade process.
- You must stop the Ambari Metrics service from Ambari Web.
- After upgrading Ambari, you must also upgrade Ambari Metrics System and add the Grafana component.
Optional: You may add the following property to the ambari.properties file.
stack.upgrade.bypass.prechecks=true
Adding this property allows the Express Upgrade process to start,if pre-check failures occur.
If your cluster has no access to the Internet, you must set up a local repository before you continue.

	Note
	During Ambari upgrade, the existing `/var/lib/ambari-server/ambari-env.sh` file is overwritten. A backup copy of `ambari-env.sh` (with extension `.rpmsave`) is created. If you have manually modified `ambari-env.sh` (for example, to change Ambari Server heap size), you will need to re-apply your changes to the new file manually.

Performing an Ambari Upgrade

If you are running Ambari Metrics service in your cluster, stop the service. From Ambari Web, browse to Services > Ambari Metrics and select Stop from the Service Actions menu.
Stop the Ambari Server. On the host running Ambari Server:
```
ambari-server stop
```
Stop all Ambari Agents. On each host in your cluster running an Ambari Agent:
```
ambari-agent stop
```

Fetch the new Ambari repo and replace the old repository file with the new repository file on all hosts in your cluster.

	Important
	Check your current directory before you download the new repository file to make sure that there are no previous versions of the `ambari.repo` file. If you do not, and a previous version exists, the new download will be saved with a numeric extension, such as `ambari.repo.1`. Make sure that the version you copy is the new version.

Select the repository appropriate for your environment:

For RHEL/CentOS/Oracle Linux 6:

wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

For RHEL/CentOS/Oracle Linux 7:

wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

For IBM Power Systems:

wget -nv http://public-repo-1.hortonworks.com/ambari/centos7-ppc/2.x/updates/2.5.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

	Note
	If your cluster does not have access to the Internet, set up a local repository before you continue.

	Note
	Ambari Server does not automatically turn off `iptables`. After upgrading the server, you must either disable `iptables` manually or make sure that you have appropriate ports available on all cluster hosts.

Upgrade Ambari Server. On the host running Ambari Server:
```
yum clean all
```
```
yum info ambari-server
```
In the info output, visually validate that there is an available version containing "2.5.2"
```
yum upgrade ambari-server ambari-agent
```

Check for upgrade success by noting progress during the Ambari Server installation process you started in Step 5.

As the process runs, the console displays output similar, although not identical, to the following:
Setting up Upgrade Process Resolving Dependencies --> Running transaction check
If the upgrade fails, the console displays output similar to the following:
Setting up Upgrade Process No Packages marked for Update
A successful upgrade displays output similar to the following:
Updated: ambari-server-2.5.2.0-220.x86_64 Complete!

	Note
	Confirm there is only one ambari-server.jar file in `/usr/lib/ambari-server`. If there is more than one JAR file with name ambari-server.jar, move all other ambari-server*.jar files except for the ambari-server-2.5.2.jar to `/tmp` before proceeding with upgrade.

Upgrade all Ambari Agents. On each host in your cluster running an Ambari Agent:
```
yum upgrade ambari-agent
```
After the upgrade process completes, check each host to make sure the new files have been installed:
```
rpm -qa | grep ambari-agent
```
Upgrade Ambari Server database schema. On the host running Ambari Server:
```
ambari-server upgrade
```
If you have configured Ambari to authenticate using LDAP or Active Directory, you must re-run
```
ambari-server setup-ldap
```
If you have configured your cluster for Hive or Oozie with an external database (Oracle, MySQL or PostgreSQL), you must re-run
```
ambari-server setup --jdbc-db={database-type} --jdbc-driver={/jdbc/driver/path}
```
to get the appropriate JDBC driver .jar file in place. For example:
```
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
```
If the original IOP cluster was configured for a non-root user, you must update /etc/sudoers file using the visudo command in all the nodes in your cluster, to remove sudoer permissions for the non-root user added in IOP non-root configuration. You should refresh the sudoers file to the original settings, as described here.
Start the Ambari Server. On the host running Ambari Server:
```
ambari-server start
```
Start all Ambari Agents. On each host in your cluster running an Ambari Agent:
```
ambari-agent start
```
Open Ambari Web.
Point your browser to http://<your.ambari.server>:8080
where <your.ambari.server> is the name of your Ambari Server host. For example, c6401.ambari.apache.org.
Important
Refresh your browser so that it loads the new version of the Ambari Web code. If you have problems, clear your browser cache manually, then restart Ambari Server.
Log in, using the Ambari administrator credentials that you have set up.
For example, the default name/password is admin/admin.
You will see a Restart indicator next to each service after upgrading. The Ambari upgrade has added to/adjusted the configuration properties of your cluster based on new configuration types and properties being made available for each service with Ambari-2.5.2. Review these changes by comparing the previous configuration with the latest version created by the Ambari upgrade.
Note
HDFS may fail to restart after upgrading Ambari on a NameNode HA-enabled, IOP cluster.
To resolve this issue, see Troubleshooting Migration Issues.

Next Steps

Post-Upgrade Tasks Mandatory

	Important
	Completing post-upgrade tasks is mandatory.

More Information

Using a Local Repository

Configuring Network Port Numbers

Set Up LDAP or Active Directory Authentication

Upgrade Ambari Metrics

Add Grafana

Mandatory Post-Upgrade Tasks

Depending on the configuration of your cluster and your current Ambari version, you must upgrade any of the following features in your cluster, as described in the following topics:

Upgrading Ambari Metrics

If your cluster includes the Ambari Metrics System (AMS) service, you must upgrade the system along with Ambari.

Adding Grafana to Ambari Metrics

If you are upgrading from IOP-4.2.0 or earlier, your Ambari Metrics service does not contain the Grafana component and it will need to be added for full functionality.

Upgrading Configurations

Certain scenarios may require that you modify configurations that Ambari did not upgrade automatically.

	Note
	If you are upgrading from IOP-4.2.5, you must upgrade Ambari Infra and upgrade Ambari Log Search.

Disable Audit to Solr

Only if you are migrating from IOP 4.2.5 cluster with Ranger and Solr.

Next Steps

Restart services, only after you complete all applicable post-upgrade tasks.

Upgrading Ambari Metrics

Prerequisites

Upgrade to Ambari 2.5.2 and perform needed post-upgrade checks. Make sure all services are up and healthy.

Steps

Make sure Ambari Metrics service is stopped. From Ambari Web, browse to Services > Ambari Metrics and select Stop from the Service Actions menu.
On every host in your cluster running a Metrics Monitor, run the following commands:
```
yum clean all
```
```
yum upgrade ambari-metrics-monitor ambari-metrics-hadoop-sink
```
Execute the following command on all hosts running the Metrics Collector:
```
yum upgrade ambari-metrics-collector
```
Execute the following command on the host running the Grafana component:
```
yum upgrade ambari-metrics-grafana
```
Start the Ambari Metrics Service.
From Ambari Web, browse to Services > Ambari Metrics select Service Actions then choose Start.
Updated Ambari Metrics Sink jars will be installed on all hosts and you must restart each service to pick up the latest sink implementations.
Please wait to restart all services until after you have completed all applicable post-upgrade tasks, for example: HDFS, YARN, Kafka, HBase, Flume, Storm.

Next Steps

Restart services, only after you complete all applicable, post-upgrade tasks.
Note
New Ambari Metrics Sinks will not be activated until all services are restarted.
If you are upgrading from IOP 4.2.0 or earlier, your Ambari Metrics service does not contain Grafana. Proceed to add Grafana to Ambari Metrics.

More Information

Add Grafana to Ambari Metrics

Adding Grafana to Ambari Metrics

As of Ambari 2.4, Grafana is included as a component of Ambari Metrics. This section walks you through how to add Grafana to the system and install Grafana on a host in the cluster.

	Note
	When using the API commands below, be sure to replace the ambari.server with the Ambari Server hostname, cluster.name with your cluster name and host.name with the host where you will run Grafana. This can be the same host that is running the Metrics Colllector.

Upgrade to Ambari 2.5.2 and perform needed post-upgrade checks. Make sure all services are up and healthy.

Add the METRICS_GRAFANA component to Ambari:

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST
http://ambari.server:8080/api/v1/clusters/cluster.name/services/AMBARI_METRICS/components/METRICS_GRAFANA

Add METRICS_GRAFANA to a host in the cluster.

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST -d
'{"host_components":[{"HostRoles":{"component_name":"METRICS_GRAFANA"}}]}'
http://ambari.server:8080/api/v1/clusters/cluster.name/hosts?Hosts/host_name=host.name

From Ambari Web, browse to Services > Ambari Metrics and you will see Grafana is in the Install Pending… state.
You need to complete the configuration of Grafana before installing and starting.
To complete the configuration, click on Services > Ambari Metrics > Configs and enter the default Grafana Admin Password in the General section. Click Save.
Browse to Hosts > host.name (the host.name used in the API call where you added Grafana). You will see the Grafana component is in an Install Pending… state. Use the Install Pending… action button and select Re-install.
Once the install operation completes, select Start to start Grafana.
To access Grafana, browse to Services > Ambari Metrics, select Quick Links and then click Grafana.

More Information

Using Grafana

Upgrading Configurations

This section describes potential cluster configuration updates that may be required.

Upgrading Kerberos krb5.conf

Ambari has added support for handling more than one KDC host . Only one kadmin host is supported by the Kerberos infrastructure. This required modifications for the krb5.conf template. In order for Ambari to properly construct the krb5.conf configuration file, make the following configuration change if your cluster meets all of these criteria:

Kerberos is enabled and Ambari is configured for automated setup, and
Ambari is managing the krb5.conf, and
You have modified the krb5.conf template content from the default content. If you have not modified the default content, Ambari will automatically update the template content as part of upgrade and these configuration updates do not need to be applied manually.

If you meet all of the above criteria, you must update the krb5.conf template content found in Services > Kerberos > Advanced:

Original Template Entry	Updated Template Entry
admin_server = {{admin_server_host\|default(kdc_host, True)}}	admin_server = {{admin_server_host\|default(kdc_host_list[0]\|trim(), True)}}
kdc = {{kdc_host}}	{% for kdc_host in kdc_host_list %} kdc = {{kdc_host\|trim()}} {%- endfor -%}

Original Template Entry

Updated Template Entry

admin_server = {{admin_server_host|default(kdc_host, True)}}

admin_server = {{admin_server_host|default(kdc_host_list[0]|trim(), True)}}

kdc = {{kdc_host}}

{% for kdc_host in kdc_host_list %}

kdc = {{kdc_host|trim()}}

{%- endfor -%}

More Information

Configure Ambari for Automated Setup

Switch from IBM Open JDK to Oracle JDK

If you are migrating from an IOP cluster with the IBM Open JDK on x86 hardware, we recommend that you modify your cluster to use the Oracle JDK (the default JDK for HDP clusters).

Verify current settings in the /etc/ambari-server/conf/ambari.properties file:

cat /etc/ambari-server/conf/ambari.properties | grep java
java.home=/usr/jdk64/java-1.8.0-openjdk-1.8.0.121-0.b13.el6_8.x86_64
java.releases=jdk1.8,jdk1.7
java.releases.ppc64le=
openjdk1.8.re=(java.*)/jre

Download and install Oracle JDK8 on all hosts in the cluster.

Note

For IOP clusters operating on IBM Power Systems, continue to use OpenJDK. You should:

For a Kerberized cluster, you should also:

Modify the java.home variable in the /etc/ambari-server/conf/ambari.properties file to point to the downloaded Oracle Open JDK home, example:
```
java.home=/usr/java/jdk1.8.0_112
```
Restart Ambari server.
```
ambari-server restart
```
When all services are re-started as a part of the Express Upgrade, the Oracle JDK will be used.

Disable Audit to Solr

If you are migrating from IOP 4.2.5 cluster with Ranger and Solr, make sure to:

Disable Audit to Solr, on the Ranger Service Configs page, as shown in the following example:
To allow for Ranger/Solr plugin migration.
Then, perform the restart required operation for this change to take effect.

	Important
	Refresh your browser so that it loads the new version of the Ambari Web code. If you have problems, clear your browser cache manually, then restart Ambari Server.

	Note
	New Ambari Metrics Sinks will not be activated until all services are restarted.