Chapter 14. Configuring Storm for Supervision

Ambari administrators should install and configure a process controller to monitor and run Apache Storm under supervision. Storm is a fail-fast application, meaning that it is designed to fail under certain circumstances, such as a runtime exception or a break in network connectivity. Without a watchdog process, these events can quickly take down an entire Storm cluster in production. A watchdog process prevents this by monitoring for failed Storm processes and restarting them when necessary. This section describes how to configure supervisord to manage the Storm processes, but adminstrators may use another process controller of their choice, such as monit or daemontools .

	Note
	Running Storm under supervision is only supported for Nimbus Server and Supervisors.

To configure Storm for operating under supervision:

Stop all Storm components.
Using Ambari Web Services > Storm > Service Actions, choose Stop, then wait until stop completes.
Stop Ambari Server.
```
ambari-server stop
```

Change Supervisor and Nimbus command scripts in the Stack definition.

On Ambari Server host, run:

sed -ir "s/scripts\/supervisor.py/scripts\/supervisor_prod.py/g" /var/lib/ambari-server/resources/stacks/HDP/2.1/services/STORM/metainfo.xml
sed -ir "s/scripts\/nimbus.py/scripts\/nimbus_prod.py/g" /var/lib/ambari-server/resources/stacks/HDP/2.1/services/STORM/metainfo.xml

Install supervisord on all Nimbus and Supervisor hosts.
1. Install EPEL repository.
```
yum install epel-release -y 
```
2. Install supervisor package for supervisord.
```
yum install supervisor -y 
```
3. Enable supervisord on autostart.
```
chkconfig supervisord on 
```
4. Change supervisord configuration file permissions.
```
chmod 600 /etc/supervisord.conf 
```

Configure supervisord to supervise Nimbus Server and Supervisors.

Append the following to /etc/supervisord.conf on all Supervisor host and Nimbus hosts accordingly.

 [program:storm-nimbus]
command=env PATH=$PATH:/bin:/usr/bin/:/usr/jdk64/jdk1.7.0_45/bin/ JAVA_HOME=/usr/jdk64/jdk1.7.0_45 /usr/lib/storm/bin/storm nimbus
user=storm
autostart=true
autorestart=true
startsecs=10
startretries=999
log_stdout=true
log_stderr=true
logfile=/var/log/storm/nimbus.out
logfile_maxbytes=20MB
logfile_backups=10

 [program:storm-supervisor]
command=env PATH=$PATH:/bin:/usr/bin/:/usr/jdk64/jdk1.7.0_45/bin/ JAVA_HOME=/usr/jdk64/jdk1.7.0_45 /usr/lib/storm/bin/storm supervisor
user=storm
autostart=true
autorestart=true
startsecs=10
startretries=999
log_stdout=true
log_stderr=true
logfile=/var/log/storm/supervisor.out
logfile_maxbytes=20MB
logfile_backups=10

	Note
	Change /usr/jdk64/jdk1.7.0_45 accordingly to the location of the jdk for Ambari in your environment

Start Ambari Server.
```
ambari-server start
```

Legal notices