Optional: Recreating the Ambari SSL Certificate Authority

The Ambari Certificate Authority, (CA) issues digital certificates used to securely connect Ambari Server and Ambari Agents. An Ambari CA certificate is valid for 365 days. If an Ambari CA certificate is corrupted, or has expired, you must recreate the CA, causing a new CA certificate and new SSL certificates for each Ambari agent to be created. This solution assumes that the certificates signed by the Ambari CA are replaceable - which is generally the case for certificates used by Ambari agents for two-way SSL connections.

On the Ambari server:

1. Stop the Ambari server: ambari-server stop.
2. Backup /var/lib/ambari-server/keys and its child directories.
3. Delete the following files from /var/lib/ambari-server/keys:
   • ca.key
   • ca.csr
   • ca.crt
   • pass.txt
   • keystore.p12
   • *.csr
   • *.crt
4. Delete the following files from /var/lib/ambari-server/keys/db:
   • index.txt.old
   • index.txt.attr.old
   • serial.old
5. Truncate the following files from /var/lib/ambari-server/keys/db:
   • index.txt
   • index.txt.attr
6. Edit the following file from /var/lib/ambari-server/keys/db:
   In serial, set the contents to be exactly:

   00

7. Delete all files under /var/lib/ambari-server/keys/db/newcerts.
8. Restart Ambari server: ambari-server restart.
   After restarting the Ambari server, the following (or similar) entries should be seen in the /var/log/ambari-server/ambari-server.log file:

   12 Jun 2017 14:38:19,606  INFO [main] ShellCommandUtil:63 - Command openssl genrsa -des3 -passout pass:**** -out /var/lib/ambari-server/keys/ca.key 4096  was finished with exit code: 0 - the operation was completely successfully
   12 Jun 2017 14:38:19,640  INFO [main] ShellCommandUtil:63 - Command openssl req -passin pass:**** -new -key /var/lib/ambari-server/keys/ca.key -out /var/lib/ambari-server/keys/ca.csr -batch was finished with exit code: 0 - the o
   peration was completely successfully.
   12 Jun 2017 14:38:19,683  INFO [main] ShellCommandUtil:63 - Command openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -key **** -selfsign -extensions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr was finished with exit code: 0 - the operation was completely successfully.
   12 Jun 2017 14:38:19,701  INFO [main] ShellCommandUtil:63 - Command openssl pkcs12 -export -in /var/lib/ambari-server/keys/ca.crt -inkey /var/lib/ambari-server/keys/ca.key -certfile /var/lib/ambari-server/keys/ca.crt -out /var/lib/ambari-server/keys/keystore.p12 -password pass:**** -passin pass:****
    was finished with exit code: 0 - the operation was completely successfully.
   12 Jun 2017 14:38:19,708  INFO [main] ShellCommandUtil:63 - Command find /var/lib/ambari-server/keys -type f -exec chmod 700 {} + was finished with exit code: 0 - the operation was completely successfully.
   12 Jun 2017 14:38:19,708  INFO [main] ShellCommandUtil:63 - Command chmod 600 /var/lib/ambari-server/keys/pass.txt was finished with exit code: 0 - the operation was completely successfully.
   12 Jun 2017 14:52:53,797  INFO [qtp-ambari-agent-34] CertificateManager:200 - Signing agent certificate
   12 Jun 2017 14:52:53,800  INFO [qtp-ambari-agent-34] CertificateManager:220 - Validating agent hostname: c6401.ambari.apache.org
   12 Jun 2017 14:52:53,800  INFO [qtp-ambari-agent-34] CertificateManager:232 - Verifying passphrase
   12 Jun 2017 14:52:53,849  INFO [qtp-ambari-agent-34] ShellCommandUtil:63 - Command openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/c6401.ambari.apache.org.csr -out /var/lib/ambari-server/keys/c6401.ambari.apache.org.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/ca.key -cert /var/lib/ambari-server/keys/ca.crt was finished with exit code: 0 - the operation was completely successfully.

On each Ambari agent host:

9. Stop the Ambari agent: ambari-agent stop.
10. Backup /var/lib/ambari-agent/keys and its child directories.
11. Delete the following files from /var/lib/ambari-server/keys:
    • ca.crt
    • *.crt
    • *.csr
    • *.key
12. Restart Ambari agent: ambari-agent restart.
    After restarting the Ambari agent, the following (or similar) entries should be seen in the /var/log/ambari-agent/ambari-agent.log file:

    INFO 2017-06-12 14:52:53,625 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way...
    INFO 2017-06-12 14:52:53,625 security.py:179 - Server certicate not exists, downloading
    INFO 2017-06-12 14:52:53,625 security.py:202 - Downloading server cert from https://localhost:8440/cert/ca/
    INFO 2017-06-12 14:52:53,693 security.py:187 - Agent key not exists, generating request
    INFO 2017-06-12 14:52:53,693 security.py:258 - openssl req -new -newkey rsa:1024 -nodes -keyout "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.key" -subj /OU=c6401.ambari.apache.org/ -out "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.csr"
    INFO 2017-06-12 14:52:53,736 security.py:195 - Agent certificate not exists, sending sign request
    INFO 2017-06-12 14:52:53,855 security.py:93 - SSL Connect being called.. connecting to the server
    INFO 2017-06-12 14:52:53,933 security.py:77 - SSL connection established. Two-way SSL authentication completed successfully.