Hadoop Security Guide
Also available as:
loading table of contents...


1. HDP Security Overview
Understanding Data Lake Security
HDP Security Features
Authentication and Perimeter Security
Data Protection
2. Authentication
Enabling Kerberos Authentication Using Ambari
Kerberos Overview
Hadoop and Kerberos Principals
Installing and Configuring the KDC
Enabling Kerberos Security
Kerberos Client Packages
Disabling Kerberos Security
Customizing the Attribute Template
Managing Admin Credentials
Configuring Ambari Authentication with LDAP or AD
Configuring Ambari for LDAP or Active Directory Authentication
Configuring Ranger Authentication with UNIX, LDAP, or AD
Encrypting Database and LDAP Passwords in Ambari
Advanced Security Options for Ambari
Configuring Ambari for Non-Root
Optional: Ambari Web Inactivity Timeout
Set Up Kerberos for Ambari Server
Optional: Set Up Two-Way SSL Between Ambari Server and Ambari Agents
Optional: Configure Ciphers and Protocols for Ambari Server
Optional: HTTP Cookie Persistence
Enabling SPNEGO Authentication for Hadoop
Configure Ambari Server for Authenticated HTTP
Configuring HTTP Authentication for HDFS, YARN, MapReduce2, HBase, Oozie, Falcon and Storm
Setting Up Kerberos Authentication for Non-Ambari Clusters
Preparing Kerberos
Configuring HDP for Kerberos
Setting up One-Way Trust with Active Directory
Configuring Proxy Users
Perimeter Security with Apache Knox
Apache Knox Gateway Overview
Configuring the Knox Gateway
Defining Cluster Topologies
Configuring a Hadoop Server for Knox
Mapping the Internal Nodes to External URLs
Configuring Authentication
Configuring Identity Assertion
Configuring Service Level Authorization
Audit Gateway Activity
Gateway Security
Setting Up Knox for WebHDFS HA
Knox CLI Testing Tools
3. Configuring Authorization in Hadoop
Installing Ranger Using Ambari
Installation Prerequisites
Ranger Installation
Enabling Ranger Plugins
Ranger Plugins - Kerberos Overview
Using Ranger to Provide Authorization in Hadoop
Opening and Closing the Ranger Console
Console Operations Summary
Configuring Services
Policy Management
Users/Groups and Permissions Administration
Reports Administration
Special Requirements for High Availability Environments
Adding a New Component to Apache Ranger
Developing a Custom Authorization Module
Apache Ranger Public REST API
4. Data Protection: Wire Encryption
Enabling RPC Encryption
Enabling Data Transfer Protocol
Enabling SSL: Understanding the Hadoop SSL Keystore Factory
Creating and Managing SSL Certificates
Obtain a Certificate from a Trusted Third-Party Certification Authority (CA)
Create and Set Up an Internal CA (OpenSSL)
Installing Certificates in the Hadoop SSL Keystore Factory (HDFS, MapReduce, and YARN)
Using a CA-Signed Certificate
Enabling SSL for HDP Components
Enable SSL for WebHDFS, MapReduce Shuffle, and YARN
Enable SSL for HttpFS
Enable SSL on Oozie
Configure Oozie HCatalogJob Properties
Enable SSL on the HBase REST Server
Enable SSL on the HBase Web UI
Enable SSL on HiveServer2
Setting up SSL with self-signed certificates
Selectively disabling SSL protocol versions
Enable SSL for Kafka Clients
Configuring the Kafka Broker
Configuring Kafka Producer and Kafka Consumer
Enable SSL for Accumulo
Generate a Certificate Authority
Generate a Certificate/Keystore Per Host
Configure Accumulo Servers
Configure Accumulo Clients
SPNEGO setup for WebHCat
Configure SSL for Hue
Configure SSL for Knox
Self-Signed Certificate with Specific Hostname for Evaluations
CA-Signed Certificates for Production
Setting Up Trust for the Knox Gateway Clients
Securing Phoenix
Set Up SSL for Ambari
Set Up Truststore for Ambari Server
Configure Ambari Ranger SSL
Configuring Ambari Ranger SSL Using Public CA Certificates
Configuring Ambari Ranger SSL Using a Self-Signed Certificate
Configure Non-Ambari Ranger SSL
Configuring Non-Ambari Ranger SSL Using Public CA Certificates
Configuring Non-Ambari Ranger SSL Using a Self Signed Certificate
Connecting to SSL-Enabled Components
Connect to SSL Enabled HiveServer2 using JDBC
Connect to SSL Enabled Oozie Server
5. Auditing in Hadoop
Using Apache Solr for Ranger Audits
Installing Solr
Configuring Solr Standalone
Configuring SolrCloud
Manually Enabling Audit Settings in Ambari Clusters
Manually Updating Ambari Solr Audit Settings
Manually Updating Ambari HDFS Audit Settings
Enabling Audit Logging in Non-Ambari Clusters
Manging Auditing in Ranger
View Operation Details
Login Sessions
6. Data Protection: HDFS Encryption
Ranger KMS Administration Guide
Installing the Ranger Key Management Service
Enable Ranger KMS Audit
Enabling SSL for Ranger KMS
Install Multiple Ranger KMS
Using the Ranger Key Management Service
Ranger KMS Properties
Troubleshooting Ranger KMS
HDFS "Data at Rest" Encryption
HDFS Encryption Overview
Configuring and Starting the Ranger Key Management Service (Ranger KMS)
Configuring and Using HDFS Data at Rest Encryption
Configuring HDP Services for HDFS Encryption
Appendix: Creating an HDFS Admin User

List of Tables

2.1. UNIX Authentication Settings
2.2. Active Directory Authentication Settings
2.3. Active Directory Custom ranger-admin-site Settings
2.4. LDAP Authentication Settings
2.5. LDAP Custom ranger-admin-site Settings
2.6. Active Directory Authentication Settings
2.7. Service Principals
2.8. Service Keytab File Names
2.9. General core-site.xml, Knox, and Hue
2.10. core-site.xml Master Node Settings -- Knox Gateway
2.11. core-site.xml Master Node Settings -- Hue
2.12. hdfs-site.xml File Property Settings
2.13. yarn-site.xml Property Settings
2.14. mapred-site.xml Property Settings
2.15. hbase-site.xml Property Settings for HBase Server
2.16. hive-site.xml Property Settings
2.17. oozie-site.xml Property Settings
2.18. webhcat-site.xml Property Settings
2.19. Supported Hadoop Services
2.20. Apache Service Gateway Directores
2.21. Cluster Topology Provider and Service Roles
2.22. gateway-site.xml Configuration Elements
2.23. LDAP Authentication and Authorization Arguments
3.1. Ranger DB Host
3.2. Driver Class Name
3.3. Ranger DB Username Settings
3.4. JDBC Connect String
3.5. DBA Credential Settings
3.6. UNIX User Sync Properties
3.7. LDAP/AD Common Configs
3.8. LDAP/AD User Configs
3.9. LDAP/AD Group Configs
3.10. UNIX Authentication Settings
3.11. LDAP Authentication Settings
3.12. AD Settings
3.13. LDAP Advanced ranger-ugsync-site Settings
3.14. AD Advanced ranger-ugsync-site Settings
3.15. Advanced ranger-ugsync-site Settings for LDAP and AD
3.16. HDFS Plugin Properties
3.17. Hive Plugin Properties
3.18. HBase Plugin Properties
3.19. Knox Plugin Properties
3.20. Knox Configuration Properties
3.21. Service Details
3.22. Config Properties
3.23. Service Details
3.24. Config Properties
3.25. Service Details
3.26. Config Properties
3.27. Service Details
3.28. Config Properties
3.29. Service Details
3.30. Config Properties
3.31. Service Details
3.32. Config Properties
3.33. Service Details
3.34. Config Properties
3.35. Service Details
3.36. Config Properties
3.37. Policy Details
3.38. User and Group Permissions
3.39. Policy Details
3.40. User and Group Permissions
3.41. Policy Details
3.42. User and Group Permissions
3.43. Policy Details
3.44. User and Group Permissions
3.45. Policy Details
3.46. User and Group Permissions
3.47. Policy Details
3.48. User and Group Permissions
3.49. Policy Details
3.50. User and Group Permissions
3.51. Knox User and Group Permissions
3.52. Policy Details
3.53. User and Group Permissions
4.1. Components that Support SSL
4.2. Configure SSL Data Protection for HDP Components
4.3. Configuration Properties in ssl-server.xml
5.1. Solr install.properties Values
5.2. Solr install.properties Values
5.3. Search Criteria
5.4. Search Criteria
5.5. Search Criteria
5.6. Agents Search Criteria
6.1. Properties in Advanced dbks-site Menu (dbks-site.xml)
6.2. Properties in Advanced kms-env
6.3. Properties in Advanced kms-properties (install.properties)
6.4. Properties in Advanced kms-site (kms-site.xml)
6.5. Properties in Advanced ranger-kms-audit (ranger-kms-audit.xml)
6.6. Properties in Advanced ranger-kms-policymgr-ssl
6.7. Properties in Advanced ranger-kms-security
6.8. Troubleshooting Suggestions