Cloudera Docs
»
2.4.0
»
Hadoop Security Guide
Hadoop Security Guide
Also available as:
Contents
1. HDP Security Overview
Understanding Data Lake Security
HDP Security Features
Administration
Authentication and Perimeter Security
Authorization
Audit
Data Protection
2. Authentication
Enabling Kerberos Authentication Using Ambari
Kerberos Overview
Hadoop and Kerberos Principals
Installing and Configuring the KDC
Use an Existing MIT KDC
Use an Existing Active Directory
Use Manual Kerberos Setup
(Optional) Install a new MIT KDC
Enabling Kerberos Security
Installing the JCE
Install the JCE
Running the Kerberos Security Wizard
Launching the Kerberos Wizard (Automated Setup)
Launching the Kerberos Wizard (Manual Setup)
Kerberos Client Packages
Disabling Kerberos Security
Customizing the Attribute Template
Managing Admin Credentials
Configuring Ambari Authentication with LDAP or AD
Configuring Ambari for LDAP or Active Directory Authentication
Setting Up LDAP User Authentication
Configure Ambari to use LDAP Server
Example Active Directory Configuration
Synchronizing LDAP Users and Groups
Specific Set of Users and Groups
Existing Users and Groups
All Users and Groups
Configuring Ranger Authentication with UNIX, LDAP, or AD
UNIX Authentication Settings
Active Directory Authentication Settings
AD Settings
Custom ranger-admin-site Settings for Active Directory (Optional)
LDAP Authentications Settings
LDAP Settings
Custom ranger-admin-site Settings for LDAP (Optional)
Advanced ranger-admin-site Settings
Encrypting Database and LDAP Passwords in Ambari
Reset Encryption
Remove Encryption Entirely
Change the Current Master Key
Configuring LDAP Authentication in Hue
Enabling the LDAP Backend
Enabling User Authentication with Search Bind
Setting the Search Base to Find Users and Groups
Specifying the URL of the LDAP Server
Specifying LDAPS and StartTLS Support
Specifying Bind Credentials for LDAP Searches
Synchronizing Users and Groups
Setting Search Bind Authentication and Importing Users and Groups
Setting LDAP Users' Filter
Setting an LDAP Groups Filter
Setting Multiple LDAP Servers
Advanced Security Options for Ambari
Configuring Ambari for Non-Root
How to Configure Ambari Server for Non-Root
How to Configure an Ambari Agent for Non-Root
Sudoer Configuration
Customizable Users
Non-Customizable Users
Commands
Sudo Defaults
Optional: Ambari Web Inactivity Timeout
Set Up Kerberos for Ambari Server
Optional: Set Up Two-Way SSL Between Ambari Server and Ambari Agents
Optional: Configure Ciphers and Protocols for Ambari Server
Optional: HTTP Cookie Persistence
Enabling SPNEGO Authentication for Hadoop
Configure Ambari Server for Authenticated HTTP
Configuring HTTP Authentication for HDFS, YARN, MapReduce2, HBase, Oozie, Falcon and Storm
Setting Up Kerberos Authentication for Non-Ambari Clusters
Preparing Kerberos
Kerberos Overview
Installing and Configuring the KDC
Creating the Database and Setting Up the First Administrator
Creating Service Principals and Keytab Files for HDP
Configuring HDP for Kerberos
Creating Mappings Between Principals and UNIX Usernames
Examples
Adding Security Information to Configuration Files
core-site.xml
HTTP Cookie Persistence
hdfs-site.xml
yarn-site.xml
mapred-site.xml
hbase-site.xml
hive-site.xml
oozie-site.xml
webhcat-site.xml
limits.conf
Configuring HBase and ZooKeeper
Configure HBase Master
Create JAAS configuration files
Start HBase and ZooKeeper services
Configure secure client side access for HBase
Optional: Configure client-side operation for secure operation - Thrift Gateway
Optional: Configure client-side operation for secure operation - REST Gateway
Configure HBase for Access Control Lists (ACL)
Configuring Hue
Configuring Phoenix Query Server
Setting up One-Way Trust with Active Directory
Configure Kerberos Hadoop Realm on the AD DC
Configure the AD Domain on the KDC and Hadoop Cluster Hosts
Configuring Proxy Users
Perimeter Security with Apache Knox
Apache Knox Gateway Overview
Knox Gateway Deployment Architecture
Supported Hadoop Services
Knox Gateway Samples
Configuring the Knox Gateway
Create and Secure the Gateway Directories
Customize the Gateway Port and Path
Manage the Master Secret
Manually Redeploy Cluster Topologies
Manually Start and Stop Apache Knox
Defining Cluster Topologies
Configuring a Hadoop Server for Knox
Setting up Hadoop Service URLs
Example Service Definitions
Validating Service Connectivity
Adding a New Service to the Knox Gateway
Service Directory Structure
Adding a New Service to the Knox Gateway
Mapping the Internal Nodes to External URLs
Setting Up a Hostmap Provider
Example of an EC2 Hostmap Provider
Example of Sandbox Hostmap Provider
Enabling Hostmap Debugging
Configuring Authentication
Setting Up LDAP Authentication
LDAP Authentication Caching
Example Active Directory Configuration
Example OpenLDAP Configuration
Testing an LDAP Provider
Setting Up HTTP Header Authentication for Federation_SSO
Example SiteMinder Configuration
Testing HTTP Header Tokens
Setting Up 2-Way SSL Authentication
Configuring Identity Assertion
Structure of the Identity-Assertion Provider
Define Pseudo Identity Assertion
Mapping Authenticated User to Cluster
Principal Mapping Enhancements
Example User Mapping
Mapping Authenticated Users to Groups
Configuring Group Mapping
Examples of Group Mapping
Configuring Service Level Authorization
Setting Up an Authorization Provider
Examples of Authorization
Audit Gateway Activity
Audit Log Fields
Change Roll Frequency of the Audit Log
Gateway Security
Implementing Web Application Security
Configuring Protection Filter Against Cross Site Request Forgery Attacks
Validate CSRF Filtering
Configuring Knox With a Secured Hadoop Cluster
Setting Up Knox for WebHDFS HA
Configure WebHDFS for Knox
Configure Knox for WebHDFS HA
Knox CLI Testing Tools
Knox CLI LDAP Authentication and Authorization Testing
3. Configuring Authorization in Hadoop
Installing Ranger Using Ambari
Overview
Installation Prerequisites
Setting Up Hadoop Group Mapping for LDAP/AD
Configure Hadoop Group Mapping for LDAP/AD Using SSSD (Recommended)
Configure Hadoop Group Mapping in core-site.xml
Manually Create the Users and Groups in the Linux Environment
Configuring MySQL for Ranger
Configuring PostgreSQL for Ranger
Configuring Oracle for Ranger
Ranger Installation
Start the Installation
Customize Services
Ranger Admin Settings
Ranger Audit Settings
Configure Ranger User Sync
Test Run Ranger Usersync
Configuring Ranger User Sync for UNIX
Configuring Ranger User Sync for LDAP/AD
Configure Ranger Authentication
Configuring Ranger UNIX Authentication
Configuring Ranger LDAP Authentication
Configuring Ranger Active Directory Authentication
Complete the Ranger Installation
Advanced Usersync Settings
UNIX Usersync Settings
Required LDAP and AD Usersync Settings
Additional LDAP and AD Usersync Settings
Configuring Ranger for LDAP SSL
Setting up Database Users Without Sharing DBA Credentials
Updating Ranger Admin Passwords
Enabling Ranger Plugins
HDFS
Hive
HBase
Kafka
Knox
YARN
Storm
Ranger Plugins - Kerberos Overview
HDFS
Hive
HBase
Knox
Using Ranger to Provide Authorization in Hadoop
Opening and Closing the Ranger Console
Console Operations Summary
Configuring Services
Configure an HBase Service
Configure an HDFS Service
Configure a Hive Service
Configure a Kafka Service
Configure a Knox Service
Configure a Solr Service
Configure a Storm Service
Configure a YARN Service
Policy Management
Create an HBase Policy
Provide User Access to HBase Database Tables from the Command Line
Create an HDFS Policy
Create a Hive Policy
Provide User Access to Hive Database Tables from the Command Line
Create a Kafka Policy
Create a Knox Policy
Create a Solr Policy
Create a Storm Policy
Create a YARN Policy
Users/Groups and Permissions Administration
Add a User
Edit a User
Add a Group
Edit a Group
Add or Edit Permissions
Reports Administration
View Reports
Search Reports
Special Requirements for High Availability Environments
Adding a New Component to Apache Ranger
Developing a Custom Authorization Module
Apache Ranger Public REST API
Service Definition APIs
Get Service Definition by ID
Get Service Definition by Name
Create Service Definition
Update Service Definition by ID
Update Service Definition by Name
Delete Service Definition by ID
Delete Service Definition by Name
Search Service Definitions
Service APIs
Get Service by ID
Get Service by Name
Create Service
Update Service by ID
Update Service by Name
Delete Service by ID
Delete Service by Name
Search Services
Policy APIs
Get Policy by ID
Get Policy by Service Name and Policy Name
Create Policy
Update Policy by ID
Update Policy by Service Name and Policy Name
Delete Policy by ID
Delete Policy by Service Name and Policy Name
Search Policies in a Service
4. Data Protection: Wire Encryption
Enabling RPC Encryption
Enabling Data Transfer Protocol
Enabling SSL: Understanding the Hadoop SSL Keystore Factory
Creating and Managing SSL Certificates
Obtain a Certificate from a Trusted Third-Party Certification Authority (CA)
Create and Set Up an Internal CA (OpenSSL)
Installing Certificates in the Hadoop SSL Keystore Factory (HDFS, MapReduce, and YARN)
Using a CA-Signed Certificate
Enabling SSL for HDP Components
Enable SSL for WebHDFS, MapReduce Shuffle, and YARN
Enable SSL for HttpFS
Enable SSL on Oozie
Configure Oozie HCatalogJob Properties
Enable SSL on the HBase REST Server
Enable SSL on the HBase Web UI
Enable SSL on HiveServer2
Enable SSL for Kafka Clients
Configuring the Kafka Broker
Configuring Kafka Producer and Kafka Consumer
Enable SSL for Accumulo
Generate a Certificate Authority
Generate a Certificate/Keystore Per Host
Configure Accumulo Servers
Configure Accumulo Clients
SPNEGO setup for WebHCat
Configure SSL for Hue
Enabling SSL on Hue by Using a Private Key
Enabling SSL on Hue Without Using a Private Key
Configure SSL for Knox
Self-Signed Certificate with Specific Hostname for Evaluations
CA-Signed Certificates for Production
Setting Up Trust for the Knox Gateway Clients
Securing Phoenix
Set Up SSL for Ambari
Set Up Truststore for Ambari Server
Configure Ambari Ranger SSL
Configuring Ambari Ranger SSL Using Public CA Certificates
Prerequisites
Configuring Ranger Admin
Configuring Ranger Usersync
Configuring Ranger Plugins for SSL
Configuring the Ranger HDFS Plugin for SSL
Configuring the Ranger KMS Plugin for SSL
Configuring the Ranger KMS Server for SSL
Configuring Ambari Ranger SSL Using a Self-Signed Certificate
Prerequisites
Configuring Ranger Admin
Configuring Ranger Usersync
Configuring Ranger Plugins
Configuring the Ranger HDFS Plugin for SSL
Configuring the Ranger KMS Plugin for SSL
Configuring the Ranger KMS Server for SSL
Configure Non-Ambari Ranger SSL
Configuring Non-Ambari Ranger SSL Using Public CA Certificates
Configuring Ranger Admin
Configuring Ranger Usersync
Configuring Ranger Plugins
Configuring Non-Ambari Ranger SSL Using a Self Signed Certificate
Configuring Ranger Admin
Configuring Ranger Usersync
Configuring Ranger Plugins
Connecting to SSL-Enabled Components
Connect to SSL Enabled HiveServer2 using JDBC
Connect to SSL Enabled Oozie Server
Use a Self-signed Certificate from Oozie Java Clients
Connect to Oozie from Java Clients
Connect to Oozie from a Web Browser
5. Auditing in Hadoop
Using Apache Solr for Ranger Audits
Prerequisites
Installing Solr
Configuring Solr Standalone
Configuring SolrCloud
Manually Enabling Audit Settings in Ambari Clusters
Manually Updating Ambari Solr Audit Settings
Manually Updating Ambari HDFS Audit Settings
Enabling Audit Logging in Non-Ambari Clusters
Manging Auditing in Ranger
View Operation Details
Access
Admin
Login Sessions
Plugins
6. Data Protection: HDFS Encryption
Ranger KMS Administration Guide
Installing the Ranger Key Management Service
Install Ranger KMS using Ambari (Kerberized Cluster)
Setting up Database Users Without Sharing DBA Credentials
Configure HDFS Encryption to use Ranger KMS Access
Use a Kerberos Principal for the Ranger KMS Repository
Enable Ranger KMS Audit
Save Audits to Solr
Save Audits to HDFS
Enabling SSL for Ranger KMS
Install Multiple Ranger KMS
Using the Ranger Key Management Service
Accessing the Ranger KMS Web UI
Listing and Creating Keys
Rolling Over an Existing Key
Deleting a Key
Ranger KMS Properties
Troubleshooting Ranger KMS
HDFS "Data at Rest" Encryption
HDFS Encryption Overview
Configuring and Starting the Ranger Key Management Service (Ranger KMS)
Configuring and Using HDFS Data at Rest Encryption
Prepare the Environment
CPU Support for AES-NI optimization
Library Support for AES-NI optimization
Verifying AES-NI Support
Create an Encryption Key
Create an Encryption Zone
Copy Files from/to an Encryption Zone
Read and Write Files from/to an Encryption Zone
Delete Files from an Encryption Zone with Trash Enabled
Configuring HDP Services for HDFS Encryption
Hive
Configuring Hive Tables for HDFS Encryption
Loading Data into an Encrypted Table
Encrypting Other Hive Directories
Additional Changes in Behavior with HDFS-Encrypted Tables
HBase
Recommendations
Steps
Changes in Behavior after HDFS Encryption is Enabled
Sqoop
Recommendations
MapReduce on YARN
Steps
Oozie
Recommendations
WebHDFS
Recommendations
Steps
Appendix: Creating an HDFS Admin User
« Prev
Next »
Update Service Definition by Name
API Name
Update Service Definition
Request Type
PUT
Request URL
service/public/v2/api/servicedef/{name}
Request Params
Application/json
• Example:
Response
200-Application/json
© 2012–2021 by Cloudera, Inc.
Document licensed under the
Creative Commons Attribution ShareAlike 4.0 License
.
Cloudera.com
|
Documentation
|
Support
|
Community