Cloudera Docs
»
2.6.5
»
Security
Security
Also available as:
Contents
1. HDP Security Overview
What's New in This Release
Understanding Data Lake Security
HDP Security Features
Administration
Authentication and Perimeter Security
Authorization
Audit
Data Protection
2. Authentication
Enabling Kerberos Authentication Using Ambari
Kerberos Overview
Kerberos Principals
Installing and Configuring the KDC
Use an Existing MIT KDC
Use an Existing Active Directory
Use Manual Kerberos Setup
(Optional) Install a new MIT KDC
Enabling Kerberos Security
Installing the JCE
Install the JCE
Running the Kerberos Security Wizard
Launching the Kerberos Wizard (Automated Setup)
Launching the Kerberos Wizard (Manual Setup)
Kerberos Client Packages
Disabling Kerberos Security
Customizing the Attribute Template
Managing Admin Credentials
Configuring HDP Components for Kerberos Using Ambari
Configuring Kafka for Kerberos Using Ambari
Preparing the Cluster
Configuring the Kafka Broker for Kerberos
Creating Kafka Topics
Producing Events/Messages to Kafka on a Secured Cluster
Consuming Events/Messages from Kafka on a Secured Cluster
Authorizing Access when Kerberos is Enabled
Kafka Authorization Command Line Interface
Authorization Examples
Grant Read/Write Access to a Topic
Grant Full Access to Topic, Cluster, and Consumer Group
Add a Principal as Producer or Consumer
Deny Access to a Principal
Remove Access
List ACLs
Configure Authorizer Settings
Troubleshooting Authorizer Settings
Appendix: Kafka Configuration Options
Server.properties key-value pairs
JAAS Configuration File for the Kafka Server
Configuration Setting for the Kafka Producer
JAAS Configuration File for the Kafka Client
Configuring Storm for Kerberos Using Ambari
Prerequisites
Designating a Storm Client Node
Dedicate or Use an Existing Gateway Node
Use an Existing Storm Node
Running Storm Commands
Running Workers as Users
Accessing the Storm UI
Accessing the Storm UI (Active Directory Trust Configuration)
Storm Security Properties
Known Issues
Configuring Ambari Authentication with LDAP or AD
Configuring Ambari for LDAP or Active Directory Authentication
Setting Up LDAP User Authentication
Configure Ambari to use LDAP Server
Example Active Directory Configuration
Synchronizing LDAP Users and Groups
Specific Set of Users and Groups
Existing Users and Groups
All Users and Groups
Configuring Ranger Authentication with UNIX, LDAP, or AD
UNIX Authentication Settings
Active Directory Authentication Settings
AD Settings
Custom ranger-admin-site Settings for Active Directory (Optional)
LDAP Authentications Settings
LDAP Settings
Custom ranger-admin-site Settings for LDAP (Optional)
Advanced ranger-admin-site Settings
Encrypting Database and LDAP Passwords in Ambari
Reset Encryption
Remove Encryption Entirely
Change the Current Master Key
Configuring LDAP Authentication in Hue
Enabling the LDAP Backend
Enabling User Authentication with Search Bind
Setting the Search Base to Find Users and Groups
Specifying the URL of the LDAP Server
Specifying LDAPS and StartTLS Support
Specifying Bind Credentials for LDAP Searches
Synchronizing Users and Groups
Setting Search Bind Authentication and Importing Users and Groups
Setting LDAP Users' Filter
Setting an LDAP Groups Filter
Setting Multiple LDAP Servers
Advanced Security Options for Ambari
Configuring Ambari for Non-Root
How to Configure Ambari Server for Non-Root
Sudoer Configuration - Ambari Server
Commands - Ambari Server
Sudo Defaults - Ambari Server
How to Configure an Ambari Agent for Non-Root
Sudoer Configuration - Ambari Agents
Customizable Users - Ambari Agents
Commands - Ambari Agents
Sudo Defaults - Ambari Agents
Optional: Ambari Web Inactivity Timeout
Optional: Set Up Kerberos for Ambari Server
Optional: Set Up Two-Way SSL Between Ambari Server and Ambari Agents
Optional: Configure Ciphers and Protocols for Ambari Server
Optional: HTTP Cookie Persistence
Enabling SPNEGO Authentication for Hadoop
Configure Ambari Server for Authenticated HTTP
Configuring HTTP Authentication for HDFS, YARN, MapReduce2, HBase, Oozie, Falcon and Storm
Enabling Browser Access to a SPNEGO-enabled Web UI
Setting Up Kerberos Authentication for Non-Ambari Clusters
Preparing Kerberos
Kerberos Overview
Installing and Configuring the KDC
Creating the Database and Setting Up the First Administrator
Creating Service Principals and Keytab Files for HDP
Configuring HDP for Kerberos
Creating Mappings Between Principals and UNIX Usernames
Examples
Adding Security Information to Configuration Files
core-site.xml
HTTP Cookie Persistence
hdfs-site.xml
yarn-site.xml
mapred-site.xml
hbase-site.xml
hive-site.xml
oozie-site.xml
webhcat-site.xml
limits.conf
Configuring HBase and ZooKeeper
Configure HBase Master
Create JAAS configuration files
Start HBase and ZooKeeper services
Configure secure client side access for HBase
Optional: Configure client-side operation for secure operation - Thrift Gateway
Optional: Configure client-side operation for secure operation - REST Gateway
Configure HBase for Access Control Lists (ACL)
Configuring Phoenix Query Server
Configuring Hue
Setting up One-Way Trust with Active Directory
Configure Kerberos Hadoop Realm on the AD DC
Configure the AD Domain on the KDC and Hadoop Cluster Hosts
Configuring Proxy Users
Perimeter Security with Apache Knox
Apache Knox Gateway Overview
Knox Gateway Deployment Architecture
Supported Hadoop Services
Knox Gateway Samples
Configuring the Knox Gateway
Create and Secure the Gateway Directories
Manage the Master Secret
Manually Redeploy Cluster Topologies
Manually Start and Stop Apache Knox
Enable WebSockets
Defining Cluster Topologies
Configuring a Hadoop Server for Knox
Setting up Hadoop Service URLs
Example Service Definitions
Validating Service Connectivity
Adding a New Service to the Knox Gateway
Service Directory Structure
Adding a New Service to the Knox Gateway
Mapping the Internal Nodes to External URLs
Setting Up a Hostmap Provider
Example of an EC2 Hostmap Provider
Example of Sandbox Hostmap Provider
Enabling Hostmap Debugging
Configuring Authentication
Authentication Providers
Setting Up LDAP Authentication
Configuring Advanced LDAP Authentication
Using Advanced LDAP Authentication
Advanced LDAP Configuration Parameters
Advanced LDAP Configuration Combinations
Advanced LDAP Authentication Errata
Problem with userDnTemplate-Based Authentication
Special Note on Parameter main.ldapRealm.contextFactory.systemPassword
Setting Up SPNEGO Authentication
Setting up PAM Authentication
LDAP Authentication Caching
Example Active Directory Configuration
Example OpenLDAP Configuration
Testing an LDAP Provider
Setting Up HeaderPreAuth Federation Provider
Setting up JWT Federation Provider
Setting up Pac4j Federation Provider
Setting up SSOCookieProvider Federation Provider
Example SiteMinder Configuration
Testing HTTP Header Tokens
Setting Up 2-Way SSL Authentication
Configuring Identity Assertion
Identity Assertion Providers Overview
Default Identity Assertion Provider
Mapping Authenticated Users to Other Users
Mapping Authenticated Users to Groups
Concat Identity Assertion Provider
Hadoop Group Lookup Identity Assertion Provider
Using GroupMappingServiceProvider to Configure Group Mapping
Regular Expression Identity Assertion Provider
SwitchCase Identity Assertion Provider
Configuring Group Mapping
Configuring Service Level Authorization
Setting Up an Authorization Provider
Examples of Authorization
Audit Gateway Activity
Audit Log Fields
Change Roll Frequency of the Audit Log
Configuring Storm Plugin Audit Log to File
Gateway Security
Implementing Web Application Security
Configuring Protection Filter Against Cross Site Request Forgery Attacks
Validate CSRF Filtering
Configuring Knox With a Secured Hadoop Cluster
Setting Up Knox Services for HA
Prerequisites
Configure WebHDFS for Knox
Configure Knox for HA
Knox CLI Testing Tools
Knox CLI LDAP Authentication and Authorization Testing
Knox SSO
Identity Providers (IdP)
Form-based Identity Provider (IdP)
SAML-based Identity Provider (IdP)
Setting up Knox SSO for Ambari
Setting up Knox SSO for Ranger Web UI
Setting up the Knox Token Service for Ranger APIs
Setting up Knox SSO for Apache Atlas
3. Configuring Authorization in Hadoop
Installing Ranger Using Ambari
Overview
Installation Prerequisites
Setting Up Hadoop Group Mapping for LDAP/AD
Configure Hadoop Group Mapping for LDAP/AD Using SSSD (Recommended)
Configure Hadoop Group Mapping in core-site.xml
Manually Create the Users and Groups in the Linux Environment
Configuring a Database Instance for Ranger
Configuring MySQL for Ranger
Configuring PostgreSQL for Ranger
Configuring Oracle for Ranger
Amazon RDS Requirements
MySQL/MariaDB Prerequisite
PostgreSQL Prerequisite
Oracle Prerequisite
Ranger Installation
Start the Installation
Customize Services
Ranger Admin Settings
Ranger Audit Settings
Configure Ranger User Sync
Test Run Ranger Usersync
Configuring Ranger User Sync for UNIX
Configuring Ranger User Sync for LDAP/AD
Automatically Assign ADMIN/KEYADMIN Role for External Users
Configure Ranger Tagsync
Configure Ranger Authentication
Configuring Ranger UNIX Authentication
Configuring Ranger LDAP Authentication
Configuring Ranger Active Directory Authentication
Complete the Ranger Installation
Advanced Usersync Settings
UNIX Usersync Settings
Required LDAP and AD Usersync Settings
Additional LDAP and AD Usersync Settings
Configuring Ranger for LDAP SSL
Setting up Database Users Without Sharing DBA Credentials
Updating Ranger Admin Passwords
Enabling Ranger Plugins
HDFS
Hive
HBase
Kafka
Knox
YARN
Storm
Atlas
Ranger Plugins - Kerberos Overview
HDFS
Hive
HBase
Knox
Using Ranger to Provide Authorization in Hadoop
About Ranger Policies
Ranger Resource-Based Policies
Ranger Tag-Based Policies
Tag Store
TagSync
Tags
Tags and Policy Evaluation
Finding Tags
Evaluating Tag-Based Policies
Using Tags in Conditions
Apache Ranger Access Conditions
Allow, Deny, and Exclude Conditions
Enable Deny Conditions for Policies
Policy Evaluation of Access Conditions
Using the Ranger Console
Opening and Closing the Ranger Console
Ranger Console Navigation
Configuring Resource-Based Services
Configure an HBase Service
Configure an HDFS Service
Configure a Hive Service
Configure a Kafka Service
Configure a Knox Service
Configure a Solr Service
Configure a Storm Service
Configure a YARN Service
Configure an Atlas Service
Resource-Based Policy Management
Configuring Resource-Based Policies
Create an HBase Policy
Provide User Access to HBase Database Tables from the Command Line
Create an HDFS Policy
Create a Hive Policy
Provide User Access to Hive Database Tables from the Command Line
Create a Kafka Policy
Create a Knox Policy
Create a Solr Policy
Create a Storm Policy
Create a YARN Policy
Create an Atlas Policy
Wildcard and Variable Reference Information
Wildcard Characters
{USER} Variable
{USER} Variable Recommended Practices and Customizability
Importing and Exporting Resource-Based Policies
Import Resource-Based Policies
Import Resource-Based Policies for a Specific Service
Import Resource-Based Policies for All Services
Export Resource-Based Policies
Export Resource-Based Policies for a Specific Service
Export All Resource-Based Policies for All Services
Row-level Filtering and Column Masking in Hive
Row-level Filtering in Hive with Ranger Policies
Dynamic Resource-Based Column Masking in Hive with Ranger Policies
Dynamic Tag-Based Column Masking in Hive with Ranger Policies
Adding Tag-based Service
Tag-Based Policy Management
Adding Tag-Based Policies
Using Tag Attributes and Values in Ranger Tag-based Policy Conditions
Adding a Tag-based PII Policy
Default EXPIRES_ON Policy
Importing and Exporting Tag-Based Policies
Import Tag-Based Policies
Export Tag-Based Policies
Users/Groups and Permissions Administration
Add a User
Edit a User
Delete a User
Add a Group
Edit a Group
Delete a Group
Add or Edit Permissions
Reports Administration
View Reports
Search Reports
Export Reports
Edit Policies from the Reports Page
Special Requirements for High Availability Environments
Adding a New Component to Apache Ranger
Developing a Custom Authorization Module
Apache Ranger Public REST API
Service Definition APIs
Get Service Definition by ID
Get Service Definition by Name
Create Service Definition
Update Service Definition by ID
Update Service Definition by Name
Delete Service Definition by ID
Delete Service Definition by Name
Search Service Definitions
Service APIs
Get Service by ID
Get Service by Name
Create Service
Update Service by ID
Update Service by Name
Delete Service by ID
Delete Service by Name
Search Services
Policy APIs
Get Policy by ID
Get Policy by Service Name and Policy Name
Create Policy
Update Policy by ID
Update Policy by Service Name and Policy Name
Delete Policy by ID
Delete Policy by Service Name and Policy Name
Search Policies in a Service
4. Data Protection: Wire Encryption
Enabling RPC Encryption
Enabling Data Transfer Protocol
Enabling SSL: Understanding the Hadoop SSL Keystore Factory
Creating and Managing SSL Certificates
Obtain a Certificate from a Trusted Third-Party Certification Authority (CA)
Create and Set Up an Internal CA (OpenSSL)
Installing Certificates in the Hadoop SSL Keystore Factory (HDFS, MapReduce, and YARN)
Using a CA-Signed Certificate
Enabling SSL for HDP Components
Enable SSL for WebHDFS, MapReduce Shuffle, Tez, and YARN
Enable SSL for HttpFS
Enable SSL on Oozie
Configure the Oozie Client to Connect Using SSL
Connect to the Oozie Web UI Using SSL
Configure Oozie HCatalogJob Properties
Enable SSL on the HBase REST Server
Enable SSL on the HBase Web UI
Enable SSL on HiveServer2
Setting up SSL with self-signed certificates
Selectively disabling SSL protocol versions
Enable SSL for Kafka Clients
Configuring the Kafka Broker
Configuring Kafka Producer and Kafka Consumer
Enable SSL for Accumulo
Generate a Certificate Authority
Generate a Certificate/Keystore Per Host
Configure Accumulo Servers
Configure Accumulo Clients
Enable SSL for Apache Atlas
Configuring Apache Atlas SSL
Credential Provider Utility Script
SPNEGO setup for WebHCat
Configure SSL for Hue
Enabling SSL on Hue by Using a Private Key
Enabling SSL on Hue Without Using a Private Key
Configure SSL for Knox
Self-Signed Certificate with Specific Hostname for Evaluations
CA-Signed Certificates for Production
Setting Up Trust for the Knox Gateway Clients
Securing Phoenix
Set Up SSL for Ambari
Set Up Truststore for Ambari Server
Configure Ambari Ranger SSL
Configuring Ambari Ranger SSL Using Public CA Certificates
Prerequisites
Configuring Ranger Admin
Configuring Ranger Usersync
Configuring Ranger Plugins for SSL
Configuring the Ranger HDFS Plugin for SSL
Configuring the Ranger KMS Plugin for SSL
Configuring the Ranger KMS Server for SSL
Configure Ranger KMS Database for SSL-enabled MySQL
Configure Ranger HBase Plugin for SSL
Configuring Ambari Ranger SSL Using a Self-Signed Certificate
Prerequisites
Configuring Ranger Admin
Configuring Ranger Usersync
Configuring Ranger Plugins
Configuring the Ranger HDFS Plugin for SSL
Configuring the Ranger KMS Plugin for SSL
Configuring the Ranger KMS Server for SSL
Configure Ranger Admin Database for SSL-Enabled MySQL
Configure Non-Ambari Ranger SSL
Configuring Non-Ambari Ranger SSL Using Public CA Certificates
Configuring Ranger Admin
Configuring Ranger Usersync
Configuring Ranger Plugins
Configuring Non-Ambari Ranger SSL Using a Self Signed Certificate
Configuring Ranger Admin
Configuring Ranger Usersync
Configuring Ranger Plugins
Connecting to SSL-Enabled Components
Connect to SSL Enabled HiveServer2 using JDBC
Connect to SSL Enabled Oozie Server
Use a Self-signed Certificate from Oozie Java Clients
Connect to Oozie from Java Clients
Connect to Oozie from a Web Browser
5. Auditing in Hadoop
Using Apache Solr for Ranger Audits
Prerequisites
Installing Externally Managed SolrCloud
Installation and Configuration Steps
Solr Installation
Configuring Externally Managed SolrCloud
Configuring Externally Managed Solr Standalone
Configuring SolrCloud for Kerberos
Configure Kerberos for SolrCloud
Configure SolrCloud for Kerberos
Connecting to Kerberos-enabled SolrCloud
Migrating Audit Logs from DB to Solr in Ambari Clusters
Manually Enabling Audit Settings in Ambari Clusters
Manually Updating Ambari Solr Audit Settings
Manually Updating HDFS Audit Settings (for Ambari installs)
Enabling Audit Logging in Non-Ambari Clusters
Managing Auditing in Ranger
View Operation Details
Differentiate Events from Multiple Clusters
Access
Admin
Login Sessions
Plugins
Plugin Status
6. ACLs on HDFS
Configuring ACLs on HDFS
Using CLI Commands to Create and List ACLs
ACL Examples
ACLS on HDFS Features
Use Cases for ACLs on HDFS
7. Data Protection: HDFS Encryption
Ranger KMS Administration
Installing the Ranger Key Management Service
Install Ranger KMS using Ambari (Kerberized Cluster)
Setting up Database Users Without Sharing DBA Credentials
Configure HDFS Encryption to use Ranger KMS Access
Use a Kerberos Principal for the Ranger KMS Repository
Store Master Key in a Hardware Security Module (HSM)
Install Ranger KMS Hardware Security Module (HSM)
Install Ranger KMS HSM Manually
Install Ranger KMS HSM via Ambari with plain text password
Install Ranger KMS HSM via Ambari with JCEKS
Configure HSM High Availability (HA)
HSM Migration
Migrate HSM to Ranger DB
Migrate Ranger DB to HSM
Optional: Clear Objects from the HSM Partition
Enable Ranger KMS Audit
Save Audits to Solr
Save Audits to HDFS
Enabling SSL for Ranger KMS
Install Multiple Ranger KMS
Using the Ranger Key Management Service
Accessing the Ranger KMS Web UI
Listing and Creating Keys
Rolling Over an Existing Key
Deleting a Key
Ranger KMS Properties
Troubleshooting Ranger KMS
HDFS "Data at Rest" Encryption
HDFS Encryption Overview
Configuring and Starting the Ranger Key Management Service (Ranger KMS)
Configuring and Using HDFS Data at Rest Encryption
Prepare the Environment
CPU Support for AES-NI optimization
Library Support for AES-NI optimization
Verifying AES-NI Support
Create an Encryption Key
Create an Encryption Zone
Copy Files from/to an Encryption Zone
Read and Write Files from/to an Encryption Zone
Delete Files from an Encryption Zone with Trash Enabled
Configuring HDP Services for HDFS Encryption
HBase
Recommendations
Steps
Changes in Behavior after HDFS Encryption is Enabled
Hive
Configuring Hive Tables for HDFS Encryption
Loading Data into an Encrypted Table
Encrypting Other Hive Directories
Additional Changes in Behavior with HDFS-Encrypted Tables
MapReduce on YARN
Steps
Oozie
Recommendations
Sqoop
Recommendations
WebHDFS
Recommendations
Steps
Appendix: Creating an HDFS Admin User
8. Running DataNodes as Non-Root
Introduction
Configuring DataNode SASL
9. Addendum
ZooKeeper ACLs Best Practices
Accumulo
Ambari Solr
Atlas
HBase
HDFS/WebHDFS
Hive/HCatalog
Kafka
Oozie
Ranger
Ranger KMS/Hadoop KMS
Slider
Storm
WebHCat
YARN
YARN Registry
ZooKeeper
Ranger AD Integration
Ranger Architecture
Ranger Audit
Ranger AD Integration
Ranger UI Authentication
Ranger UI Authorization
Ranger Usersync
Configuration
Ranger User Management
Issue Ranger Group Mapping
« Prev
Next »
Setting up JWT Federation Provider
For information on the JWT federation provider, see the
Apache Knox documentation
.
© 2012–2021 by Cloudera, Inc.
Document licensed under the
Creative Commons Attribution ShareAlike 4.0 License
.
Cloudera.com
|
Documentation
|
Support
|
Community