Configuring a Hadoop Server for Knox
The Apache Knox Gateway redirects external requests to an internal Hadoop service using service name and URL of the service definition.
This chapter describes:
- 1. HDP Security Overview
- 2. Authentication
- Enabling Kerberos Authentication Using Ambari
- Configuring HDP Components for Kerberos Using Ambari
- Configuring Kafka for Kerberos Using Ambari
- Preparing the Cluster
- Configuring the Kafka Broker for Kerberos
- Creating Kafka Topics
- Producing Events/Messages to Kafka on a Secured Cluster
- Consuming Events/Messages from Kafka on a Secured Cluster
- Authorizing Access when Kerberos is Enabled
- Appendix: Kafka Configuration Options
- Configuring Storm for Kerberos Using Ambari
- Configuring Kafka for Kerberos Using Ambari
- Configuring Ambari Authentication with LDAP or AD
- Configuring LDAP Authentication in Hue
- Enabling the LDAP Backend
- Enabling User Authentication with Search Bind
- Setting the Search Base to Find Users and Groups
- Specifying the URL of the LDAP Server
- Specifying LDAPS and StartTLS Support
- Specifying Bind Credentials for LDAP Searches
- Synchronizing Users and Groups
- Setting Search Bind Authentication and Importing Users and Groups
- Setting LDAP Users' Filter
- Setting an LDAP Groups Filter
- Setting Multiple LDAP Servers
- Advanced Security Options for Ambari
- Enabling SPNEGO Authentication for Hadoop
- Setting Up Kerberos Authentication for Non-Ambari Clusters
- Preparing Kerberos
- Configuring HDP for Kerberos
- Creating Mappings Between Principals and UNIX Usernames
- Adding Security Information to Configuration Files
- Configuring HBase and ZooKeeper
- Configure HBase Master
- Create JAAS configuration files
- Start HBase and ZooKeeper services
- Configure secure client side access for HBase
- Optional: Configure client-side operation for secure operation - Thrift Gateway
- Optional: Configure client-side operation for secure operation - REST Gateway
- Configure HBase for Access Control Lists (ACL)
- Configuring Phoenix Query Server
- Configuring Hue
- Setting up One-Way Trust with Active Directory
- Configuring Proxy Users
- Perimeter Security with Apache Knox
- Apache Knox Gateway Overview
- Configuring the Knox Gateway
- Defining Cluster Topologies
- Configuring a Hadoop Server for Knox
- Mapping the Internal Nodes to External URLs
- Configuring Authentication
- Authentication Providers
- Setting Up LDAP Authentication
- Configuring Advanced LDAP Authentication
- Setting Up SPNEGO Authentication
- LDAP Authentication Caching
- Example Active Directory Configuration
- Example OpenLDAP Configuration
- Testing an LDAP Provider
- Setting Up HTTP Header Authentication for Federation_SSO
- Example SiteMinder Configuration
- Testing HTTP Header Tokens
- Setting Up 2-Way SSL Authentication
- Configuring Identity Assertion
- Configuring Service Level Authorization
- Audit Gateway Activity
- Gateway Security
- Setting Up Knox Services for HA
- Knox CLI Testing Tools
- Knox SSO
- 3. Configuring Authorization in Hadoop
- Installing Ranger Using Ambari
- Overview
- Installation Prerequisites
- Ranger Installation
- Start the Installation
- Customize Services
- Complete the Ranger Installation
- Advanced Usersync Settings
- Configuring Ranger for LDAP SSL
- Setting up Database Users Without Sharing DBA Credentials
- Updating Ranger Admin Passwords
- Enabling Ranger Plugins
- Ranger Plugins - Kerberos Overview
- Using Ranger to Provide Authorization in Hadoop
- About Ranger Policies
- Using the Ranger Console
- Configuring Resource-Based Services
- Resource-Based Policy Management
- Create an HBase Policy
- Provide User Access to HBase Database Tables from the Command Line
- Create an HDFS Policy
- Create a Hive Policy
- Provide User Access to Hive Database Tables from the Command Line
- Create a Kafka Policy
- Create a Knox Policy
- Create a Solr Policy
- Create a Storm Policy
- Create a YARN Policy
- Create an Atlas Policy
- Row-level Filtering and Column Masking in Hive
- Adding Tag-based Service
- Adding Tag-based Policies
- Users/Groups and Permissions Administration
- Reports Administration
- Special Requirements for High Availability Environments
- Adding a New Component to Apache Ranger
- Developing a Custom Authorization Module
- Apache Ranger Public REST API
- Installing Ranger Using Ambari
- 4. Data Protection: Wire Encryption
- Enabling RPC Encryption
- Enabling Data Transfer Protocol
- Enabling SSL: Understanding the Hadoop SSL Keystore Factory
- Creating and Managing SSL Certificates
- Enabling SSL for HDP Components
- Enable SSL for WebHDFS, MapReduce Shuffle, Tez, and YARN
- Enable SSL for HttpFS
- Enable SSL on Oozie
- Enable SSL on the HBase REST Server
- Enable SSL on the HBase Web UI
- Enable SSL on HiveServer2
- Enable SSL for Kafka Clients
- Enable SSL for Accumulo
- Enable SSL for Apache Atlas
- SPNEGO setup for WebHCat
- Configure SSL for Hue
- Configure SSL for Knox
- Securing Phoenix
- Set Up SSL for Ambari
- Configure Ambari Ranger SSL
- Configure Non-Ambari Ranger SSL
- Connecting to SSL-Enabled Components
- 5. Auditing in Hadoop
- Using Apache Solr for Ranger Audits
- Migrating Audit Logs from DB to Solr in Ambari Clusters
- Manually Enabling Audit Settings in Ambari Clusters
- Enabling Audit Logging in Non-Ambari Clusters
- Managing Auditing in Ranger
- 6. Data Protection: HDFS Encryption
- Ranger KMS Administration
- Installing the Ranger Key Management Service
- Store Master Key in a Hardware Security Module (HSM)
- Enable Ranger KMS Audit
- Enabling SSL for Ranger KMS
- Install Multiple Ranger KMS
- Using the Ranger Key Management Service
- Ranger KMS Properties
- Troubleshooting Ranger KMS
- HDFS "Data at Rest" Encryption
- HDFS Encryption Overview
- Configuring and Starting the Ranger Key Management Service (Ranger KMS)
- Configuring and Using HDFS Data at Rest Encryption
- Configuring HDP Services for HDFS Encryption
- Appendix: Creating an HDFS Admin User
- Ranger KMS Administration
- 7. Addendum