Configuring Knox for NiFi
Creating Knox Certificates using the TLS Toolkit
About This Task
You must create Knox certificates so that Knox and NiFi can use two-way SSL communication to achieve a secure proxy relationship.
Steps
As the Knox user, create a
nifi-ca-config.json
file on each Knox node, in a location accessible to Knox.For example, to create the file on a Knox node at
/home/knox
using the vi text editor, enter the following:sudo su - knox vi /home/knox/nifi-ca-config.json
Populate the
nifi-ca-config.json
file with the following information:{ "dn" : "CN=$KNOX_HOSTNAME, OU=KNOX", "keyStore" : "/home/knox/knox-nifi-keystore.jks", "keyStoreType" : "jks", "keyStorePassword" : "$KEYSTORE_PASSWORD", "keyPassword" : "$KEY_PASSWORD", "token" : "$NIFI_CA_TOKEN_VALUE", "caHostname" : "$NIFI_CA_HOSTNAME", "port" : $NIFI_CA_PORT, "trustStore" : "/home/knox/knox-nifi-truststore.jks", "trustStorePassword" : "$TRUSTSTORE_PASSWORD", "trustStoreType" : "jks" }
$KNOX_HOSTNAME is the FQDN for the cluster node on which Knox is installed.
$KEYSTORE_PASSWORD is the password you want the TLS Toolkit to use for the KeyStore and TrustStore for Knox when communicating with NiFi.
$KEY_PASSWORD is required when SSL is enabled. If you are using the NiFi Certificate Authority (CA), this value is automatically generated if the field is left blank. You can find this value in the Key password field in the NiFi Advanced nifi-ambari-ssl-config configuration section.
$NIFI_CA_TOKEN_VALUE is the token that NiFi CA uses to verify a NiFi node identity before issuing certificates. You can find this value in the NiFi CA Token field in the NiFi Advanced nifi-ambari-ssl-config configuration section.
$NIFI_CA_HOSTNAME is the FQDN for the cluster node on which the NiFi CA is installed.
$NIFI_CA_PORT is the port used by the NiFi CA.
You can find this value in the NiFi Certificate Authority port field in the NiFi Advanced nifi-ambari-ssl-config configuration section.
$TRUSTSTORE_PASSWORD is the truststore password.
Note You can set keyStorePassword, keyPassword, and trustStorePassword to the Knox Master Secret to make it easier to import the keyStore and trustStore created by the NiFi Certificate Authority into the Knox keystore.
Example nifi-ca-config.json file
{ "days" : 1095, "keySize" : 2048, "keyPairAlgorithm" : "RSA", "signingAlgorithm" : "SHA256WITHRSA", "dn" : "CN=slo-hdf-test5.field.hortonworks.com, OU=KNOX", "domainAlternativeNames" : null, "keyStore" : "/home/knox/knox-nifi-keystore.jks", "keyStoreType" : "jks", "keyStorePassword" : "admin", "keyPassword" : "admin", "token" : "token", "caHostname" : "slo-hdf-test5.field.hortonworks.com", "port" : 10443, "dnPrefix" : "CN=", "dnSuffix" : ", OU=NIFI", "reorderDn" : true, "trustStore" : "/home/knox/knox-nifi-truststore.jks", "trustStorePassword" : "admin", "trustStoreType" : "jks" }
Creating Certificates for Knox
About This Task
You must create the keystore and truststore used by Knox when proxying NiFi.
Before You Begin
Ensure that Java is set correctly for your environment.
Steps
As the root user, start the NiFi TLS Toolkit:For the location of the TLS Toolkit, see the HDF Release Notes for release-specific download information.
/var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.0.0-564/bin/tls-toolkit.sh client --subjectAlternativeNames "CN=$KNOX_HOSTNAME, OU=KNOX" -F -f /home/knox/config.json
The toolkit requests a new certificate and creates two new files containing the keystore and truststore:
/home/knox/knox-nifi-keystore.jks /home/knox/knox-nifi-truststore.jks
Import the Knox certificate for NiFi into the Knox
gateway.jks
file:keytool -importkeystore -srckeystore /home/knox/knox-nifi-keystore.jks -destkeystore /usr/hdp/current/knox-server/data/security/keystores/gateway.jks -deststoretype JKS -srcstorepass $KEYSTORE_PASSWORD -deststorepass $KNOX_MASTER_PASSWORD
The
gateway.jks
file should now contain a PrivateKeyEntry for NiFi.Import the NiFi CA truststore into the Knox
gateway.jks
file:keytool -importkeystore -srckeystore /home/knox/knox-nifi-truststore.jks -destkeystore /usr/hdp/current/knox-server/data/security/keystores/gateway.jks -deststoretype JKS -srcstorepass $KEYSTORE_PASSWORD -deststorepass $KNOX_MASTER_PASSWORD
The
gateway.jks
file should now contain a trustedCertEntry for NiFi.Knox uses the
gateway.jks
file to fince secure certificates in the truststore.Verify that the proper keys are in the
gateway.jks
file:keytool -keystore /usr/hdp/current/knox-server/data/security/keystores/gateway.jks -storepass $KEYSTORE_PASSWORD -list -v
You should see nifi-key and nifi-cert added to the gateway-identity key.
More Information
Verify the Knox NiFi Service Configuration
If you are running HDP 2.6.5, the
/usr/hdp/current/knox-server/data/services/nifi/1.4.0/service.xml
file
should
contain the following content. You do not need to manually edit this file.
<dispatch classname="org.apache.hadoop.gateway.dispatch.NiFiDispatch"
ha-classname="org.apache.hadoop.gateway.dispatch.NiFiHaDispatch" use-two-way-ssl="false"/>
If you are running HDP 2.6.4 or earlier,
set
the use-two-way-ssl
attribute to true
.
In the
/usr/hdp/current/knox-server/data/services/nifi/1.4.0/service.xml
file,
make
sure the dispatch element contains the following: <dispatch
classname="org.apache.hadoop.gateway.dispatch.NiFiDispatch"
use-two-way-ssl="true"/>
Creating an Advanced Topology
Steps
As the Knox user, create
flow-management.xml
in/usr/hdp/current/knox-server/conf/topologies
.Add the following content to
flow-management.xml
:If you have modified the ShiroProvider in the topology specified in the Advanced Topology section of the Ambari Knox configs, and want to authenticate Flow Management topology users in the same manner as Advanced Topology users, ensure that those modifications are reflected in your added content.
<?xml version="1.0" encoding="utf-8"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <topology> <gateway> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>sessionTimeout</name> <value>30</value> </param> <param> <name>redirectToUrl</name> <value>/gateway/knoxsso/knoxauth/login.html</value> </param> <param> <name>restrictedCookies</name> <value>rememberme,WWW-Authenticate</value> </param> <param> <name>main.ldapRealm</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> </param> <param> <name>main.ldapContextFactory</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> </param> <param> <name>main.ldapRealm.contextFactory</name> <value>$ldapContextFactory</value> </param> <param> <name>main.ldapRealm.userDnTemplate</name> <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> </param> <param> <name>main.ldapRealm.contextFactory.url</name> <value>ldap://localhost:33389</value> </param> <param> <name>main.ldapRealm.authenticationCachingEnabled</name> <value>false</value> </param> <param> <name>main.ldapRealm.contextFactory.authenticationMechanism</name> <value>simple</value> </param> <param> <name>urls./**</name> <value>authcBasic</value> </param> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> </gateway> <service> <role>NIFI</role> <url>$NIFI_HTTP_SCHEME://$NIFI_HOST:$NIFI_HTTP_SCHEME_PORT</url> <param name="useTwoWaySsl" value="true"/> </service> </topology>
$NIFI_HTTP_SCHEME specifies whether NiFi is communicating over HTTP or HTTPS.
$NIFI_HOST is the FQDN of the host machine on which NiFi is running.
$NIFI_HTTP_SCHEME_PORT specifies the port on which NiFi is running. This value varies depending on HTTP or HTTPS usage. In Advanced nifi-ambari-ssl-config, if Enable SSL? is selected, use the port specified byNiFi HTTP port (SSL), and if not, use the port specified by NiFi HTTP port (non-SSL). The port values are available in Advanced nifi-ambari-config.
Save the configuration and restart Knox, or continue by configuring Knox SSO.
Configuring Knox SSO
Steps
Navigate to Advaned knoxsso-topology and, in the KNOXSSO service definition, edit
the Knox SSO token time-to-live value:
<param> <name>knoxsso.token.ttl</name> <value>36000000</value> </param>
Update the whitelist redirect regex property with a regex value that represents the host or domain in which the NiFi host is running:
<param> <name>knoxsso.redirect.whitelist.regex</name> <value>^https?:\/\/(hdf-test\.field\.hortonworks\.com|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value> </param>
If you want to use Knox SSO authentication, perform the following steps:
On each cluster node with Knox installed, replace the ShiroProvider federation provider in the
flow-management.xml
file with the following content:<provider> <role>federation</role> <name>SSOCookieProvider</name> <enabled>true</enabled> <param> <name>sso.authentication.provider.url</name> <value>https://$KNOX_HOSTNAME:$KNOX_PORT/gateway/knoxsso/api/v1/websso</value> </param> </provider>
where:
$KNOX_HOSTNAME is the FQDN of the Knox host.
$KNOX_PORT is the port Knox is using.
Your new
flow-managementso.xml
file looks similar to the following:<?xml version="1.0" encoding="utf-8"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <topology> <gateway> <provider> <role>federation</role> <name>SSOCookieProvider</name> <enabled>true</enabled> <param> <name>sso.authentication.provider.url</name> <value>https://hdf-test.field.hortonworks.com:8443/gateway/knoxsso/api/v1/websso</value> </param> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> </gateway> <service> <role>NIFI</role> <url>https://hdf-test-nifi.field.hortonworks.com:9091</url> <param name="useTwoWaySsl" value="true" /> </service> </topology>
If you want to access NiFi directly and still use Knox SSO:
Export the Knox SSO certificate:
$KNOX_INSTALL_DIR/bin/knoxcli.sh export-cert
Set the following properties in the Advanced nifi-properties section in Ambari:
nifi.security.user.knox.url=https://$KNOX_HOSTNAME:$KNOX_PORT/gateway/knoxsso/api/v1/websso nifi.security.user.knox.publicKey=<path-to>/gateway-identity.pem nifi.security.user.knox.cookieName=hadoop-jwt nifi.security.user.knox.audiences=
The cookieName property must align with what is configured in Knox. The audiences property is used to only accept tokens from a particular audience. The audiences value is configured as part of Knox SSO.
Save the configuration and restart Knox.