Known issues in Flow Management

After upgrading Flow Management Data Hub clusters to Cloudera Public Cloud 7.2.18, you may encounter an issue where the NiFi or NiFi Registry user interface is inaccessible, displaying the following error:

Unable to view the user interface

To resolve this issue in upgraded clusters, you must manually reconfigure the authorization provider to use LDAP. A script is available to automate the configuration update for both NiFi and NiFi Registry.

Follow the steps below to manually configure LDAP on your Flow Management Data Hub cluster:

1. Identify the management node of the Flow Management cluster and copy the Fully Qualified Domain Name (FQDN).
2. SSH into the management node.
3. Copy the script content provided below and save it to a file on the management node.
4. Set executable permissions on the script file: chmod 755 [***script_name***].sh
5. Run the script passing the management node’s FQDN as an argument by using the following command: ./[***script_name***].sh FQDN_OF_MANAGEMENT_NODE
6. When prompted, enter your Cloudera username and password to authorize the changes.

After completing these steps, NiFi and/or NiFi Registry will be configured to use the LDAP user group provider.

The script includes two functions, nifi and nifiregistry, which configure the LDAP user group provider for their respective services. Running the script updates NiFi and/or NiFi Registry to use LDAP, resolving the “Unable to view the user interface” error.

#!/bin/bash
clear
#init incoming variables
GREEN="\033[1;32m"
ORANGE="\033[38;2;255;165;0m"
RESET="\033[0m"
RED="\033[1;31m"
CM_HOST=$('hostname')

#Get my auth
echo -ne "${ORANGE}User Name: " # Need to bracket this var to avoid a space in front
read -s USERNAME
echo -ne "\nEnter Password: $RESET"
read -s PASSWORD
echo #to prevent weird need to hit enter twice
AUTH="$USERNAME:$PASSWORD"

# GetMY LDAP INFO

#extract password and ldap info from cm.settings
if [[ ! -f /etc/cloudera-scm-server/cm.settings ]]; then
  echo -ne "${RED}Error: File /etc/cloudera-scm-server/cm.settings does not exist." \
"\nMust be on Management node of DataHub\n"
exit 1
fi
LDAP_URL=$(awk '/setsettings LDAP_URL/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
LDAP_BIND_DN=$(awk '/setsettings LDAP_BIND_DN/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
LDAP_BIND_PW=$(awk '/setsettings LDAP_BIND_PW/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
LDAP_USER_SEARCH_BASE=$(awk '/setsettings LDAP_USER_SEARCH_BASE/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
LDAP_GROUP_SEARCH_BASE=$(awk '/setsettings LDAP_GROUP_SEARCH_BASE/ {print $NF}' /etc/cloudera-scm-server/cm.settings)


# Get My CM_API and if this fails it could be bad password or host so I will ERROR
CM_API=$(curl -s -k -u "$AUTH" https://$CM_HOST:7183/api/version)
if [[ ${#CM_API} -gt 4 ]]; then # This means probably bad user or password
    echo -ne "$RED Error! Most likely bad credentials below is response\n\n$CM_API $RESET"
    exit 1
fi
CM_HOST_API_URL="https://$CM_HOST:7183/api/$CM_API"
CM_CLUSTER_NAME=$(curl -s -k -u "$AUTH" -X GET "$CM_HOST_API_URL/clusters?clusterType=any&view=SUMMARY" |\
jq -r '.items[].name')


nifi ()
{
SERVICE="nifi-NIFI-BASE"

  mapfile -t CM_ROLES < <(curl --header "Content-Type: application/json" --silent --insecure  --request GET \
  "$CM_HOST_API_URL/clusters/$CM_CLUSTER_NAME/services/$SERVICE/roleConfigGroups" \
  -u $AUTH | jq -r '.items[].name' | grep -v "GATEWAY")

  cat > .cloudera-$SERVICE.json <<- EOF
  {"items":[
    {"name":"nifi.ldap.url","value":"$LDAP_URL"},
{"name":"nifi.ldap.manager.dn","value":"$LDAP_BIND_DN"},
    {"name":"nifi.ldap.manager.password","value":"$LDAP_BIND_PW"},
    {"name":"nifi.ldap.user.search.base","value":"$LDAP_USER_SEARCH_BASE"},
{"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Search Base","value":"$LDAP_GROUP_SEARCH_BASE"},
    {"name":"nifi.ldap.enabled","value":"true"},
    {"name":"xml.authorizers.userGroupProvider.shell-user-group-provider.enabled","value":"false"},
    {"name":"nifi.ldap.authentication.strategy","value":"LDAPS"},
    {"name":"nifi.ldap.tls.protocol","value":"TLS"},
    {"name":"nifi.ldap.tls.keystore.type","value":"jks"},
    {"name":"nifi.ldap.tls.truststore.type","value":"jks"},
    {"name":"nifi.ldap.tls.keystore","value":"\${nifi.security.keystore}"},
    {"name":"nifi.ldap.tls.truststore","value":"\${nifi.security.truststore}"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Object Class","value":"top"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Group Name Attribute","value":"memberOf"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Identity Attribute","value":"uid"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Name Attribute","value":"cn"},
    {"name":"xml.authorizers.userGroupProvider.composite-user-group-provider.property.User Group Provider 2","value":"ldap-user-group-provider"},
    {"name":"staging/login-identity-providers.xml_role_safety_valve","value":"<property><name>xml.loginIdentityProviders.provider.ldap-provider.property.TLS - Keystore Password</name><value>\${nifi.security.keystorePasswd}</value></property><property><name>xml.loginIdentityProviders.provider.ldap-provider.property.TLS - Truststore Password</name><value>\${nifi.security.truststorePasswd}</value></property>"},
    {"name":"staging/authorizers.xml_role_safety_valve","value":"<property><name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Keystore Password</name><value>\${nifi.security.keystorePasswd}</value></property><property><name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Truststore Password</name><value>\${nifi.security.truststorePasswd}</value></property>"}
  ]}
EOF
updateService
}

updateService ()
{
  echo -ne "\n$GREEN Calling CM api to change $SERVICE config\n $RESET"
  for role in "${CM_ROLES[@]}"; do
    echo -ne "$ORANGE \n Updating role $role *** $RESET\n"
    curl -s --header "Content-Type: application/json" --insecure  --request PUT --data @.cloudera-$SERVICE.json \
    -u $AUTH "$CM_HOST_API_URL/clusters/$CM_CLUSTER_NAME/services/$SERVICE/roleConfigGroups/$role/config" > /dev/null
  done
  curl -s --header "Content-Type: application/json" --insecure  --request POST \
  -u $AUTH "$CM_HOST_API_URL/clusters/$CM_CLUSTER_NAME/services/$SERVICE/commands/restart" > /dev/null
  rm -f .cloudera-$SERVICE.json
  echo -ne "\n$GREEN Configured $ORANGE ldap-user-group provider $GREEN on SERVICE: $ORANGE $SERVICE\n\n\n"
}

nifiregistry ()
{
  SERVICE="nifiregistry"
  mapfile -t CM_ROLES < <(curl --header "Content-Type: application/json" --silent --insecure  --request GET \
    "$CM_HOST_API_URL/clusters/$CM_CLUSTER_NAME/services/$SERVICE/roleConfigGroups" \
    -u $AUTH | jq -r '.items[].name' | grep -v "GATEWAY")
  cat > .cloudera-$SERVICE.json <<- EOF
  {"items":[
    {"name": "nifi.registry.ldap.url","value":"$LDAP_URL"},
    {"name": "nifi.registry.ldap.manager.dn","value":"$LDAP_BIND_DN"},
    {"name":"nifi.registry.ldap.manager.password","value":"$LDAP_BIND_PW"},
    {"name":"nifi.registry.ldap.user.search.base","value":"$LDAP_USER_SEARCH_BASE"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Search Base","value":"$LDAP_GROUP_SEARCH_BASE"},
    {"name":"nifi.registry.ldap.enabled","value":"true"},
    {"name":"xml.authorizers.userGroupProvider.shell-user-group-provider.enabled","value":"false"},
    {"name":"nifi.registry.ldap.authentication.strategy","value":"LDAPS"},
    {"name":"nifi.registry.ldap.tls.protocol","value":"TLS"},
    {"name":"nifi.registry.ldap.tls.keystore.type","value":"jks"},
    {"name":"nifi.registry.ldap.tls.truststore.type","value":"jks"},
    {"name":"nifi.registry.ldap.tls.keystore","value":"\${nifi.registry.security.keystore}"},
    {"name":"nifi.registry.ldap.tls.truststore","value":"\${nifi.registry.security.truststore}"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Object Class","value":"top"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Group Name Attribute","value":"memberOf"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Identity Attribute","value":"uid"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Name Attribute","value":"cn"},
    {"name":"xml.authorizers.userGroupProvider.composite-user-group-provider.property.User Group Provider 2","value":"ldap-user-group-provider"},
    {"name":"staging/identity-providers.xml_role_safety_valve","value":"<property><name>xml.loginIdentityProviders.provider.ldap-provider.property.TLS - Keystore Password</name><value>\${nifi.registry.security.keystorePasswd}</value></property><property><name>xml.loginIdentityProviders.provider.ldap-provider.property.TLS - Truststore Password</name><value>\${nifi.registry.security.truststorePasswd}</value></property>"},
    {"name":"staging/authorizers.xml_role_safety_valve","value":"<property><name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Keystore Password</name><value>\${nifi.registry.security.keystorePasswd}</value></property><property><name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Truststore Password</name><value>\${nifi.registry.security.truststorePasswd}</value></property>"}
  ]}
EOF
updateService
}
nifi
nifiregistry
echo -ne "$GREEN\nRestarting $ORANGE NiFi / NiFi Registry $RESET\n"
sleep 10
exit 0

When using the PutIcebergCDC processor, you may encounter an error if the Hadoop Configuration Resources property specified for the Catalog Service only includes the standard Hadoop configuration files from CDP environment (/etc/hadoop/conf/core-site.xml, /etc/hadoop/conf/ssl-client.xml, and /etc/hive/conf/hive-site.xml). The error message states: Failed to specify server’s Kerberos principal name.

Workaround: To resolve this issue, simply add the hdfs-site.xml file to the Hadoop Configuration Resources of the PutIcebergCDC processor’s Catalog Service.

To ensure that Cloudera Manager accurately reflects the NiFi metrics for the NiFi service, you need to update the Flow NiFi access policies in Ranger and include the 'nifi' group.

In Apache NiFi 1.17, the dependency on Apache Avro has been upgraded to 1.11.0. However, the InferAvroSchema processor depends on the hadoop-libraries NAR from which the Avro version comes from, causing a NoSuchMethodError exception.

Having well defined schemas ensures consistent behavior, allows for proper schema versioning, and prevents downstream systems from generating errors because of unexpected schema changes. Besides, schema inference may not always be 100% accurate and can be an expensive operation in terms of performances.

Use the ExtractRecordSchema processor with the proper Reader to infer the Avro schema for your data.

Due to the invalid Python version defined for the NiFi service, the Python API based processors (such as PromptChatGPT, QueryPinecone, and so on) will remain invalid as the NiFi service will be unable to download the associated dependencies. The issue can be resolved by changing the version for the nifi.python.command property.

When using the PutIcebergCDC processor, you may encounter an error if the Hadoop Configuration Resources property specified for the Catalog Service only includes the standard Hadoop configuration files from CDP environment (/etc/hadoop/conf/core-site.xml, /etc/hadoop/conf/ssl-client.xml, and /etc/hive/conf/hive-site.xml). The error message states: Failed to specify server’s Kerberos principal name.

Workaround: To resolve this issue, simply add the hdfs-site.xml file to the Hadoop Configuration Resources of the PutIcebergCDC processor’s Catalog Service.