Known issues in Flow Management

After upgrading to Cloudera Public Cloud 7.2.18, you may encounter the error message "Unable to view the user interface." This issue occurs because, in versions prior to 7.2.18, NiFi group authorization relied on the host's SSSD configuration for group synchronization.

To resolve this issue in upgraded clusters, follow the steps below to manually configure your Flow Management Data Hub cluster to use the LDAP user group provider:

1. Identify the management node of the Flow Management cluster and copy the Fully Qualified Domain Name (FQDN).
2. SSH into the management node.
3. Copy the script provided below and save it to a file.
4. Set executable permissions on the script file: chmod 755 script_name.sh
5. Run the script using the following command: ./script_name.sh FQDN_OF_MANAGEMENT_NODE
6. Enter your Cloudera credentials when prompted for a username and password.

After completing these steps, NiFi will be configured to use the LDAP user group provider, resolving the "Unable to view the user interface" issue.

#!/bin/bash
clear
#init incoming variables
GREEN="\033[1;32m"
ORANGE="\033[38;2;255;165;0m"
RESET="\033[0m"
RED="\033[1;31m"
CM_HOST=$1

#Get my auth
echo -ne "${ORANGE}User Name: " # Need to bracket this var to avoid a space in front
read -s USERNAME
echo -ne "\nEnter Password: $RESET"
read -s PASSWORD
echo #to prevent weird need to hit enter twice
AUTH="$USERNAME:$PASSWORD"

# Get My CM_API and if this fails it could be bad password or host so I will ERROR
CM_API=$(curl -s -k -u "$AUTH" https://$CM_HOST:7183/api/version)
if [[ ${#CM_API} -gt 4 ]]; then # This means probably bad user or password
    echo -ne "$RED Error! Most likely bad credentials below is response\n\n$CM_API $RESET"
    exit 1
fi

CM_HOST_API_URL="https://$CM_HOST:7183/api/$CM_API"
CM_CLUSTER_NAME=$(curl -s -k -u "$AUTH" -X GET "$CM_HOST_API_URL/clusters?clusterType=any&view=SUMMARY" |\
jq -r '.items[].name')
mapfile -t CM_ROLES < <(curl --header "Content-Type: application/json" --silent --insecure  --request GET \
"$CM_HOST_API_URL/clusters/$CM_CLUSTER_NAME/services/nifi-NIFI-BASE/roleConfigGroups" \
-u $AUTH | jq -r '.items[].name' | grep -v "nifi-NIFI-BASE-GATEWAY-BASE")

#extract password and ldap info from cm.settings
LDAP_URL=$(awk '/setsettings LDAP_URL/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
LDAP_BIND_DN=$(awk '/setsettings LDAP_BIND_DN/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
LDAP_BIND_PW=$(awk '/setsettings LDAP_BIND_PW/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
LDAP_USER_SEARCH_BASE=$(awk '/setsettings LDAP_USER_SEARCH_BASE/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
LDAP_GROUP_SEARCH_BASE=$(awk '/setsettings LDAP_GROUP_SEARCH_BASE/ {print $NF}' /etc/cloudera-scm-server/cm.settings)

echo -ne "$GREEN Building CM API payload to move away from shell-user-group and into ldap-user-group provider\n $RESET"

cat > .cloudera-payload.json <<- EOF
{"items":[
    {"name":"nifi.ldap.url","value":"$LDAP_URL"},
    {"name":"nifi.ldap.manager.dn","value":"$LDAP_BIND_DN"},
    {"name":"nifi.ldap.manager.password","value":"$LDAP_BIND_PW"},
    {"name":"nifi.ldap.user.search.base","value":"$LDAP_USER_SEARCH_BASE"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Search Base","value":"$LDAP_GROUP_SEARCH_BASE"},
    {"name":"nifi.ldap.enabled","value":"true"},
    {"name":"xml.authorizers.userGroupProvider.shell-user-group-provider.enabled","value":"false"},
    {"name":"nifi.ldap.authentication.strategy","value":"LDAPS"},
    {"name":"nifi.ldap.tls.protocol","value":"TLS"},
    {"name":"nifi.ldap.tls.keystore.type","value":"jks"},
    {"name":"nifi.ldap.tls.truststore.type","value":"jks"},
    {"name":"nifi.ldap.tls.keystore","value":"\${nifi.security.keystore}"},
    {"name":"nifi.ldap.tls.truststore","value":"\${nifi.security.truststore}"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Object Class","value":"top"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Group Name Attribute","value":"memberOf"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Identity Attribute","value":"uid"},
    {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Name Attribute","value":"cn"},
    {"name":"xml.authorizers.userGroupProvider.composite-user-group-provider.property.User Group Provider 2","value":"ldap-user-group-provider"},
    {"name":"staging/login-identity-providers.xml_role_safety_valve","value":"<property><name>xml.loginIdentityProviders.provider.ldap-provider.property.TLS - Keystore Password</name><value>\${nifi.security.keystorePasswd}</value></property><property><name>xml.loginIdentityProviders.provider.ldap-provider.property.TLS - Truststore Password</name><value>\${nifi.security.truststorePasswd}</value></property>"},
    {"name":"staging/authorizers.xml_role_safety_valve","value":"<property><name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Keystore Password</name><value>\${nifi.security.keystorePasswd}</value></property><property><name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Truststore Password</name><value>\${nifi.security.truststorePasswd}</value></property>"}
]}
EOF

echo -ne "\n$GREEN Call CM api to change nifi config\n\n $RESET"
for role in "${CM_ROLES[@]}"; do
   echo -ne "$ORANGE Updating role $role *** $RESET\n"
   curl -s --header "Content-Type: application/json" --insecure  --request PUT --data @.cloudera-payload.json \
   -u $AUTH "$CM_HOST_API_URL/clusters/$CM_CLUSTER_NAME/services/nifi-NIFI-BASE/roleConfigGroups/$role/config" > /dev/null
done
echo -ne "$GREEN\nRestarting NiFi $RESET\n"
curl -s --header "Content-Type: application/json" --insecure  --request POST \
-u $AUTH "$CM_HOST_API_URL/clusters/$CM_CLUSTER_NAME/services/nifi-NIFI-BASE/commands/restart" > /dev/null
rm -f .cloudera-payload.json 

When using the PutIcebergCDC processor, you may encounter an error if the Hadoop Configuration Resources property specified for the Catalog Service only includes the standard Hadoop configuration files from CDP environment (/etc/hadoop/conf/core-site.xml, /etc/hadoop/conf/ssl-client.xml, and /etc/hive/conf/hive-site.xml). The error message states: Failed to specify server’s Kerberos principal name.

Workaround: To resolve this issue, simply add the hdfs-site.xml file to the Hadoop Configuration Resources of the PutIcebergCDC processor’s Catalog Service.

To ensure that Cloudera Manager accurately reflects the NiFi metrics for the NiFi service, you need to update the Flow NiFi access policies in Ranger and include the 'nifi' group.

In Apache NiFi 1.17, the dependency on Apache Avro has been upgraded to 1.11.0. However, the InferAvroSchema processor depends on the hadoop-libraries NAR from which the Avro version comes from, causing a NoSuchMethodError exception.

Having well defined schemas ensures consistent behavior, allows for proper schema versioning, and prevents downstream systems from generating errors because of unexpected schema changes. Besides, schema inference may not always be 100% accurate and can be an expensive operation in terms of performances.

Use the ExtractRecordSchema processor with the proper Reader to infer the Avro schema for your data.

Due to the invalid Python version defined for the NiFi service, the Python API based processors (such as PromptChatGPT, QueryPinecone, and so on) will remain invalid as the NiFi service will be unable to download the associated dependencies. The issue can be resolved by changing the version for the nifi.python.command property.

When using the PutIcebergCDC processor, you may encounter an error if the Hadoop Configuration Resources property specified for the Catalog Service only includes the standard Hadoop configuration files from CDP environment (/etc/hadoop/conf/core-site.xml, /etc/hadoop/conf/ssl-client.xml, and /etc/hive/conf/hive-site.xml). The error message states: Failed to specify server’s Kerberos principal name.

Workaround: To resolve this issue, simply add the hdfs-site.xml file to the Hadoop Configuration Resources of the PutIcebergCDC processor’s Catalog Service.