Known issues in Flow Management

To resolve this issue in upgraded clusters, follow the steps below to manually configure your Flow Management Data Hub cluster to use the LDAP user group provider:

1. Identify the management node of the Flow Management cluster and copy the Fully Qualified Domain Name (FQDN).
2. SSH into the management node.
3. Copy the script provided below and save it to a file.
4. Set executable permissions on the script file: chmod 755 script_name.sh
5. Run the script using the following command: ./script_name.sh FQDN_OF_MANAGEMENT_NODE
6. Enter your Cloudera credentials when prompted for a username and password.

After completing these steps, NiFi will be configured to use the LDAP user group provider, resolving the "Unable to view the user interface" issue.

#!/bin/bash
              clear
              #init incoming variables
              GREEN="\033[1;32m"
              ORANGE="\033[38;2;255;165;0m"
              RESET="\033[0m"
              RED="\033[1;31m"
              CM_HOST=$1
              
              #Get my auth
              echo -ne "${ORANGE}User Name: " # Need to bracket this var to avoid a space in front
              read -s USERNAME
              echo -ne "\nEnter Password: $RESET"
              read -s PASSWORD
              echo #to prevent weird need to hit enter twice
              AUTH="$USERNAME:$PASSWORD"
              
              # Get My CM_API and if this fails it could be bad password or host so I will ERROR
              CM_API=$(curl -s -k -u "$AUTH" https://$CM_HOST:7183/api/version)
              if [[ ${#CM_API} -gt 4 ]]; then # This means probably bad user or password
              echo -ne "$RED Error! Most likely bad credentials below is response\n\n$CM_API $RESET"
              exit 1
              fi
              
              CM_HOST_API_URL="https://$CM_HOST:7183/api/$CM_API"
              CM_CLUSTER_NAME=$(curl -s -k -u "$AUTH" -X GET "$CM_HOST_API_URL/clusters?clusterType=any&view=SUMMARY" |\
              jq -r '.items[].name')
              mapfile -t CM_ROLES < <(curl --header "Content-Type: application/json" --silent --insecure  --request GET \
              "$CM_HOST_API_URL/clusters/$CM_CLUSTER_NAME/services/nifi-NIFI-BASE/roleConfigGroups" \
              -u $AUTH | jq -r '.items[].name' | grep -v "nifi-NIFI-BASE-GATEWAY-BASE")
              
              #extract password and ldap info from cm.settings
              LDAP_URL=$(awk '/setsettings LDAP_URL/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
              LDAP_BIND_DN=$(awk '/setsettings LDAP_BIND_DN/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
              LDAP_BIND_PW=$(awk '/setsettings LDAP_BIND_PW/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
              LDAP_USER_SEARCH_BASE=$(awk '/setsettings LDAP_USER_SEARCH_BASE/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
              LDAP_GROUP_SEARCH_BASE=$(awk '/setsettings LDAP_GROUP_SEARCH_BASE/ {print $NF}' /etc/cloudera-scm-server/cm.settings)
              
              echo -ne "$GREEN Building CM API payload to move away from shell-user-group and into ldap-user-group provider\n $RESET"
              
              cat > .cloudera-payload.json <<- EOF
              {"items":[
              {"name":"nifi.ldap.url","value":"$LDAP_URL"},
              {"name":"nifi.ldap.manager.dn","value":"$LDAP_BIND_DN"},
              {"name":"nifi.ldap.manager.password","value":"$LDAP_BIND_PW"},
              {"name":"nifi.ldap.user.search.base","value":"$LDAP_USER_SEARCH_BASE"},
              {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Search Base","value":"$LDAP_GROUP_SEARCH_BASE"},
              {"name":"nifi.ldap.enabled","value":"true"},
              {"name":"xml.authorizers.userGroupProvider.shell-user-group-provider.enabled","value":"false"},
              {"name":"nifi.ldap.authentication.strategy","value":"LDAPS"},
              {"name":"nifi.ldap.tls.protocol","value":"TLS"},
              {"name":"nifi.ldap.tls.keystore.type","value":"jks"},
              {"name":"nifi.ldap.tls.truststore.type","value":"jks"},
              {"name":"nifi.ldap.tls.keystore","value":"\${nifi.security.keystore}"},
              {"name":"nifi.ldap.tls.truststore","value":"\${nifi.security.truststore}"},
              {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Object Class","value":"top"},
              {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Group Name Attribute","value":"memberOf"},
              {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Identity Attribute","value":"uid"},
              {"name":"xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Name Attribute","value":"cn"},
              {"name":"xml.authorizers.userGroupProvider.composite-user-group-provider.property.User Group Provider 2","value":"ldap-user-group-provider"},
              {"name":"staging/login-identity-providers.xml_role_safety_valve","value":"<property><name>xml.loginIdentityProviders.provider.ldap-provider.property.TLS - Keystore Password</name><value>\${nifi.security.keystorePasswd}</value></property><property><name>xml.loginIdentityProviders.provider.ldap-provider.property.TLS - Truststore Password</name><value>\${nifi.security.truststorePasswd}</value></property>"},
              {"name":"staging/authorizers.xml_role_safety_valve","value":"<property><name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Keystore Password</name><value>\${nifi.security.keystorePasswd}</value></property><property><name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Truststore Password</name><value>\${nifi.security.truststorePasswd}</value></property>"}
              ]}
              EOF
              
              echo -ne "\n$GREEN Call CM api to change nifi config\n\n $RESET"
              for role in "${CM_ROLES[@]}"; do
              echo -ne "$ORANGE Updating role $role *** $RESET\n"
              curl -s --header "Content-Type: application/json" --insecure  --request PUT --data @.cloudera-payload.json \
              -u $AUTH "$CM_HOST_API_URL/clusters/$CM_CLUSTER_NAME/services/nifi-NIFI-BASE/roleConfigGroups/$role/config" > /dev/null
              done
              echo -ne "$GREEN\nRestarting NiFi $RESET\n"
              curl -s --header "Content-Type: application/json" --insecure  --request POST \
              -u $AUTH "$CM_HOST_API_URL/clusters/$CM_CLUSTER_NAME/services/nifi-NIFI-BASE/commands/restart" > /dev/null
              rm -f .cloudera-payload.json 

To resolve this issue, simply add the hdfs-site.xml file to the Hadoop Configuration Resources of the PutIcebergCDC processor’s Catalog Service.

Having well defined schemas ensures consistent behavior, allows for proper schema versioning, and prevents downstream systems from generating errors because of unexpected schema changes. Besides, schema inference may not always be 100% accurate and can be an expensive operation in terms of performances.

Use the ExtractRecordSchema processor with the proper Reader to infer the Avro schema for your data.

To resolve this issue, simply add the hdfs-site.xml file to the Hadoop Configuration Resources of the PutIcebergCDC processor’s Catalog Service.