Fixed Issues in Apache Ranger

Review the list of Ranger issues that are resolved in Cloudera Runtime 7.1.8.

CDPD-35657: Improved create, update, or delete group user mappings API call to reduce memory usage in ranger admin (and avoid OOM). These APIs are called by usersync for updating group memberships from sync source to ranger admin. Also added code to use common DB transaction framework.

This issue is resolved.

CDPD-21274: Support recursive delete operations for Ranger Ozone plugin including upgrade patch for service def changes. Also added plugin side change to retrieve owner information from the request context.

CDPD-29766: When SSO with Knox is enabled, logout should land in Knox logout page.

CDPD-30778: Updated Ozone servicedef to make hadoop.security.authorization config an optional property. Also added corresponding upgrade patch.

CDPD-29402: Introduced new usersync configuration to treat service users like "rangerusersync", "admin", "rangertagsync" that require "ROLE_SYS_ADMIN" privileges.

Fixed issue where removing a role assignment rule from usersync configuration is not updating the role of the affected users.

Also fixed an issue where the role (other than “ROLE_USER”) assigned to an external user manually from Ranger UI, is not reset automatically when usersync is restarted.

CDPD-29265: Modified code to ignore case while validating a user for update (for DBs like MySQL)

CDPD-25879: Introduced new usersync configuration "ranger.usersync.ldap.groupnames" that accepts ";" separated list of group names with wildcards, shortname, or DN format. During startup of usersync added logic to read this configuration to compute the user search filter. Also added new unit tests to cover some functional and error cases.

CDPD-33198: RangerSolrAuthorizer is a common implementation call for both SearchComponent (for Document level Authorization) and AuthorizationPlugin (for collection level Authorization). RangerSolrAuthorizer implementation close() shutdowns the plugin and should be avoided when the call is for SearchComponent. Added check to get the authorizer class name to determine if the call is for SearchComponent or for the authorization plugin.

CDPD-28887: Made changes to use the Logger to log the messages by removing System.out.println(...).

CDPD-28932: 1. Created new API /permissionlist to reduce response object size of the permission listing page.

2. Code optimization for GET API /permission/{id}

This improved response time of the 'permission listing page' & 'edit module permission page' in Ranger permission tab GUI.

OPSAPS-63953: Export script is updated.

OPSAPS-62954: After the fix the default policies created in Ranger Admin should contain the actual configured service users and principal.

OPSAPS-62307: Ranger configurations now expose a safety-valve for authorization-migration-site.xml to allow users to configure required properties for custom configuration of properties which user can configure during migration of policies from Sentry to Ranger.

CDPD-43771: Improves the running time of java patch 55.

CDPD-43675: Improves vastly the running time of java patch 56 under heavy load(users/groups).

CDPD-43647: Improves vastly the running time of java patch 54 under heavy load(users/groups).

CDPD-43148: Remove duplicate resource-signature's policies during ranger upgrade.

CDPD-42963: RangerClient renews Kerberos ticket after a lifetime with the fix.

CDPD-42607: Incremental Sync config parameter is read from config with the fix.

CDPD-41379: Fixed the client IP and blank resource name for KMS audit logs.

CDPD-41280: Fix Java patch J10033 and J10046 failure during ranger upgrade.

CDPD-41200: Show the alert only once if the resource lookup fails.

CDPD-41153: Updating the service config during upgrade which has unsupported access types e.g - others, solr_admin.

CDPD-41148: Test ozone connection for ozone service works with the fix.

CDPD-41126: Code fixed for Ranger API Resource Metrics REST to update "Up time of JVM".

CDPD-40961: Opensearch Support on Ranger Admin.

CDPD-40792: Upgrade google-oauth-client to 1.33.3 due to CVE-2021-22573.

CDPD-40778: Upgrade netty to 4.1.77 due to CVE-2022-24823.

CDPD-40519: Upgrade tomcat to 8.5.79 due to CVE-2022-29885.

CDPD-40440: Upgraded the spring security version as part of CVE fix.

CDPD-40268: 1. Fixed infinite loop in filtering out Ancestor resources 2. Filter out default-db from the mapped Hive resource if any other resource is also mapped. 3. Checking all mapped hive resources for access permissions 4. Added a config flag - ranger-rms.enable.database.sync (default true). If true, database level sync is enabled 5. Fixed ALTER_TABLE notification event processing 6. Added more debug messages to ChainedPlugin

CDPD-40121: Upgraded the guava_version as part of CVE fix.

CDPD-39931: Remove duplicate access types entries during the ranger policy creation.

CDPD-39899: Fixed Upgrade Failure issue when we have sole policy like: 1. policy1 -> resourece -> * 2. policy2 -> resource -> *,xyz Then during upgrade solr Java patch was failing fixed the same.

CDPD-39780: 1. Fixed infinite loop in filtering out Ancestor resources 2. Filter out default-db from the mapped Hive resource if any other resource is also mapped. 3. Checking all mapped hive resources for access permissions 4. Added a config flag - ranger-rms.enable.database.sync (default true). If true, database level sync is enabled 5. Fixed ALTER_TABLE notification event processing 6. Added more debug messages to ChainedPlugin

CDPD-39690: There is a change in external user 'status' (i.e x_portal_user tables column) which are getting synced into ranger admin, default ‘status’ value of synced users are getting set as 0(disabled) which was not the case in 7.1.4 This is the behaviour change between 7.1.4 and later versions: 1. Added change to mark external users status as enable(1). 2. written a java patch to update the status of existing external users during upgrade.

CDPD-39359: Replace ElasticSearch to OpenSearch 1.3.2 in Ranger due to CVE. ElasticSearch cannot be upgraded due to Licensing issue hence it is replaced with OpenSearch 1.3.2.

CDPD-39328: Add python3 support in ranger install scripts.

CDPD-39319: Fix NullPointerException in get service REST call.

CDPD-39317: Updated atlas default audit filter to avoid auditing for atlas read-entity by nifi service user.

CDPD-39234: As part of this change we have introduced new password policy in ranger admin i.e - Password should be minimum 8 characters, at least one uppercase letter, one lowercase letter and one numeric.

CDPD-39182: Remove semicolon from c3p0 preferredTestQuery.

CDPD-38586: Upgrade Spring Framework to 5.3.18 due to CVE-2022-22963, CVE-2022-22965.

CDPD-37019: Fixes null pointer exception in Java Patch 54.

CDPD-36320: Current Password policies(validation) is not strong enough as it expect the "minimum 8 characters with minimum one alphabet and one numeric".

In this improvement Jira will enhance the password policies to "minimum 8 characters, at least one uppercase letter, one lowercase letter and one numeric".

CDPD-35870: This provides a feature to skip the {OWNER} policy in authzmigration tool.

CDPD-35865: There is a change in external user 'status' (i.e x_portal_user tables column) which are getting synced into ranger admin, default ‘status’ value of synced users are getting set as 0(disabled) which was not the case in 7.1.4 This is the behaviour change between 7.1.4 and later versions.

Added change to mark external users status as enable(1) and written a java patch to update the status of existing external users.

CDPD-35631: Fixed role update operation issue for role admin user. (A non admin user should be able to update the role if he/she is role admin).

CDPD-35628: If RangerRMS cannot renew it's ticket cache due to a KDC communication problem then it will not retry it and we'll see periodic "No ticket found in the cache" error messages.

If that happens, then it won't have a valid Kerberos ticket it will not be able to communicate with other services, like HMS.

CDPD-34449: Added support for JWT authentication to access Ranger RMS API.

CDPD-35463: Ranger RMS download mapping API was an open API. After changes of CDPD-34449 it will add JWT authentication for the RMS API. This JIRA adds support for Kerberos authentication to access Ranger RMS API.

If JWT authentication is not provided it will fall back on kerberos authentication for backward compatibility.

CDPD-40344: Enable authentication for Ranger RMS download mapping API. was an open API.

Prior to 7.1.8 Ranger RMS download mapping API was an open API. Now it is made secured and it requires JWT/Kerberos authentication to access this API.

CDPD-35455: Added jackson-core libs for kafka plugin.

CDPD-35453: Set ranger service name in RangerServiceResource object to avoid tag import failure.

CDPD-35073: Upgrade jquery-ui 1.12 to 1.13.0+ due to CVEs.

CDPD-35048: Hive service users (specified by configuration parameter "ranger.plugin.hdfs.privileged.user.names" with a default value of "admin,dpprofiler,hue,beacon,hive,impala") do not need to be authorized by hive-hdfs chained plugin using Hive policies for access to the storage locations that map to hive entities. Such locations are already allowed access through HDFS default policies.

CDPD-34762: User/group/tags/resource attributes should be easily accessible in condition expressions, with expressions like:

USER.state == 'CA' UG['test'].dept == 'MKTG' REQ.accessType == 'SELECT' RES.database == 'hr' RES.table == 'employee' TAG._type == 'PII' TAG.attr1 == 'value1' TAGS.PII.attr1 == 'value1' TNAMES.length == 2 TNAMES.indexOf('PCI') != -1

CDPD-34750: This change is to add the support for retry for policies download, ugsync, tagsync.

CDPD-34723: Policy engine evaluates policies in the following order: priority, has-deny, has-no-deny. When multiple policies have same priority/has-deny/has-no-deny, the ordering is not deterministic. This doesn't impact the result for access policies - as all denies will be evaluated before allows. However, the result for masking/row-filter can vary when multiple policies exists for a given resource, and these policies define different mask/filter for a given user/group/role.

Given name of a policy is unique within a service, using policy name as the secondary sorting key will result in deterministic evaluation order.

CDPD-34517: Made the required changes for initialising the Key store in case LUNA HSM with Ranger KMS DB.

CDPD-34023: Chmod and Chown will honor the ranger policy in both with fallback enabled as well as disabled.

Workaround is to have the parent directory RX permission in HDFS for the failing folders/files.

CDPD-33933: Corrected the JSON format.

CDPD-33856: Upgrade protobuf-java to 3.19.3 due to CVE-2021-22569.

CDPD-33729: This is to fix the bug in authorizing StorageBased Handler in Hive commands.

CDPD-33697: Code fixed for Auditor role in Ranger, not to edit policy's name and description.

CDPD-33606: Upgraded the Kylin version.

CDPD-33196: HDFS audit files rollover improvement to trigger rollover in monitoring thread and this enables the audit files to be closed exactly at the specified time.

CDPD-32975: Storm library version in Ranger upgraded to fix the CVE.

CDPD-32974: kylin library version in Ranger upgraded to fix the CVE.

CDPD-32900: Metric details for kms are not getting collected. Fixed issue of metric info to be provided API.

CDPD-32879: Added a config "ranger-rms.max.requested.notifications" to limit the size of requested notifications during the delta-sync. If we set the config value < 1 or > 50000; the default value will be treated as maxRequestedNotifications=50000. The default value for MAX_REQUESTED_NOTIFICATIONS is 50000.

It also fix the bug found during testing.

1. handleDeltaSync loop runs infinite when it tries to fetch notifications in batch.

2. Full-sync does not reset last_known_version=-1 in x_rms_mapping_provider table.

CDPD-31886: Oracle JDBC Storage Handler based Hive operation gets authorized correctly now.

Work around is to have "*" policy for the storage handler.

CDPD-31780: This change is to integrate/certify the Ranger DB KMS with GCP.

CDPD-31574: Provide an option to optimize space needed by Trie objects.

CDPD-31546: A delegate admin user should be able to add another user with all or subset of permissions they have.

CDPD-31508: This fix will make policy resource signature unique in a service.

CDPD-31466: Code fixed to update policy guid to unique value.

CDPD-31361: Fixed the performance issue of J10045 patch is taking more time to apply when we upgrade form CDH-7.1.5 to CDH-7.1.8.

CDPD-31357: Click on the policy resource field that time all available resource options are listed down.

CDPD-31225: Fixed the memory issue for Ranger KMS.

CDPD-31159: Coarse URI check on the URL will be done to fix the performance issue when URI location contains lot of files / folders. This feature can be enabled by the param. xasecure.hive.uri.permission.coarse.check=true in ranger-hive-security.xml.

CDPD-31076: Created framework to execute DB patch dependent on Java patch.

CDPD-30888: Show Role Grant command failure fixed in this JIRA.

CDPD-30653: Fix for the issue of incremental policy updates do not work correctly for multiple security zones.

CDPD-30472:Added a config "xasecure.audit.solr.limit.query.req.size" (defaults to Integer.MAX_VALUE). This config will limit the query size stored in Audit logs. Recommended value is 1024, however default value is set to Integer.MAX_VALUE.

Setting appropriate (small) value of this config will help optimize large size audit events and avoid load on solr/hdfs.

CDPD-30419: Improvement done in the logging of Ranger by remove unnecessary details and making the log pollution free.

CDPD-30408: Ranger-tagsync supports a new configuration parameter ("ranger.tagsync.dest.ranger.max.batch.size" - default value: 1) that controls the size of the batch of Atlas notifications that are uploaded in one go to Ranger admin.

CDPD-30032: Kafka library in Apache ranger upgraded to 2.8 version.

CDPD-29866: This issue was caused because a feature build in https://jira.cloudera.com/browse/CDPD-27138 was ported to 7.2.12.0 and this JIRA took care of it.

CDPD-29748: Problem: Policy version in access audit is not matching with the policy version seen in policy view. x_policy table has version column which contains the latest policy version. this is updated during policy update.

Solution: x_policy table has version column which contains the latest policy version. this is updated during policy update. RangerPolicy object version field should be updated from the existing x_policy table version field value during the delta calculation. updated policy object shall be retrieved by plugins and passed to solr along with access audit logs.

CDPD-29489: Authorization of URL in Hive command will now handle with and without trailing "/" and treat both the urls the same way like how POSIX behave.

CDPD-29334: This code fix closes all the connection to RMS after fetching the notification.

CDPD-29322: Fixed usersync code to update other attributes properly for users or groups that have empty values.

CDPD-29211: This fix upgrade Spring Security to 5.5.1+ due to CVE-2021-22119.

CDPD-29186: Added _csrf token header validation to prevent CSRF attack.

CDPD-28836: Made changes to fetch all the public group and current user({USER}) assigned policies when user search for policies on report page with username.

CDPD-28758: Users who need to rotate Access logs need to add property ranger.accesslog.rotate.max.days and configure numeric value for same in the safety valve for ranger-admin-site.xml.

CDPD-28752: Provided sorting on specific columns on policy and on audit listing page.

CDPD-28669: Update algorithm to build Ranger policy-database object from Ranger policy-view object.

CDPD-28619: After spring upgrade, spring-jcl-5.3.7.jar is loaded into ranger classpath which causes this issue. To fix this issue we have removed spring-jcl-5.3.7.jar from Ranger-admin, Ranger-RMS, Ranger-RAZ pom.xml. Now ranger does not add spring-jcl-5.3.7.jar in it's packaging.

CDPD-28535: This patch has changes to improve error message while deleting users and groups which are associated with role(s).

CDPD-28453: deleteSnapshot command authorization in Ranger HDFS authorizer.

CDPD-28217: This fix is a part of the implementation of Database level ACL Synchronization feature for RMS.

CDPD-28141: Upgarded Logredactor version as part of CVE fix.

CDPD-28057: Added X-Permitted-Cross-Domain-Policies Response header.

CDPD-28050: Overriedden searchModuleDef function in XModuleDefService which will fetch users and groups only once instead of performing same operation for every ModuleDef. Creating a Map object by traversing through all users & groups was costly operation. Therefore defined two new functions getXXGroupIdNameMap() and getXXPortalUserIdXXUserNameMap() which will use sql query to return map of id & name instead of the actual objects. This code reduces database calls as well as memory consumption and improves response time.

CDPD-28049: Improved response time dramatically for GET API /service/xusers/lookup/users. API response returns instantaneously for 50,000 users lookup.

CDPD-27740: json-smart library version in Ranger upgraded to fix the CVE.

CDPD-27335: Ranger Admin / KMS / KMS-KTS server work directory can now be configured through the parameter {{ranger.tomcat.work.dir}}.

Ranger RMS server work directory can now be configured through the parameter {{ranger-rms.tomcat.work.dir}}.

Ranger Raz server work directory can now be configured through the parameter {{ranger.raz.tomcat.work.dir}}.

CDPD-27138: Apache Ranger REST Client to download policies, tags and roles from Ranger admin will use cookie session. Earlier each of the plugin has to do kerberos login to get a TGT to download policy, tags and roles. With this feature Session Cookie is enabled by default in RangerAdminClient and it will be used instead of hitting KDC for TGT for validating the user. This improve performance as well and reduce the load on KDC.

CDPD-26774: CVE fixed by removing htrace-core4.jar from Ranger repo.

CDPD-26575: Spring framework version in Ranger upgraded to fix the CVE.

CDPD-26069: This is bug for the issue around ALTER Operations on StorageBase Location in the hive commands.

CDPD-25938: As part of this change we have removed one policy item from storage policy with rangerlookup user which has unused access type.

CDPD-25594: When there are large no. of group mappings, all the DB updates are cached in memory and causes OOM issue in ranger. In order to fix this, added code to create individual DB transaction for add/update of user group mapping from Ranger Usersync.

CDPD-24787: Exposes two new attributes from the users/groups page in Ranger UI.

CDPD-24277: Code fixed for Ranger role to be not deleted if the role is used in ranger audit filters in service plugin.

CDPD-23560: Convert existing custom mysql UDFs to procedure.

CDPD-23446: Create ranger users in a separate transaction to avoid service creation failure.

CDPD-21398: After the fix Ranger Admin now allows user to customise the private keystore instead of using the default value.

CDPD-17304: Made changes to use the custom config set zip file to create the config-set which is used by Ranger Admin start script to create the collection, Previously config-set location was fixed.

CDPD-13144: The description of the fix is in the jira name.

CDPD-5963: Implement SHOW ROLE GRANT in Hive ranger plugin which gets the grant details from Ranger Policy.

CDPD-7143: Users can now configure Ranger Admin to allow browser based kerberos authenticated login. If enabled without any ticket on the browsers Users may land on a blank page and may need users to refresh the page which can redirect them to the login page.

CDPD-7813: This fixes the auditing for INSERT, UPDATE, DELETE, TRUNCATE statement of hive table. Prior to this fix all these commands are identified as UPDATE only and auditing was ambiguous.