Migrating Audit Logs from DB to Solr in Ambari Clusters
It is recommended that you store audits in both HDFS and Solr. Audit to DB is no longer recommended and the option is disabled in the Ambari UI. If your logs were previously stored on DB, you can migrate the logs to Solr.
Note | |
---|---|
By default, Solr only indexes the last 30 days' logs. |
Before you migrate your audit logs from DB to Solr, make sure your cluster meets the following requirements:
Solr must be installed and running (see Using Apache Solr for Ranger Audits).
All plug-ins must be upgraded and writing audit logs to Solr (i.e., plugins must not be writing audit logs to DB.)
The DB server must be running, and the credentials used to connect Audit to DB must be available.
Ranger must be running with the audit source as Solr, and the Solr URL must be configured.
To migrate your audit logs from DB to Solr, complete the following instructions:
Configure the properties
ranger.audit.source.type
andranger.audit.solr.urls
:Property Name Sample Value Location ranger.audit.source.type
solr
Ranger>Configs>Advanced>Advanced ranger-admin-site ranger.audit.solr.urls
Syntax: http://
Example:<solr_host>
:<port>
/solr/ranger_auditshttp://192.168.0.2:8983/solr/ranger_audits
Example:http://192.168.0.2:8886/solr/ranger_audits
Ranger>Configs>Ranger Audit Verify or enter the
ranger.jpa.audit.jdbc.url
value.After upgrading Ranger and changing the audit log destination from DB to Solr, Ambari may not automatically populate the required property values. If necessary, you can add these as custom properties from Ambari.
Select Ranger>Configs>Advanced>Custom ranger-admin-site, then click Add Property....
Enter the following information on the Add Property pop-up:
Type: preloaded with the value
ranger-admin-site.xml
Key: enter
ranger.jpa.audit.jdbc.url
Value: enter the JDBC audit string for your DB platform:
Table 5.4. JDBC Audit String
DB Platform Syntax Example Value MySQL jdbc:mysql://
DB_HOST
:PORT
/audit_name
jdbc:mysql://c6401.ambari.apache.org:3306/ranger_audit Oracle For Oracle SID:
jdbc:oracle:thin:@
AUDIT_HOST
:PORT
:SID
jdbc:oracle:thin:@c6401.ambari.apache.org:1521:ORCL For Oracle Service Name:
jdbc:oracle:thin:@//
AUDIT_HOST
[:PORT
][/ServiceName
]jdbc:oracle:thin:@//c6401.ambari.apache.org:1521/XE PostgreSQL jdbc:postgresql:// AUDIT_HOST
/audit_name
jdbc:postgresql://c6401.ambari.apache.org:5432/ranger_audit MS SQL jdbc:sqlserver:// AUDIT_HOST
;databaseName=audit_name
jdbc:sqlserver://c6401.ambari.apache.org:1433;databaseName=ranger_audit SQLA jdbc:sqlanywhere:host= AUDIT_HOST
;database=audit_name
jdbc:sqlanywhere:host=c6401.ambari.apache.org:2638;database=ranger_audit
Restart Ranger Admin.
Navigate to the Ranger admin directory and run the following command:
$/path/to/java
-Dlogdir=ews/logs -Dlog4j.configuration=db_patch.log4j.xml -cp ews/webapp/WEB-INF/classes/conf:ews/webapp/WEB-INF/classes/lib/*:ews/webapp/WEB-INF/:ews/webapp/META-INF/:ews/webapp/WEB-INF/lib/*:ews/webapp/WEB-INF/classes/:ews/webapp/WEB-INF/classes/META-INF:/usr/share/java/mysql-connector-java.jar org.apache.ranger.patch.cliutil.DbToSolrMigrationUtilIf the script succeeds, it prints the following details on the screen:
Processing batch 'n' of total 'noOfBatches' (Where each batch contains 10000 rows.)
Total number of migrated audit logs.
If the script fails to migrate data, it returns the error: Migration process failed, Please refer ranger_db_patch.log file.