Migrating Audit Logs from DB to Solr in Ambari Clusters
How to migrate your audit logs from DB to Solr. It is recommended that you store audits in both HDFS and Solr.
Before you migrate your audit logs from DB to Solr, make sure
your cluster meets the following requirements:
- Solr must be installed and running (see “Using Apache Solr for Ranger Audits”).
- All plug-ins must be upgraded and writing audit logs to Solr (i.e., plugins must not be writing audit logs to DB.)
- The DB server must be running, and the credentials used to connect Audit to DB must be available.
- Ranger must be running with the audit source as Solr, and the Solr URL must be configured.
-
Configure the properties
ranger.audit.source.type
andranger.audit.solr.urls
:Property Name Sample Value Location ranger.audit.source.type
solr
Ranger>Configs>Advanced>Advanced ranger-admin-site ranger.audit.solr.urls
Syntax: http://<solr_host>:<port>/solr/ranger_audits
Example:http://192.168.0.2:8983/solr/ranger_audits
Example:http://192.168.0.2:8886/solr/ranger_audits
Ranger>Configs>Ranger Audit -
Verify or enter the
ranger.jpa.audit.jdbc.url
value. -
-
After upgrading Ranger and changing the audit log destination from DB to Solr,
Ambari may not automatically populate the required property values. If necessary, you
can add these as custom properties from Ambari.
- Select Ranger>Configs>Advanced>Custom ranger-admin-site, then click Add Property....
-
Enter the following information on the Add Property
pop-up:
- Type: preloaded with the value
ranger-admin-site.xml
-
Key: enter
ranger.jpa.audit.jdbc.url
- Value: enter the JDBC audit string for your DB platform:
Table 1. JDBC Audit String DB Platform Syntax Example Value MySQL jdbc:mysql://DB_HOST:PORT/audit_name
jdbc:mysql://c6401.ambari.apache.org:3306/ranger_audit Oracle For Oracle SID:
jdbc:oracle:thin:@AUDIT_HOST:PORT:SID
jdbc:oracle:thin:@c6401.ambari.apache.org:1521:ORCL For Oracle Service Name:
jdbc:oracle:thin:@//AUDIT_HOST[:PORT][/ServiceName]
jdbc:oracle:thin:@//c6401.ambari.apache.org:1521/XE PostgreSQL jdbc:postgresql://AUDIT_HOST/audit_name jdbc:postgresql://c6401.ambari.apache.org:5432/ranger_audit MS SQL jdbc:sqlserver://AUDIT_HOST;databaseName=audit_name jdbc:sqlserver://c6401.ambari.apache.org:1433;databaseName=ranger_audit SQLA jdbc:sqlanywhere:host=AUDIT_HOST;database=audit_name jdbc:sqlanywhere:host=c6401.ambari.apache.org:2638;database=ranger_audit - Type: preloaded with the value
-
Restart Ranger Admin:
service ranger-admin restart
. -
Navigate to the Ranger admin directory and run the following command:
$/path/to/java -Dlogdir=ews/logs -Dlog4j.configuration=db_patch.log4j.xml -cp ews/webapp/WEB-INF/classes/conf:ews/webapp/WEB-INF/classes/lib/*:ews/webapp/WEB-INF/:ews/webapp/META-INF/:ews/webapp/WEB-INF/lib/*:ews/webapp/WEB-INF/classes/:ews/webapp/WEB-INF/classes/META-INF:/usr/share/java/mysql-connector-java.jar org.apache.ranger.patch.cliutil.DbToSolrMigrationUtil
If the script succeeds, it prints the following details on the screen:- Processing batch 'n' of total 'noOfBatches' (Where each batch contains 10000 rows.)
- Total number of migrated audit logs.
If the script fails to migrate data, it returns the error: Migration process failed, Please refer ranger_db_patch.log file.