Ambari Dataplane Profiler Configs

Provides various cluster configurations, including: atlasUrl

rangerAuditDir

metastoreUrl

metastoreKeytab

metastorePrincipal

The format is a standard CRON expression.

This will periodically refresh the metrics cache.

Specifies the configuration required for interactive Livy sessions the profiler creates.

These sessions will be swapped with new ones based on their lifetime. Lifetime of session is decided by the configurations below.

session.lifetime.minutes - Session lifetime in minutes after its creation before it will be swapped.

session.lifetime.requests - Maximum number of requests a session can process before it will swapped.

session.max.errors - Number of adjacent errors after which session will be swapped

There are two separate session.config sections describing interactive session’s Spark configurations. Both read and write has same schema and following Livy session properties can be specified here.

name,heartbeatTimeoutInSecond,driverMemory,driverCores,executorMemory,executorCores,numExecutors,queue

For more on above properties refer to Livy documentation.

The properties session.config.read.timeoutInSeconds and session.config.write.timeoutInSeconds specifies timeouts for requests using interactive session.

Notes:

session.starting.message and session.dead.message are for internal use.

	Important
	Do not modify.

It is advisable to have a separate YARN queue for sessions created by the profiler.

session {
        	lifetime {
                    	minutes = 2880
                    	requests = 500
                    	}
        	max.errors = 20
        	starting.message = "java.lang.IllegalStateException: Session is in state starting"
        	dead.message = "java.lang.IllegalStateException: Session is in state dead"
        	config {
                    	read {
                                	name = "dpprofiler-read"                                                                                                   	heartbeatTimeoutInSecond = 172800
                                	timeoutInSeconds = 90
                                	driverMemory = "5G"
                                	driverCores = 4
                                	executorMemory = "4G"
                                	executorCores = 2
                                	numExecutors = 25
                                	queue = "profilerqueue"
                    	  	}
                    	write {
                                	name = "dpprofiler-write"
                                	heartbeatTimeoutInSecond = 172800
                                	timeoutInSeconds = 90
                                	driverMemory = "2G"
                                	driverCores = 2
                                	executorMemory = "1G"
                                	executorCores = 1
                                	numExecutors = 4
                                	queue = "profilerqueue"
                    	 	}
        	     	}
	}

Data directory. If using h2, data is stored here.

dpprofiler.http.port

dpprofiler${principalSuffix}@REALM.COM

principalSuffix is a random string which is generated by Ambari for a cluster. This string is used to uniquely identify services on a cluster in case of multiple clusters being managed by single KDC

FQDN - fully qualified domain name of the machine

<configuration>

<conversionRule conversionWord="coloredLevel" converterClass="play.api.libs.logback.ColoredLevel" />

<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>{{dpprofiler_log_dir}}/application.log</file>
<encoder>
<pattern>%date [%level] from %logger in %thread - %message%n%xException</pattern>
</encoder>
</appender>

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%coloredLevel %logger{15} - %message%n%xException{10}</pattern>
</encoder>
</appender>

<appender name="ASYNCFILE" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
</appender>

<appender name="ASYNCSTDOUT" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="STDOUT" />
</appender>

<logger name="play" level="INFO" />
<logger name="application" level="DEBUG" />

<!-- Off these ones as they are annoying, and anyway we manage configuration ourselves -->
<logger name="com.avaje.ebean.config.PropertyMapLoader" level="OFF" />
<logger name="com.avaje.ebeaninternal.server.core.XmlConfigLoader" level="OFF" />
<logger name="com.avaje.ebeaninternal.server.lib.BackgroundThread" level="OFF" />
<logger name="com.gargoylesoftware.htmlunit.javascript" level="OFF" />

<root level="WARN">
<appender-ref ref="ASYNCFILE" />
<appender-ref ref="ASYNCSTDOUT" />
</root>

</configuration>