Key-Value Store Indexer Properties in CDH 5.4.0
Role groups:
lilyhbaseindexerdefaultgroup
Advanced
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve) | For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration. | HBASE_INDEXER_role_env_safety_valve | false | ||
Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml | For advanced use only, a string to be inserted into hbase-indexer-site.xml for this role only. | hbase_indexer_config_safety_valve | false | ||
Java Configuration Options for Lily HBase Indexer | These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here. | -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled | hbase_indexer_java_opts | false | |
System Group | The group that the HBase Indexer process should run as. | hbase | hbase_indexer_process_groupname | true | |
System User | The user that the HBase Indexer process should run as. | hbase | hbase_indexer_process_username | true | |
Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve) | For advanced use only, a string to be inserted into log4j.properties for this role only. | log4j_safety_valve | false | ||
Heap Dump Directory | Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role. | oom_heap_dump_dir | /tmp | oom_heap_dump_dir | false |
Dump Heap When Out of Memory | When set, generates heap dump file when java.lang.OutOfMemoryError is thrown. | false | oom_heap_dump_enabled | true | |
Kill When Out of Memory | When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown. | true | oom_sigkill_enabled | true | |
Automatically Restart Process | When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure. | false | process_auto_restart | true |
Logs
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
HBase Indexer Log Directory | Directory where HBase Indexer will place its log files. | /var/log/hbase-solr | hbase_indexer_log_dir | true | |
Lily HBase Indexer Logging Threshold | The minimum log level for Lily HBase Indexer logs | INFO | log_threshold | false | |
Lily HBase Indexer Maximum Log File Backups | The maximum number of rolled log files to keep for Lily HBase Indexer logs. Typically used by log4j or logback. | 10 | max_log_backup_index | false | |
Lily HBase Indexer Max Log Size | The maximum size, in megabytes, per log file for Lily HBase Indexer logs. Typically used by log4j or logback. | 200 MiB | max_log_size | false |
Monitoring
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Enable Health Alerts for this Role | When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold | true | enable_alerts | false | |
Enable Configuration Change Alerts | When set, Cloudera Manager will send alerts when this entity's configuration changes. | false | enable_config_alerts | false | |
File Descriptor Monitoring Thresholds | The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit. | Warning: 50.0 %, Critical: 70.0 % | hbase_indexer_fd_thresholds | false | |
Lily HBase Indexer Host Health Test | When computing the overall Lily HBase Indexer health, consider the host's health. | true | hbase_indexer_host_health_enabled | false | |
Lily HBase Indexer Process Health Test | Enables the health test that the Lily HBase Indexer's process state is consistent with the role configuration | true | hbase_indexer_scm_health_enabled | false | |
Heap Dump Directory Free Space Monitoring Absolute Thresholds | The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. | Warning: 10 GiB, Critical: 5 GiB | heap_dump_directory_free_space_absolute_thresholds | false | |
Heap Dump Directory Free Space Monitoring Percentage Thresholds | The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured. | Warning: Never, Critical: Never | heap_dump_directory_free_space_percentage_thresholds | false | |
Log Directory Free Space Monitoring Absolute Thresholds | The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. | Warning: 10 GiB, Critical: 5 GiB | log_directory_free_space_absolute_thresholds | false | |
Log Directory Free Space Monitoring Percentage Thresholds | The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured. | Warning: Never, Critical: Never | log_directory_free_space_percentage_thresholds | false | |
Process Swap Memory Thresholds | The health test thresholds on the swap memory usage of the process. | Warning: Any, Critical: Never | process_swap_memory_thresholds | false | |
Role Triggers | The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health
system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has all of the following fields:
|
[] | role_triggers | true | |
Unexpected Exits Thresholds | The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role. | Warning: Never, Critical: Any | unexpected_exits_thresholds | false | |
Unexpected Exits Monitoring Period | The period to review when computing unexpected exits. | 5 minute(s) | unexpected_exits_window | false |
Performance
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Maximum Process File Descriptors | If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value. | rlimit_fds | false |
Ports and Addresses
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
HBase Indexer HTTP Port | HTTP port used by HBase Indexer. | hbaseindexer.http.port | 11060 | hbase_indexer_http_port | true |
Resource Management
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Java Heap Size of Lily HBase Indexer in Bytes | Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx. | 1 GiB | hbase_indexer_java_heapsize | false | |
Cgroup CPU Shares | Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager. | cpu.shares | 1024 | rm_cpu_shares | true |
Cgroup I/O Weight | Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager. | blkio.weight | 500 | rm_io_weight | true |
Cgroup Memory Hard Limit | Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit. | memory.limit_in_bytes | -1 MiB | rm_memory_hard_limit | true |
Cgroup Memory Soft Limit | Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit. | memory.soft_limit_in_bytes | -1 MiB | rm_memory_soft_limit | true |
Security
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Role-Specific Kerberos Principal | Kerberos principal used by the Lily HBase Indexer roles. | hbase | kerberos_role_princ_name | true |
Stacks Collection
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Stacks Collection Data Retention | The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted. | stacks_collection_data_retention | 100 MiB | stacks_collection_data_retention | false |
Stacks Collection Directory | The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory. | stacks_collection_directory | stacks_collection_directory | false | |
Stacks Collection Enabled | Whether or not periodic stacks collection is enabled. | stacks_collection_enabled | false | stacks_collection_enabled | true |
Stacks Collection Frequency | The frequency with which stacks are collected. | stacks_collection_frequency | 5.0 second(s) | stacks_collection_frequency | false |
Stacks Collection Method | The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped. | stacks_collection_method | jstack | stacks_collection_method | false |
service_wide
Advanced
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml | For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration. | keystore_indexer_sentry_safety_valve | false | ||
Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve) | For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration. | ks_indexer_env_safety_valve | false |
Monitoring
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Enable Service Level Health Alerts | When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold | true | enable_alerts | false | |
Enable Configuration Change Alerts | When set, Cloudera Manager will send alerts when this entity's configuration changes. | false | enable_config_alerts | false | |
Healthy Lily HBase Indexer Monitoring Thresholds | The health test thresholds of the overall Lily HBase Indexer health. The check returns "Concerning" health if the percentage of "Healthy" Lily HBase Indexers falls below the warning threshold. The check is unhealthy if the total percentage of "Healthy" and "Concerning" Lily HBase Indexers falls below the critical threshold. | Warning: 95.0 %, Critical: 90.0 % | ks_indexer_indexers_healthy_thresholds | false | |
Service Triggers | The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the
health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has all of the following
fields:
|
[] | service_triggers | true | |
Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) | For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones. | smon_derived_configs_safety_valve | false |
Morphlines
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Custom Mime-types File | Text that goes verbatim into custom-mimetypes.xml file used by HBase Indexers. | <?xml version=1.0 encoding=UTF-8?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <mime-info> <mime-type type=text/space-separated-values> <glob pattern=*.ssv/> </mime-type> <mime-type type=avro/binary> <magic priority=50> <match value=0x4f626a01 type=string offset=0/> </magic> <glob pattern=*.avro/> </mime-type> <mime-type type=mytwittertest/json+delimited+length> <magic priority=50> <match value=[0-9]+(\r)?\n\\" type=regex offset=0:16/> </magic> </mime-type> <mime-type type=application/hadoop-sequence-file> <magic priority=50> <match value=SEQ[\0-\6] type=regex offset=0/> </magic> </mime-type> </mime-info> | custom_mimetypes_file | false | |
Grok Dictionary File | Text that goes verbatim into grok-dictionary.conf file used by HBase Indexers. | USERNAME [a-zA-Z0-9._-]+ USER %USERNAME INT (?:[+-]?(?:[0-9]+)) BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))) NUMBER (?:%BASE10NUM) BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+)) BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b POSINT \b(?:[1-9][0-9]*)\b NONNEGINT \b(?:[0-9]+)\b WORD \b\w+\b NOTSPACE \S+ SPACE \s* DATA .*? GREEDYDATA .* #QUOTEDSTRING (?:(?<!\\)(?:(?:\\.|[^\\])*|(?:'(?:\\.|[^\\'])*')|(?:`(?:\\.|[^\\`])*`))) QUOTEDSTRING (?>(?<!\\)(?>(?>\\.|[^\\]+)+||(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``)) UUID [A-Fa-f0-9]8-(?:[A-Fa-f0-9]4-)3[A-Fa-f0-9]12 # Networking MAC (?:%CISCOMAC|%WINDOWSMAC|%COMMONMAC) CISCOMAC (?:(?:[A-Fa-f0-9]4\.)2[A-Fa-f0-9]4) WINDOWSMAC (?:(?:[A-Fa-f0-9]2-)5[A-Fa-f0-9]2) COMMONMAC (?:(?:[A-Fa-f0-9]2:)5[A-Fa-f0-9]2) IP (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]1, 2)[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]1, 2)[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]1, 2)[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]1, 2))(?![0-9]) HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]0, 62)(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]0, 62))*(\.?|\b) HOST %HOSTNAME IPORHOST (?:%HOSTNAME|%IP) #HOSTPORT (?:%IPORHOST=~/\./:%POSINT) # WH # paths PATH (?:%UNIXPATH|%WINPATH) UNIXPATH (?>/(?>[\w_%!$@:., -]+|\\.)*)+ #UNIXPATH (?<![\w\/])(?:/[^\/\s?*]*)+ LINUXTTY (?>/dev/pts/%NONNEGINT) BSDTTY (?>/dev/tty[pq][a-z0-9]) TTY (?:%BSDTTY|%LINUXTTY) WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+ URIPROTO [A-Za-z]+(\+[A-Za-z+]+)? URIHOST %IPORHOST(?::%POSINT:port)? # uripath comes loosely from RFC1738, but mostly from what Firefox # doesn't turn into %XX URIPATH (?:/[A-Za-z0-9$.+!*'(), ~:;=#%_\-]*)+ #URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)? URIPARAM \?[A-Za-z0-9$.+!*'|(), ~#%&/=:;_?\-\[\]]* URIPATHPARAM %URIPATH(?:%URIPARAM)? URI %URIPROTO://(?:%USER(?::[^@]*)?@)?(?:%URIHOST)?(?:%URIPATHPARAM)? # Months: January, Feb, 3, 03, 12, December MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b MONTHNUM (?:0?[1-9]|1[0-2]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) # Days: Monday, Tue, Thu, etc... DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?) # Years? YEAR (?>\d\d)1, 2 # Time: HH:MM:SS #TIME \d2:\d2(?::\d2(?:\.\d+)?)? # I'm still on the fence about using grok to perform the time match, # since it's probably slower. # TIME %POSINT<24:%POSINT<60(?::%POSINT<60(?:\.%POSINT)?)? HOUR (?:2[0123]|[01]?[0-9]) MINUTE (?:[0-5][0-9]) # '60' is a leap second in most time standards and thus is valid. SECOND (?:(?:[0-5][0-9]|60)(?:[:., ][0-9]+)?) TIME (?!<[0-9])%HOUR:%MINUTE(?::%SECOND)(?![0-9]) # datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it) DATE_US %MONTHNUM[/-]%MONTHDAY[/-]%YEAR DATE_EU %MONTHDAY[./-]%MONTHNUM[./-]%YEAR ISO8601_TIMEZONE (?:Z|[+-]%HOUR(?::?%MINUTE)) ISO8601_SECOND (?:%SECOND|60) TIMESTAMP_ISO8601 %YEAR-%MONTHNUM-%MONTHDAY[T ]%HOUR:?%MINUTE(?::?%SECOND)?%ISO8601_TIMEZONE? DATE %DATE_US|%DATE_EU DATESTAMP %DATE[- ]%TIME TZ (?:[PMCE][SD]T) DATESTAMP_RFC822 %DAY %MONTH %MONTHDAY %YEAR %TIME %TZ DATESTAMP_OTHER %DAY %MONTH %MONTHDAY %TIME %TZ %YEAR # Syslog Dates: Month Day HH:MM:SS SYSLOGTIMESTAMP %MONTH +%MONTHDAY %TIME PROG (?:[\w._/%-]+) SYSLOGPROG %PROG:program(?:\[%POSINT:pid\])? SYSLOGHOST %IPORHOST SYSLOGFACILITY <%NONNEGINT:facility.%NONNEGINT:priority> HTTPDATE %MONTHDAY/%MONTH/%YEAR:%TIME %INT # Shortcuts QS %QUOTEDSTRING # Log formats SYSLOGBASE %SYSLOGTIMESTAMP:timestamp (?:%SYSLOGFACILITY )?%SYSLOGHOST:logsource %SYSLOGPROG: COMBINEDAPACHELOG %IPORHOST:clientip %USER:ident %USER:auth \[%HTTPDATE:timestamp\] (?:%WORD:verb %NOTSPACE:request(?: HTTP/%NUMBER:httpversion)?|%DATA:rawrequest) %NUMBER:response (?:%NUMBER:bytes|-) %QS:referrer %QS:agent # Log Levels LOGLEVEL ([T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?) | grok_dictionary_conf_file | false | |
Morphlines File | Text that goes into morphlines.conf file used by HBase Indexers. The text goes verbatim into the config file except that $ZK_HOST is replaced by the ZooKeeper quorum of the Solr service. | SOLR_LOCATOR : # Name of solr collection collection : collection # ZooKeeper ensemble zkHost : $ZK_HOST morphlines : [ id : morphline importCommands : [org.kitesdk.**, com.ngdata.**] commands : [ extractHBaseCells mappings : [ inputColumn : data:* outputField : data type : string source : value ] logDebug format : output record: , args : [@] ] ] | morphlines_conf_file | false |
Other
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
HBase Service | Name of the HBase service that this Key-Value Store Indexer service instance depends on | hbase_service | true | ||
Solr Service | Name of the Solr service that this Key-Value Store Indexer service instance depends on | solr_service | true |
Policy File Based Sentry
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
Sentry Global Policy File | HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format. | sentry.hbaseindexer.provider.resource | /user/hbaseindexer/sentry/sentry-provider.ini | keystore_indexer_provider_resource | false |
Enable Sentry Authorization using Policy Files | Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. Sentry is supported only on CDH 5.4 or later deployments and requires authentication to be turned on for HBase Indexer. | false | keystore_indexer_sentry_enabled | false | |
Sentry User to Group Mapping Class | The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop groups or local groups defined in the policy file. When configured with Hadoop groups, Sentry will ask the HDFS Namenode for group mapping for a given user to determine authorization access. | sentry.provider | org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider | keystore_indexer_sentry_provider | false |
Security
Display Name | Description | Related Name | Default Value | API Name | Required |
---|---|---|---|---|---|
HBase Indexer Secure Authentication | Authentication mechanism used by HBase Indexer. | hbaseindexer.authentication.type | simple | hbase_indexer_security_authentication | false |
HBase Indexer TLS/SSL Certificate Trust Store File | The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HBase Indexer might connect to. This is used when HBase Indexer is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) being connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead. | keystore_indexer_truststore_file | false | ||
HBase Indexer TLS/SSL Certificate Trust Store Password | The password for the HBase Indexer TLS/SSL Certificate Trust Store File. Note that this password is not required to access the trust store: this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information. | keystore_indexer_truststore_password | false |