The Cloudera Manager upgrade process automatically migrates data from existing databases to the local datastore. The upgrade process occurs only once for Host Monitor and Service Monitor, though it can be spread across multiple runs of Host Monitor and Service Monitor if they are restarted before it completes. Resource usage (CPU, memory, and disk) by Host Monitor and Service Monitor will be higher than normal during the process.

You can monitor the progress of migrating data from a Cloudera Manager 4 database to the Cloudera Manager 5 datastore in the Host Monitor and Service Monitor logs. Log statements starting with LDBTimeSeriesDataMigrationTool identify the upgrade process. The important statements are: Starting DB migration when migration is first started and Migration progress: {} total, {} migrated, {} errors as progress is reported. Progress is reported with partition counts, so it'll be something like 3 total, 0 migrated, 0 errors to start, up to 3 total, 3 migrated, 0 errors at the end.

After migration completes, the migrated data is summarized in statements such as Running the LDBTimeSeriesRollupManager at {}, forMigratedData={} with table names. At this point, the external database will never again be used by Host Monitor and Service Monitor and the database configurations can be removed (connection information, username, password, etc.).

The Service Monitor stores time series data and health data, Impala query metadata, and YARN application metadata.

By default, the data is stored in /var/lib/cloudera-service-monitor/ on the Service Monitor host. This can be changed by modifying the Service Monitor Storage Directory configuration (firehose.storage.base.directory). To change this configuration on an active system, see Moving Monitoring Data on an Active Cluster.

See Data Granularity and Time-Series Metric Data for an explanation of how metric data is stored within Cloudera Manager and for the impact the storage limits have on data retention.

The default values are fairly small, so you should examine disk usage after several days of activity to determine how much space is needed.

The Host Monitor stores time series data and health data.

By default, the data is stored in /var/lib/cloudera-host-monitor/ on the Host Monitor’s host. This can be changed by modifying the Host Monitor Storage Directory configuration (firehose.storage.base.directory). To change this configuration on an active system see Moving Monitoring Data on an Active Cluster.

See the next section for an explanation of how metric data is stored within Cloudera Manager and for the impact these limits have on data retention.

The default value is fairly small so we encourage you to examine disk usage after several days of activity to determine how much space they need.

The Service Monitor and Host Monitor store metric data store time-series metric data in a variety of ways. When the data is first received it is written as is to the metric store. Over time, the raw data is summarized to and stored at various data granularities. For example, after ten minutes a single ten-minute summary point is written containing the average of the metric over the period as well as the minimum, the maximum, the standard deviation, and a variety of other statistics. This process is repeated to produce hourly, six-hourly, daily, and weekly summaries. This data summarization system is only for metric data. Impala query monitoring and YARN application monitoring do not have a similar system. For those systems, when the storage limit is reached, the oldest stored records are deleted.

The Service Monitor and Host Monitor internally manage the amount of their overall storage space to dedicate to each data granularity level. When the limit for a particular level is reached, the oldest data points at that level are deleted. Note that metric data for that time period remains available at the lower granularity levels. That is, when an hourly point for a particular time is deleted to free up space, a daily point still exists covering that hour. Since each of these data granularities consumes significantly less storage than the previous summary level, lower granularity levels can be retained for longer periods of time. In particular, given a reasonable amount of storage, weekly points can normally be retained indefinitely.

Some features, notably detailed display of health results, depend on the presence of raw data. Health history is maintained by the event store dictated by its retention policies.

There are two ways to change where monitoring data is stored on a cluster: basic and advanced.

There are two memory-related configuration options: Java heap size and non-Java memory size. The memory required or recommended for both of these configuration options depends on the size of the cluster. In addition to the memory configured, the Host and Service Monitor will also take advantage of the Linux page cache. Having memory free for use as page cache on the Service and Host Monitor hosts will improve performance.