Understanding the purge date used by the purge event
Describes the Cloudera Observability On-Premises purge event's criteria that is based on the file's data group and the data group's retention limit and how the purge date is calculated.
The purge event’s criteria is based on the maximum data retention policy, described in days,
for the following HDFS data groups:
- Temporary data, when the retention period exceeds 8 days
- Staging data, when the retention period exceeds 31 days
- Detailed data, when the retention period exceeds 181 days
- Summarized data, when the retention period exceeds 731 days
The purge date is calculated by subtracting the retention days, specified by the maximum data retention period policy, from the current date and comparing the resultant date with the data’s timestamp date. If the data’s timestamp date is less than or equal to the resultant date the data is removed.
The data’s timestamp date is determined by where the data resides:
- If the data resides in the cloudera-dbus root directory, the timestamp date is extracted from the subdirectory name. For example, if the directory name is /cloudera-dbus/HiveAudit/2021030623. The timestamp date extracted by the purge event is 2021/03/06, using the YYYY/MM/DD date format.
- If the data resides in a cloudera-sigma-olap-impala, cloudera-sigma-partial-pse, cloudera-sigma-pse-extended, or cloudera-sigma-sdx-payloads root directory, the timestamp date is extracted from the file’s last modified time.
Obsolete data can be purged from the following HDFS root directories:
- cloudera-dbus
- cloudera-sigma-olap-impala
- cloudera-sigma-partial-pse
- cloudera-sigma-pse-extended
- cloudera-sigma-sdx-payloads
