Operating System Requirements
This topic describes the operating system requirements for Cloudera software.
CDP Private Cloud Base Supported Operating Systems
Operating System | Version (bold=new) |
---|---|
RHEL/CentOS/Oracle |
7.6, 7.7, 7.8, 7.9 |
SLES |
12 SP5 Not supported when using the Using the Trial Installer
( |
Ubuntu |
18.04 Not supported for Cloudera Schema Registry |
IBM PowerPC on RHEL | Supported on RHEL 7.6, 7.7. The following
components are not supported:
|
Debian | Not Supported |
Operating System | Version (bold=new) |
---|---|
RHEL/CentOS/Oracle |
7.6, 7.7, 7.8, 7.9 |
Ubuntu |
18.04 Not supported for Cloudera Schema Registry |
Debian | Not Supported |
Operating System | Version |
---|---|
RHEL/CentOS/Oracle |
7.6, 7.7, 7.8, 7.9 |
Debian | Not Supported |
Operating System and IBM PowerPC support matrix
- IBM PowerPC only and CDP Private Cloud Base
- IBM PowerPC CPU, IBM Spectrum Scale Storage, and CDP Private Cloud Base. This is a
subset of what is supported generally on IBM PowerPC.
IBM PowerPC Support Documentation PowerPC 8 and 9 generally without Spectrum Scale Storage https://www.ibm.com/docs/en/linux-on-systems?topic=lpo-supported-linux-distributions-virtualization-options-power8-power9-linux-power-systems PowerPC 10 generally without Spectrum Scale Storage https://www.ibm.com/docs/en/linux-on-systems?topic=lpo-supported-linux-distributions-virtualization-options-power10-linux-power-servers IBM Spectrum Scale Storage with CDP Private Cloud Base on x86 and PowerPC combinations https://www.ibm.com/docs/en/spectrum-scale-bda?topic=requirements-support-matrix
Runtime and Cloudera Manager Supported Operating Systems
Runtime provides parcels for select versions of RHEL-compatible operating systems.
Software Dependencies
- Python - CDP Private Cloud Base, with the exception of Hue,
is supported on the Python version that is included in the operating system by default, as
well as higher versions, but is not compatible with Python 3.0 or higher.
For example, CDP Private Cloud Base requires Python 2.7 or higher on RHEL 7 compatible operating systems.
Spark 2 requires Python 2.7 or higher. If the right level of Python is not picked up by default, set the
PYSPARK_PYTHON
andPYSPARK_DRIVER_PYTHON
environment variables to point to the correct Python executable before running the pyspark command.Python 3 is not supported.
- Perl - Cloudera Manager requires perl.
- python-psycopg2 - Cloudera Manager 7 has a dependency on the
package
python-psycopg2
. Hue in Runtime 7 requires a higher version ofpsycopg2
than is required by the Cloudera Manager dependency. For more information, see Installing thepsycopg2
Python Package. - iproute package - CDP Private Cloud Base has a dependency
on the
iproute
package. Any host that runs the Cloudera Manager Agent requires the package. The required version varies depending on the operating system:Table 4. iproute package Operating System iproute version RHEL iproute Ubuntu iproute2
Filesystem Requirements
Supported Filesystems
The Hadoop Distributed File System (HDFS) is designed to run on top of an underlying filesystem in an operating system. Cloudera recommends that you use either of the following filesystems tested on the supported operating systems:
- ext3: This is the most tested underlying filesystem for HDFS.
- ext4: This scalable extension of ext3 is supported in more recent Linux releases.
- XFS: This is the default filesystem in RHEL 7.
- S3: Amazon Simple Storage Service
Kudu Filesystem Requirements - Kudu is supported on ext4 and
XFS. Kudu requires a kernel version and filesystem that supports hole
punching. Hole punching is the use of the
fallocate(2)
system call with the
FALLOC_FL_PUNCH_HOLE
option set.
File Access Time
Linux filesystems keep metadata that record when each file was
accessed. This means that even reads result in a write to the disk. To
speed up file reads, Cloudera recommends that you disable this option,
called atime
, using the noatime
mount
option in /etc/fstab
:
/dev/sdb1 /data1 ext4 defaults,noatime 0
Apply the change without rebooting:
mount -o remount /data1
Filesystem Mount Options
The filesystem mount
options have a
sync
option that allows you to write
synchronously.
Using the sync
filesystem mount option reduces
performance for services that write data to disks, such as HDFS, YARN,
Kafka and Kudu. In CDH, most writes are already replicated. Therefore,
synchronous writes to disk are unnecessary, expensive, and do not
measurably improve stability.
NFS and NAS options are not supported for use as DataNode Data Directory mounts, even when using Hierarchical Storage features.
Mounting /tmp
as a filesystem with the
noexec
option is sometimes done as an enhanced
security measure to prevent the execution of files stored there.
However, this causes multiple problems with various parts of Cloudera
Manager and CDH. Therefore, Cloudera does not support mounting
/tmp
with the noexec
option.
Filesystem Requirements
You can control resource allocation for Cloudera Manager and CDP Runtime services
(nproc
, nofile
, etc) from
/etc/security/limits.conf, and through init
scripts on traditional SysV Init systems. However, on systems using systemd
the limits either needs to be set in the service’s unit file, or in
/etc/systemd/system.conf, or in files present under
/etc/systemd/system.conf.d/*. This is due to a known limitation with
systemd
as it does not use PAM login sessions
(pam_limits.so
) for daemon services, thereby ignoring the limits defined
in /etc/security/limits.conf. Both Cloudera Manager Agent and
Supervisord
(responsible for starting CDP Runtime services) are
daemonised during system initialisation.
- For system-wide change, uncomment the process properties from
/etc/systemd/system.conf, or create an
override.conf
under /etc/systemd/system.conf.d/. This requires a *nix system reboot for the changes to take effect. For more information, see Limits.conf. - To apply custom limits on CDP Runtime services, add the required process properties to
the [Service] section in
/usr/lib/systemd/system/cloudera-scm-supervisord.service.For instance, to customise the number of child processes a process can fork. You can set the property as follows:
LimitNPROC=<value>
Then reload the configuration by running the following command for the limits to be applied in the subsequent service restarts:# systemctl daemon-reload
Here are the list of available process properties.
nscd for Kudu
Although not a strict requirement, it's highly recommended that you
usenscd
to cache both DNS name resolution and static
name resolution for Kudu.