Apache ZooKeeper Known Issues

ZooKeeper JMX did not support TLS when managed by Cloudera Manager

The ZooKeeper service optionally exposes a JMX port used for reporting and metrics. By default, Cloudera Manager enables this port, but prior to Cloudera Manager 6.1.0, it did not support mutual TLS authentication on this connection. While JMX has a password-based authentication mechanism that Cloudera Manager enables by default, weaknesses have been found in the authentication mechanism, and Oracle now advises JMX connections to enable mutual TLS authentication in addition to password-based authentication. A successful attack may leak data, cause denial of service, or even allow arbitrary code execution on the Java process that exposes a JMX port. Beginning in Cloudera Manager 6.1.0, it is possible to configure mutual TLS authentication on ZooKeeper’s JMX port.

Products affected: ZooKeeper

Releases affected: Cloudera Manager 6.1.0 and lower, Cloudera Manager 5.16 and lower

Users affected: All

Date/time of detection: June 7, 2018

Severity (Low/Medium/High): 9.8 High (CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H)

Impact: Remote code execution

CVE: CVE-2018-11744

Immediate action required: Upgrade to Cloudera Manager 6.1.0 and enable TLS for the ZooKeeper JMX port by turning on the configuration settings “Enable TLS/SSL for ZooKeeper JMX” and “Enable TLS client authentication for JMX port” on the ZooKeeper service and configuring the appropriate TLS settings. Alternatively, disable the ZooKeeper JMX port via the configuration setting “Enable JMX Agent” on the ZooKeeper service.

Addressed in release/refresh/patch: Cloudera Manager 6.1.0

Adding New ZooKeeper Servers Can Lead to Data Loss

When the number of new ZooKeeper servers exceeds the number that already exist in the ZooKeeper service (for example, if you increase the number of servers from 1 to 3), and a Start command is immediately issued to the ZooKeeper service, the new servers can form a quorum, which causes data loss in existing servers.

Users of the following versions of Cloudera Manager are affected:

5.0.0–5.0.5, 5.1.0–5.1.4, 5.2.0–5.2.4, and 5.3.0–5.3.2

Workaround: If you use a version of Cloudera Manager listed above, upgrade to the next available maintenance release with the bug fix (within the minor version), or to Cloudera Manager 5.4.

The ZooKeeper server cannot be migrated from version 3.4 to 3.3, then back to 3.4, without user intervention.

Upgrading from 3.3 to 3.4 is supported, as is downgrading from 3.4 to 3.3. However, moving from 3.4 to 3.3 and back to 3.4 will fail. 3.4 is checking the datadir for acceptedEpoch and currentEpoch files and comparing these against the snapshot and log files contained in the same directory. These epoch files are new in 3.4.

As a result: 1) Upgrading from 3.3 to 3.4 is fine - the *Epoch files do not exist, and the server creates them. 2) Downgrading from 3.4 to 3.3 is also fine as version 3.3 ignores the *Epoch files. 3) Going from 3.4 to 3.3 then back to 3.4 fails because 3.4 sees invalid *Epoch files in the datadir; 3.3 will have ignored them, applying changes to the snapshot and log files without updating the *Epoch files.

Bug: ZOOKEEPER-1149

Cloudera Bug: CDH-5272

Anticipated Resolution: See workaround

Workaround: Delete the *Epoch files if this situation occurs — the version 3.4 server will recreate them as in case 1) above.