Release Notes

2.4.2

Cloudbreak 2.4.2 is a general availability release, which is suitable for production deployments.

New Features

Image Catalog Update

The image catalog has been updated to include new base and prewarmed images for HDP and Ambari. Refer to Image Catalog Updates.

New UI/UX

Cloudbreak 2.4.0 introduces a new user interface.

All major options are now easily accessible from the collapsible navigation menu. All UI options and wizards have been redesigned in order to make cluster creation and management more intuitive.

The following screenshot shows the cluster dashboard:

The following screenshot shows cluster details page:

Note that autoscaling is available from the Autoscaling tab.

Note the Show CLI Command available from Actions menu. This allows you to generate a CLI template of the cluster. A similar option is available on the last page of the create cluster wizard.

The following screenshot shows the create cluster wizard:

Note the Basic/Advanced toggle button, which allows you to switch between the basic and advanced wizard view. In the Settings you can select which view you would like to see by default.

New CLI

Cloudbreak 2.4.0 introduces the new CLI tool, which replaces Cloudbreak Shell. All commands start with a singular object followed by an action, for example, blueprint create and blueprint list. To download the CLI, select Download CLI from the navigation pane. Refer to Install CLI, Get Started with the CLI, and CLI Reference.

Support for Kerberos

Creating Kerberos-enabled clusters is supported:

Refer to Enabling Kerberos Security.

Support for Configuring an External RDBMS for Cloudbreak

Using an external RDBMS for Cloudbreak is supported and recommended for production environments.
For configuration instructions, refer to Configuring External Cloudbreak Database.

Support for Migrating Cloudbreak Instance

Migrating Cloudbreak from one machine to another is supported.
For migration instructions, refer to Moving a Cloudbreak Instance.

Prewarmed Images

To accelerate cluster creation, CLoudbreak 2.4.0 introduces prewarmed images, which include the operating system, as well as the default version of Ambari and HDP. By default, Cloudbreak 2.4 launches clusters from these prewarmed images, instead of using base images (which were used by default in earlier versions of Cloudbreak). Default base images are still available in case you would like to use different Ambari and HDP versions than those provided with prewarmed images.
For more information, refer to Prewarmed and Base Images.

Providing Your Own JDK

Providing your own JDK on a custom base image is supported. For instructions, refer to "Advanced topics" in the https://github.com/hortonworks/cloudbreak-images repository.

Generating CLI Templates

After specifying the parameters for your cluster in the Cloudbreak web UI, you can copy the content of the CLI JSON file that can be used to create a cluster via Cloudbreak CLI. For more information, refer to Obtain Cluster JSON Template from the UI.

Furthermore, Cloudbreak web UI includes an option in the UI which allows you to generate the create command for resources such as credentials, blueprints, clusters, and recipes. For more information, refer to Obtain CLI Command from the UI.

New Recipe Types

New types of recipes are introduced:

Refer to updated Recipes documentation.

Disabling Cloud Providers

You can hide cloud providers available in Cloudbreak by adding the CB_ENABLEDPLATFORMS environment variable in Profile and setting it to the provider(s) that you would like to have available. For more information, refer to Disable Providers.

Support for Ambari 2.6 (2.6.1.3+)

Cloudbreak 2.4.0 introduces support for Ambari 2.6 and uses Ambari 2.6.1.3 by default. If you would like to use Ambari 2.6, you must use Ambari version 2.6.1.3 or newer.

If you decide to use Ambari 2.6.1, you should be aware of the following:

Ambari Master Key

Cloudbreak 2.4.0 allows you to specify the Ambari Master Key. The Ambari Server Master Key is used to configure Ambari to encrypt database and Kerberos credentials that are retained by Ambari as part of the Ambari setup. The option is available on the Security page of the create cluster wizard.

Behavioral Changes

Custom Images

The functionality which enables you to create custom images was changed and improved. Refer to Custom Images.

Prewarmed Images Are Used By Default

By default, Cloudbreak 2.4.0 launches clusters from prewarmed images, instead of base images (which were used by default in earlier versions of Cloudbreak). For more information, refer to Prewardmed and Base Images.

Removal of Cloudbreak Shell

Cloudbreak Shell is not available in Cloudbreak 2.x. It was replaced by the Cloudbreak CLI.

Removal of Platforms

The Platforms feature was removed and is not available in Cloudbreak 2.x.

Removal of Mesos

Mesos cloud provider is not supported and the corresponding options were removed in Cloudbreak 2.x.

Removal of Templates

Earlier versions of Cloudbreak allowed you to save infrastructure, network, and security group templates. This feature was removed. Instead, you can define VMs, storage, networks, and security groups as part of the create cluster wizard.

cbd update Improvements

The following improvements were added for cbd update:

(BUG-94801) New Default Images Including OS with Spectre and Meltdown Patches

All default images used by Cloudbreak were rebuilt. Each image now includes an updated version of the operating system with the following CPU security fixes:

Image Catalog Updates

June 1, 2018

Update 1 for Cloudbreak 2.4.2:

Default Ambari version 2.6.2.0
Default HDP version 2.6.5.0-292

April 3, 2018

The following versions are included with Cloudbreak 2.4.1:

Default Ambari version 2.6.1.3
Default HDP version 2.6.4.5-2

Fixed Issues

Issue Issue Description Category Fix Version
BUG-103678 Remove instanceProfileStrategy recommendation from CLI. Stability 2.4.2
BUG-101805 OrchestratorBootstrapRunner keeps polling after cluster termination: If a cluster is terminated and a salt job is in progress, then the polling continues until timeout. Stability 2.4.2
BUG-100754 Sync should cleanup the terminated instances' salt metadata from the cluster. Stability 2.4.2
BUG-100597 Unable to create Cloudbreak autoscaling policy. Stability 2.4.2
BUG-100437 Failing psql commands return code 0, so the cluster install is not failing when it should. Stability 2.4.2
BUG-100392 Switch to HTTPS SmartSense gateway for all cloudbreak deployments Stability 2.4.2
BUG-100311 If the cluster creation template is not correct or a blueprint has some issues then cluster creation template is not logged. Stability 2.4.2
BUG-99779 Create clusters with custom hdp and ambari versions does not work. Stability 2.4.2
BUG-99581 Cloudbreak cluster UI indicates that manual recovery is needed for a healthy node. Stability 2.4.2
BUG-99322 @#$%|:&*; blueprint name throws 'duplicate key value violates unique constraint' error. Stability 2.4.2
BUG-98792 Pre-ambari-start recipe does not wait for agent on server node. Stability 2.4.2
BUG-92878 Cannot read imagecatalog from gist raw view. (because image catalog must have content-type json, but in case of gist, it's text). Stability 2.4.2
BUG-99168 All clusters created on Google Cloud Platform fail. Stability 2.4.1
BUG-99400 Time-based cluster autoscaling does not work. Stability 2.4.1
BUG-99505 Sync is not working for an AWS instance that was terminated a long time ago. Stability 2.4.1
BUG-98277 Network interface handling in CloudBreak should be improved. Stability 2.4.1
BUG-97395 Networks are duplicated on networks tab of the cluster create wizard. Stability 2.4.1
BUG-97259 "Update failed" status after downscale failed, even though cluster was not modified and its status should be "Running". Stability 2.4.1
BUG-97207 Changing lifecycle management on YARN causes NPE. Stability 2.4.1
BUG-99189 ImageCatalog PUT endpoint is not secured. Security 2.4.1
BUG-97895 LDAP password should be removed from Cloudbreak logs. Security 2.4.1
BUG-97300 Cloudbreak should show proper error messages when the given credential is not valid anymore. Usability 2.4.1
BUG-97296 GCP credential creation should validate whether resources are available with the credential. Usability 2.4.1
BUG-99635 After deleting a default credential and creating a new credential, credential is missing from the create cluster wizard. Usability 2.4.1
BUG-97660 Ignore repository warnings checkbox are missing after changing base image Ambari or HDP to a custom one. Usability 2.4.1
BUG-97307 Ignore repository warnings checkbox is not selectable after change the HDP VDF URL. Usability 2.4.1
BUG-96764 "Failed to remove instance" error when using the delete icon. Usability 2.4.1
BUG-97390 Cloudbreak should support longer resource ID-s on AWS. Usability 2.4.1
BUG-99512 Azure ES_v3 instances should support premium storage. Usability 2.4.1
BUG-97206 Backend should return only images for enabled platforms. Usability 2.4.1

Known Issues

Known Issues: Cloudbreak

(BUG-96788) Azure Availability Set Option Is Not Available for Instance Count of 1

When creating a cluster, the Azure availability set feature is not available for host groups with the instance count of 1.

Workaround:

This issue will be fixed in a future release.

If you would like to use the Azure availability sets feature now, you must add at least 2 instances to the host group for which you want to use them. The option Azure availability sets is available on the advanced Hardware and Storage page of the create cluster wizard.

(BUG-92605) Cluster Creation Fails with ResourceInError

Cluster creation fails with the following error:

Infrastructure creation failed. Reason: Failed to create the stack for CloudContext{id=3689, name='test-exisitngnetwork', platform='StringType{value='OPENSTACK'}', owner='e0307f96-bd7d-4641-8c8f-b95f2667d9c6'} due to: Resource CREATE failed: ResourceInError: resources.ambari_volume_master_0_0: Went to status error due to "Unknown"

Workaround:

This may mean that the volumes that you requested exceed volumes available on your cloud provider account. When creating a cluster, on the advanced Hardware and Storage page of the create cluster wizard, try reducing the amount of requested storage. If you need more storage, try using a different region or ask your cloud provider admin to increase the resource quota for volumes.

(BUG-91013) Incorrect Node Status After Cluster Restart

You may sporadically experience an issue where after you stop and restart a cluster, the node status displayed in the "Hardware" section is incorrect.

(BUG-93241) Error When Scaling Multiple Host Groups

Scaling of multiple host groups fails with the following error:

Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; nested exception is org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1

Workaround:

Scaling multiple host groups at once is not supported. If you would like to scale multiple host groups: scale the first host group and wait until scaling has completed, then scale the second host group, and so on.

(BUG-97044) Show CLI Command Copy JSON Button Does Not Work

When using the Show CLI Command > Copy the JSON or Copy the Command button with Firefox, the content does not does not get copied if adblock plugin or other advertise blocker plugins are present.

Workaround:

Use a browser without an adblock plugin.

(BUG-95607) Special Characters in Blueprint Name Cause an Error in CLI

When registering a blueprint via blueprint create CLI command, if the name of the blueprint includes one or more of the following special characters @#$%|:&*; you will get an error similar to: 

cb blueprint create from-url --name test@# --url https://myurl.com/myblueprint.bp  
[1] 7547
-bash: application.yml: command not found
-bash: --url: command not found
 ~  integration-test  1  time="2018-02-01T12:56:44+01:00" level="error" msg="the following parameters are missing: url\n"

Workaround:
When specifying a bleuprint name that contains special characters, make sure to use quotes, for example "test@#".

(BUG-93257) Clusters Are Missing From History

After changing the dates on the History page multiple times, the results displayed may sometimes be incorrect.

Workaround:

Refresh the page if you think that the history displayed may be incorrect.

Known Issues: Ambari 2.6.1.3 and HDP 2.6.4.0

The known issues described here were discovered when testing Cloubdreak with Ambari 2.6.1.3 and HDP 2.6.4.0, which are used by default in Cloudbreak 2.4.0.

For general Ambari 2.6.1.3 and HDP 2.6.4.0 known issues, refer to:
Ambari 2.6.1.5 Release Notes
HDP 2.6.4.0 Release Notes

(AMBARI-14149) Ambari Cluster Cannot Be Started After Stop

When using Ambari version 2.5.0.3, after stopping and starting a cluster, Event History shows the following error:

Ambari cluster could not be started. Reason: Failed to start Hadoop services.
2/7/2018, 12:47:05 PM
Starting Ambari services.
2/7/2018, 12:47:04 PM
Manual recovery is needed for the following failed nodes:   
[host-10-0-0-4.openstacklocal, host-10-0-0-3.openstacklocal, host-10-0-0-5.openstacklocal

Ambari dashboard shows that nodes are not sending heartbeats.

Workaround:

This issue is fixed in Ambari version 2.5.1.0 and newer.

(BUG-96784) Hive LLAP Start Takes More Than an Hour

Hive LLAP start times out in Ambari, but eventually it starts (after 15 minutes on AWS and after one hour or so on other providers).

(BUG-96707) Druid Overload Does Not Start

Druid overload start fails with the following error when using Ambari 2.6.1.3 and HDP 2.6.4.0:

ERROR [main] io.druid.cli.CliOverlord - Error when starting up. Failing. com.google.inject.ProvisionException: Unable to provision

(BUG-97080) Ambari Fils In Some Cases When an mpack is Installed

If we set the following properties then cluster install may fail (in 20-30% of the cases), because of the Ambari agent cache being updated concurrently:

/etc/ambari-server/conf/ambari.properties  
agent.auto.cache.update=true*  
*/etc/ambari-agent/conf/ambari-agent.ini  
parallel_execution=1