Defining External Database Servers

Cloudera Director needs information about external database servers before it can use them. This section describes defining database server templates and using Amazon Relational Database Service (RDS) to create new database servers..

The Database Server Template

A database server template can refer to either an existing database server or a server to be created. The following are the basic elements of a database server template:
  • name - A unique name for the server within the environment
  • type - The type of database server, such as “MYSQL” or “POSTGRESQL”
  • hostname - The name of the server host
  • port - The listening port of the server
  • username - The name of the administrative account for the server
  • password - The password for the administrative account

The hostname and port are optional in a template. If they are not present, Cloudera Director assumes that the template refers to a server that does not yet exist and must be created.

A database server template also supports a table of key-value pairs of configuration information, which Cloudera Director may require when creating a new server. A template also supports a second table of tag data, which Cloudera Director can employ for certain cloud providers, including Amazon Web Services.

Defining a Database Server Using the API

The Cloudera Director server has a REST service endpoint for managing external database server definitions. The operations supported by the endpoint are described in the table below.
  • Each service URI begins with “/api/v2/environments/{environment}”, where “{environment}” is the name of the environment within which the database server definition is scoped.
  • They all use JSON for input data and response data.
Operation Description Notes
POST /databaseServers/ Define a new database. Admin required.
GET /databaseServers/ List all database servers.  
DELETE /databaseServers/{name} Delete a database server definition. Admin required.
PUT /databaseServers/{name} Update a database server definition. Admin required.
GET /databaseServers/{name} Get a database server definition.  
GET /databaseServers/{name}/status Get the status of a database server.  
GET /databaseServers/{name}/template Get the template from which a database server was defined.  

If a database server template without a host and port is posted to Cloudera Director, Cloudera Director will asynchronously begin the process of creating the server on a cloud provider. The provider is selected based on the environment.

Similarly, if a database server definition is deleted, and the server was originally created by Cloudera Director, Cloudera Director will begin the process of deleting the database from the cloud provider. Before deleting a server definition, be sure to make any backups of the server that you need.

The status of a database server indicates its current position in the server lifecycle. The following values can be returned by the GET database server status operation:

Status Description
BOOTSTRAPPING Cloudera Director is in the process of creating the server.
BOOTSTRAP_FAILED Cloudera Director failed to create the server.
READY The server is available for use.
TERMINATING Cloudera Director is in the process of destroying the server.
TERMINATE_FAILED Cloudera Director failed to terminate the server.
TERMINATED The server has been destroyed.

Defining a Database Server Using the Client Configuration File

Database server templates can be provided in the configuration file passed to the Cloudera Director standalone client. Define external database servers in the databaseServers section of a configuration file.

See the API section above for a description of the different parts of a template. The following example defines two existing database servers.

databaseServers {
   mysql1 {
      type: mysql
      host: 1.2.3.4
      port: 3306
      user: root
      password: password
   }
   postgres1 {
      type: postgresql
      host: 1.2.3.4
      port: 5432
      user: postgres
      password: password
   }
} 

The following example defines a server that Cloudera Director must create using RDS.

databaseServers {
   mysqlt1 {
      type: mysql
      user: root
      password: password
      instanceClass: db.m3.medium
      engineVersion: 5.5.40b
      dbSubnetGroupName: default
      vpcSecurityGroupIds: sg-abcd1234
      allocatedStorage: 10
      tags {
         owner: jsmith
      }
   }
} 

You cannot include both existing servers and servers that Cloudera Director must create, in the same configuration file. You can create new database servers separately in a cloud provider and then define them as existing servers in the configuration file.

Using Amazon RDS for External Databases

Cloudera Director can use Amazon Relational Database Service (RDS) to create new database servers. These servers can be used to host external databases for Cloudera Manager and CDH cluster services.

Creating a Template to Use Amazon RDS as an External Database

An external database server to be created on RDS is defined by a template just like any other server, except that the host and port are not specified; these are determined as the server is being created.
  • name - A unique name for the server within the environment
  • type - The type of database server, such as “MYSQL”
  • username - The name of the administrative account for the server
  • password - The password for the administrative account
The key-value configuration information in the template for an RDS server must include information required by RDS to create a new instance. Cloudera recommends that you specify the engine version in a template. If you do not specify the version, RDS defaults to a recent version, which can change over time.
key description example
instanceClass Instance type for database server instance db.m3.medium
dbSubnetGroupName Name of the DB subnet group which the instance spans default
engineVersion (optional) Version of database engine 5.5.40b
vpcSecurityGroupIds Comma-separated list of security groups for the new instance sg-abc123,sg-def456
allocatedStorage Storage in gigabytes for new server 10
availabilityZone (optional) Preferred availability zone for the new server us-east-1d
backupRetentionPeriod The number of days for which automated backups are retained (0 to disable) 30
skipFinalSnapshot Whether to skip a final snapshot before the instance is deleted (true or false) false
storageEncrypted Whether stored data on RDS instances is encrypted (true or false) true

Defining a Database Server in AWS Using RDS: web UI

You can define an RDS database in AWS using the Cloudera Director web UI when you create a CM instance. In the Database Server section near the top of the Add Cloudera Manager wizard, click the dropdown list and select either Create Database Server Instance or Register Existing Database Server:


Select Create Database Server Instance to create a new MySQL database server with RDS. In the Create Database Server Instance window, enter credentials and configuration values for the database server:


For more information about configuring a database in Amazon RDS see the Amazon Relational Database Service Documentation.

Select Register Existing Database Server to use an existing MySQL or PostgreSQL database server. In the Register Existing Database Server window, enter information and credentials about your existing database server.


Defining a Database Server in AWS Using RDS: API

Use the previously described REST service endpoint for external database server definitions to create and destroy external database servers using RDS. The environment in which servers are defined must already be configured to use AWS, and your account must have permission to create and delete RDS instances.

When an external database server template is submitted through POST to the endpoint, and the template lacks a host and port, Cloudera Director accepts the definition for the server and asynchronously begins the process of creating the new server. The complete existing server definition, including the host and port, will eventually be available through GET.

Likewise, when the definition is deleted using DELETE, Cloudera Director begins destroying the server.

While a new server is being created on RDS, you may begin the process of bootstrapping new deployments and new clusters whose external database templates refer to the server. The bootstrap process will proceed in tandem with the server creation, and pause when necessary to wait for the new RDS instance to be available for use.

When a deployment or cluster is terminated, Cloudera Director leaves RDS instances alone. This makes it possible for multiple deployments and clusters to share the same external database servers that Cloudera Director creates on RDS.

Defining a Database Server in AWS Using RDS: Client Configuration File

The following example defines a server that Cloudera Director must create using RDS:
databaseServers {
   mysqlt1 {
      type: mysql
      user: root
      password: password
      instanceClass: db.m3.medium
      engineVersion: 5.5.40b
      dbSubnetGroupName: default
      vpcSecurityGroupIds: sg-abcd1234
      allocatedStorage: 10
      tags {
         owner: jsmith
      }
   }
}
The following example of an external database template uses the new server that Cloudera Director needs to create. The databaseServerName item matches the name of the new server:
cluster {
   #... databaseTemplates: {
   HIVE {
        name: hivetemplate
        databaseServerName: mysqlt1
        databaseNamePrefix: hivemetastore
        usernamePrefix: hive
   }
}