Defining External Database Servers
Cloudera Director needs information about external database servers before it can use them. This section describes defining database server templates and using Amazon Relational Database Service (RDS) to create new database servers..
Continue reading:
- The Database Server Template
- Defining a Database Server Using the API
- Defining a Database Server Using the Client Configuration File
- Using Amazon RDS for External Databases
- Defining a Database Server in AWS Using RDS: Web UI
- Defining a Database Server in AWS Using RDS: API
- Defining a Database Server in AWS Using RDS: Client Configuration File
The Database Server Template
- name - A unique name for the server within the environment
- type - The type of database server, such as “MYSQL” or “POSTGRESQL”
- hostname - The name of the server host
- port - The listening port of the server
- username - The name of the administrative account for the server
- password - The password for the administrative account
The hostname and port are optional in a template. If they are not present, Cloudera Director assumes that the template refers to a server that does not yet exist and must be created.
A database server template also supports a table of key-value pairs of configuration information, which Cloudera Director might require when creating a new server. A template also supports a second table of tag data, which Cloudera Director can employ for certain cloud providers, including Amazon Web Services.
Defining a Database Server Using the API
- Each service URI begins with “/api/v9/environments/{environment}”, where “{environment}” is the name of the environment within which the database server definition is scoped.
- They all use JSON for input data and response data.
Operation | Description | Notes |
---|---|---|
POST /databaseServers/ | Define a new database. | Admin required. |
GET /databaseServers/ | List all database servers. | |
DELETE /databaseServers/{name} | Delete a database server definition. | Admin required. |
PUT /databaseServers/{name} | Update a database server definition. | Admin required. |
GET /databaseServers/{name} | Get a database server definition. | |
GET /databaseServers/{name}/status | Get the status of a database server. | |
GET /databaseServers/{name}/template | Get the template from which a database server was defined. |
If a database server template without a host and port is posted to Cloudera Director, Cloudera Director will asynchronously begin the process of creating the server on a cloud provider. The provider is selected based on the environment.
Similarly, if a database server definition is deleted, and the server was originally created by Cloudera Director, Cloudera Director will begin the process of deleting the database from the cloud provider. Before deleting a server definition, be sure to make any backups of the server that you need.
The status of a database server indicates its current position in the server lifecycle. The following values can be returned by the GET database server status operation:
Status | Description |
---|---|
BOOTSTRAPPING | Cloudera Director is in the process of creating the server. |
BOOTSTRAP_FAILED | Cloudera Director failed to create the server. |
READY | The server is available for use. |
TERMINATING | Cloudera Director is in the process of destroying the server. |
TERMINATE_FAILED | Cloudera Director failed to terminate the server. |
TERMINATED | The server has been destroyed. |
Defining a Database Server Using the Client Configuration File
Database server templates can be provided in the configuration file passed to the Cloudera Director client. Define external database servers in the databaseServers section of a configuration file.
See the API section above for a description of the different parts of a template. The following example defines two existing database servers.
databaseServers { mysql1 { type: mysql host: 1.2.3.4 port: 3306 user: root password: password } postgres1 { type: postgresql host: 1.2.3.4 port: 5432 user: postgres password: password } }
The following example defines a server that Cloudera Director must create using RDS.
databaseServers { mysqlt1 { type: mysql user: root password: password instanceClass: db.m3.medium engineVersion: 5.5.40b dbSubnetGroupName: default vpcSecurityGroupIds: sg-abcd1234 allocatedStorage: 10 tags { owner: jsmith } } }
You cannot include both existing servers and servers that Cloudera Director must create, in the same configuration file. You can create new database servers separately in a cloud provider and then define them as existing servers in the configuration file.
Using Amazon RDS for External Databases
Creating a Template to Use Amazon RDS as an External Database
- name - A unique name for the server in the environment
- type - The type of database server, such as “MYSQL”
- username - The name of the administrative account for the server
- password - The password for the administrative account
Key | Description | Example |
---|---|---|
instanceClass | Instance type for the database server instance | db.m3.medium |
dbSubnetGroupName | Name of the database subnet group that the instance spans | default |
engineVersion | (optional) Version of the database engine | 5.5.40b |
vpcSecurityGroupIds | Comma-separated list of security groups for the new instance | sg-abc123,sg-def456 |
allocatedStorage | Storage in gigabytes for the new server | 10 |
availabilityZone | (optional) Preferred availability zone for the new server | us-east-1d |
backupRetentionPeriod | Number of days for which automated backups are retained (0 to disable) | 30 |
skipFinalSnapshot | Whether to skip a final snapshot before the instance is deleted (true to skip; otherwise false) | false |
storageEncrypted | Whether stored data on RDS instances is encrypted (true to encrypt; otherwise false) | true |
Defining a Database Server in AWS Using RDS: Web UI
You can define an RDS database in AWS using the Cloudera Director web UI when you create a Cloudera Manager instance. In the Database Server section near the top of the Add Cloudera
Manager wizard, click the dropdown list and select either Create Database Server Instance or Register Existing Database
Server:
![](../images/director_create_db_for_cm.jpg)
![](../images/director_create_db_server_instance_scrn.jpg)
Select Register Existing Database Server to use an existing MySQL or PostgreSQL database server. In the Register Existing
Database Server window, enter information and credentials about your existing database server.
![](../images/director_register_existing_db_for_cm.jpg)
Defining a Database Server in AWS Using RDS: API
Use the previously described REST service endpoint for external database server definitions to create and destroy external database servers using RDS. The environment in which servers are defined must already be configured to use AWS, and your account must have permission to create and delete RDS instances.
When an external database server template is submitted through POST to the endpoint, and the template lacks a host and port, Cloudera Director accepts the definition for the server and asynchronously begins the process of creating the new server. The complete existing server definition, including the host and port, are eventually available through GET.
Likewise, when the definition is deleted using DELETE, Cloudera Director begins destroying the server.
While a new server is being created on RDS, you can begin bootstrapping new deployments and new clusters that have external database templates that refer to the server. The bootstrap process proceeds in tandem with the server creation, and pauses when necessary to wait for the new RDS instance to be available.
When a deployment or cluster is terminated, Cloudera Director does not terminate the RDS instances. As a result, multiple deployments and clusters can share the same external database servers that Cloudera Director creates on RDS.
Defining a Database Server in AWS Using RDS: Client Configuration File
databaseServers { mysqlt1 { type: mysql user: root password: password instanceClass: db.m3.medium engineVersion: 5.7.40b dbSubnetGroupName: default vpcSecurityGroupIds: sg-abcd1234 allocatedStorage: 10 tags { owner: jsmith } } }
cluster { #... databaseTemplates: { HIVE { name: hivetemplate databaseServerName: mysqlt1 databaseNamePrefix: hivemetastore usernamePrefix: hive } }