4. Configuring Memory as Storage

Use the following steps to configure DataNode memory as storage:

1. Shut Down the DataNode

Shut down the DataNode using the applicable commands in the Controlling HDP Services Manually section of the HDP Reference Guide.

2. Mount a Portion of DataNode Memory for HDFS

To use DataNode memory as storage, you must first mount a portion of the DataNode memory for use by HDFS.

For example, you would use the following commands to allocate 2GB of memory for HDFS storage:

sudo mkdir -p /mnt/hdfsramdisk
sudo mount -t tmpfs -o size=2048m tmpfs /mnt/hdfsramdisk
Sudo mkdir -p /usr/lib/hadoop-hdfs

3. Assign the RAM_DISK Storage Type and Enable Short-Circuit Reads

Edit the following properties in the /etc/hadoop/conf/hdfs-site.xml file to assign the RAM_DISK storage type to DataNodes and enable short-circuit reads.

For example:

 <property>
 <name>dfs.data.dir</name>
 <value>file:///grid/3/aa/hdfs/data/,[RAM_DISK]file:///mnt/hdfsramdisk/</value>
 </property>
 
 <property>
 <name>dfs.client.read.shortcircuit</name>
 <value>true</value>
 </property>
 
 <property>
 <name>dfs.domain.socket.path</name>
 <value>/var/lib/hadoop-hdfs/dn_socket</value>
 </property>
 
 <property>
 <name>dfs.checksum.type</name>
 <value>NULL</value>
 </property>

4. Set the LAZY_PERSIST Storage Policy on Files or Directories

Set a storage policy on a file or a directory.

Command:

hdfs dfsadmin -setStoragePolicy <path> <policyName>

Arguments:

Example:

hdfs dfsadmin -setStoragePolicy /memory1 LAZY_PERSIST 

Get the storage policy of a file or a directory.

Command:

hdfs dfsadmin -getStoragePolicy <path>

Arguments:

Example:

hdfs dfsadmin -getStoragePolicy /memory1 LAZY_PERSIST 

5. Start the DataNode

Start the DataNode using the applicable commands in the Controlling HDP Services Manually section of the HDP Reference Guide.

Using Mover to Apply Storage Policies

When you update a storage policy setting on a file or directory, the new policy is not automatically enforced. You must use the HDFS mover data migration tool to actually move blocks as specified by the new storage policy.

The mover data migration tool scans the specified files in HDFS and checks to see if the block placement satisfies the storage policy. For the blocks that violate the storage policy, it moves the replicas to the applicable storage type in order to fulfill the storage policy requirements.

Command:

hdfs mover [-p <files/dirs> | -f <local file name>] 

Arguments:

	Note
	When both -p and -f options are omitted, the default path is the root directory.