Adding a Flume Service

Required Role:

1. On the Home page, click to the right of the cluster name and select Add a Service. A list of service types display. You can add one type of service at a time.
2. Select the Flume service and click Continue.
3. Select the radio button next to the services on which the new service should depend and click Continue.
4. Customize the assignment of role instances to hosts. The wizard evaluates the hardware configurations of the hosts to determine the best hosts for each role. The wizard assigns all worker roles to the same set of hosts to which the HDFS DataNode role is assigned. These assignments are typically acceptable, but you can reassign role instances to hosts of your choosing, if desired.

   Click a field below a role to display a dialog containing a pageable list of hosts. If you click a field containing multiple hosts, you can also select All Hosts to assign the role to all hosts or Custom to display the pageable hosts dialog.

   The following shortcuts for specifying hostname patterns are supported:

   • Range of hostnames (without the domain portion)

     Range Definition	Matching Hosts
     10.1.1.[1-4]	10.1.1.1, 10.1.1.2, 10.1.1.3, 10.1.1.4
     host[1-3].company.com	host1.company.com, host2.company.com, host3.company.com
     host[07-10].company.com	host07.company.com, host08.company.com, host09.company.com, host10.company.com

   • IP addresses
   • Rack name

   Click the View By Host button for an overview of the role assignment by hostname ranges.

For general discussion of adding a service, see Adding a Service.

Configuring the Flume Agents

Required Role:

After you create a Flume service, you must first configure the agents before you start them. For detailed information about Flume agent configuration, see the Flume User Guide.

The default Flume agent configuration provided in the Configuration File property of the Agent default role group is a configuration for a single agent in a single tier; you should replace this with your own configuration. When you add new agent roles, they are placed (initially) in the Agent default role group.

Agents that share the same configuration should be members of the same agent role group. You can create new role groups and can move agents between them. If your Flume configuration has multiple tiers, you must create an agent role group for each tier, and move each agent to be a member of the appropriate role group for their tier.

A Flume agent role group Configuration File property can contain the configuration for multiple agents, since each configuration property is prefixed by the agent name. You can set the agents' names using configuration overrides to change the name of a specific agent without changing its role group membership. Different agents can have the same name — agent names do not have to be unique.

1. Go to the Flume service.
2. Click the Configuration tab.
3. Select the Agent default role group in the left hand column. The settings you make here apply to the default role group, and thus will apply to all agent instances unless you associate those instances with a different role group, or override them for specific agents.
4. Set the Agent Name property to the name of the agent (or one of the agents) defined in the flume.conf configuration file. The agent name can be comprised of letters, numbers, and the underscore character. You can specify only one agent name here — the name you specify will be used as the default for all Flume agent instances, unless you override the name for specific agents. You can have multiple agents with the same name — they will share the configuration specified in on the configuration file.
5. Copy the contents of the flume.conf file, in its entirety, into the Configuration File property. Unless overridden for specific agent instances, this property applies to all agents in the group. You can provide multiple agent configurations in this file and use agent name overrides to specify which configuration to use for each agent.
     Important: The name-value property pairs in the Configuration File property must include an equal sign (=). For example, tier1.channels.channel1.capacity = 10000.

Setting a Flume Agent Name

Required Role:

If you have specified multiple agent configurations in a Flume agent role group Configuration File property, you can set the agent name for an agent that uses a different configuration. Overriding the agent name will point the agent to the appropriate properties specified in the agent configuration.

1. Go to the Flume service.
2. Click the Configuration tab.
3. Select an agent role group in the left hand column.
4. To override the agent name for one or more instances within the group, move your cursor over the Value column of the Agent Name property row, and click Override Instances.
5. Select the agent role instances to override.
6. In the field Change value of selected instances to: select Other. (You can use the Inherited Value setting to return to the role group value.)
7. In the field that appears, type the name for the selected agent instance.
8. Click Apply to have your change take effect.

Using Flume with HDFS or HBase Sinks

If you want to use Flume with HDFS or HBase sinks, you can add a dependency to that service from the Flume configuration page. This will automatically add the correct client configurations to the Flume agent's classpath.

  Note: If you are using Flume with HBase, make sure that the /etc/zookeeper/conf/zoo.cfg file either does not exist on the host of the Flume agent that is using an HBase sink, or that it contains the correct ZooKeeper quorum.

Using Flume with Solr Sinks

Cloudera Manager provides a set of configuration settings under the Flume service to configure the Flume Morphline Solr Sink. See Configuring the Flume Morphline Solr Sink for Use with the Solr Service for detailed instructions.

Updating Flume Agent Configurations

Required Role:

1. Go to the Flume service.
2. Select Actions > Update Config.