Deploying and managing Kafka Connect connectors in SMM

Configuring properties

Each row on the Connector Configuration page represents the key (name) and configuration value of a specific property. You can use the icons next to each property to add or remove properties. Clicking next to a property opens a context menu that includes additional configuration options. The options available in the context menu depend on the property. For example, you can edit and reset the connector.class property, but you cannot configure its type.

Adding properties

Click to add a new property. An empty row appears for the new property. Enter property keys in the left field and property values in the right field. When you select a key field, a list of suggested properties appear that are valid for the connector. The list is automatically filtered when you start typing.

Deleting properties

Click to delete a property. Use Actions > Remove to remove all properties from the configuration with the exception of connector.class.

Editing properties

Click Edit to edit the configuration value of a property in a separate window. Use this option if the value you add is long, complex, or consists of multiple lines.

If you are deploying a NiFi Stateless Sink or Source connector, the modal that opens with for the flow.snapshot property is unique and includes the Browse... and Save and Enhance options. Browse... enables you to upload a flow definition JSON from your machine, Save and Enhance adds the parameters specified in your flow definition to the connector configuration. For more information on flow.snapshot configuration, see Configuring flow.snapshot for Stateless NiFi connectors.

Resetting properties

You can use Reset value to reset the value of a single property. Alternatively, you can use Actions > Reset to reset the full configuration. The reset options behave uniquely depending on the configuration page that you use them on.

When deploying new connectors using the Connector Configuration page of the wizard:

• Reset value resets the value to the default value set in the configuration template. This option is not available for properties that are not part of the template.
• Actions > Reset resets all configuration properties and reverts the configuration to the default configuration template.

When editing the configuration of existing connectors on the Connector Profile > Connector Settings page:

• Reset value resets the value to the last value that the connector was deployed with. This option is not available for new properties that you add to the configuration while editing the configuration.
• Actions > Reset resets all configuration properties and reverts to the configuration that the connector was last deployed with.

Type configuration

The String , Boolean, Number, and Password options enable you to set the type of the property. Setting the type of a property changes the property's value field depending on the type you select.

• String sets the type of the property to string and changes the input field to a single line text box.
• Number sets the type to number and changes the input field to a text box that only allows negative and positive whole numbers and decimals.
• Boolean sets the type of the property to boolean and changes the input field to a checkbox.
• Password sets the type of the property to password.

  Selecting Password hides (locks) the value on the UI and stores it in a secure manner. Once the connector is deployed, properties marked as passwords are encrypted and stored securely in an internal Kafka topic. The actual values only resolve at runtime. These values cannot be retrieved or otherwise read from the configuration of the connector. In most cases, you use this option to hide passwords or other types of sensitive credentials found in connector configurations, however, you can use this option for any property value that your organization considers sensitive.

Property search and filtering

At the top of the page you can find a search bar that you can use to search and filter for connector properties. Search only works for property keys, values are disregarded. When you search for a string, the Properties section of the page is automatically filtered and only the relevant properties are shown. The number of matches is displayed next to Properties.

In addition to searching, you can filter properties based on their group and importance. To do so, click the filter icon and select the relevant group and importance that you want to search for. To clear all applied filters, click Reset Filters.

Viewing property help

Hovering over (Help), found next to some property values, displays information about that property. What information is displayed is property dependent, but at minimum you are presented with a description of the property. If available, the group, importance, and expected type of a property is also displayed. The icon is only available for properties that have their metadata (description, type, group, and so on) defined.

Importing configurations

To import a configuration, click Actions > Import. Clicking Import opens the Connector Config… modal.

Using the modal you can:

• Type or paste your configuration.
• Use Browse… to upload the contents of a file from your machine.

The configuration that you add can be a full configuration that contains all properties necessary for the connector. Alternatively, it can be a partial configuration that contains a select number of properties that you want to add to the configuration. The configuration you include in the text box must be in JSON format. For example:

{
  "tasks.max": 1,
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "value.converter": "com.cloudera.dim.kafka.connect.converts.AvroConverter",
  "value.converter.passthrough.enabled": true,
  "value.converter.schema.registry.url": "http://schema-registry:9090/api/v1",
  "topics": "avro_topic",
  .
  .
  .
}

After adding properties, you can use the Import, Import and Enhance, or Add options to import the configuration. Each option imports the properties in a different way.

• Import

  This option populates the Properties section with the keys and values you added in the Import Connector Config... modal. This option overwrites all existing properties in the configuration.

• Import and Enhance

  This option populates the Properties section with the keys and values you added in the Import Connector Config... modal. Additionally, properties that are most likely needed for the respective connector are also added.

  Import and Enhance is used specifically for the NiFi Stateless Source and NiFi Stateless Sink connectors. When this option is used, SMM parses the dataflow specified in flow.snapshot, extracts all parameters that are available in the dataflow, and adds them to the connector configuration. This option overwrites all existing properties in the configuration.

• Add

  This option amends the existing configuration with the properties from the Import Connector Config... modal. Use this option if you want to batch import additional properties to your configuration. For example, you can use this option to batch add security properties.

Exporting Configurations

To export a configuration, click Actions > Export. Clicking Export opens the Configuration of the connector modal.

• Click Copy to Clipboard to copy the configuration to the clipboard.
• Click Export to file to download the configuration as a JSON file.

Validating configurations

Before you can deploy or redeploy a connector, you must validate the configuration. This is done using Validate. If SMM finds any errors in the configuration, the properties that contain errors are highlighted in red, and an error message with the details regarding the configuration issue is displayed.

If the configuration is missing a mandatory property, the name of the missing property is displayed in an Errors section on the top of the page. You can add missing mandatory properties by clicking Add missing configurations.

If the connector configuration is filtered when you run the validation and SMM finds errors with properties that are filtered, an error message is displayed on the top of the page notifying you that not all validation errors are visible. In a case like this, click Reset Filters to view all validation errors.