Cloudera Docs

Prerequisites

Learn how to collect the information you need to deploy the Confluent Cloud to Snowflake ReadyFlow, and meet other prerequisites.

For your data ingest source

  • You have created a Confluent Cloud Kafka cluster.

  • You have created at least one Kafka topic.

  • You have created a schema for your data in the Confluent Cloud Schema Registry.

  • You have the Confluent Cloud Kafka broker endpoint.

  • You have the Confluent Cloud Kafka client API Key and Secret.

  • You have the Kafka Consumer Group ID.

  • You have the Confluent Cloud Schema Registry endpoint.

  • You have the Confluent Cloud Schema Registry API Key and Secret.

For DataFlow

  • You have enabled DataFlow for an environment.

    For information on how to enable DataFlow for an environment, see Enabling DataFlow for an Environment.

  • You have created a Machine User to use as the CDP Workload User.

  • You have given the CDP Workload User the EnvironmentUser role.
    1. From the Management Console, go to the environment for which DataFlow is enabled.
    2. From the Actions drop down, click Manage Access.
    3. Identify the user you want to use as a Workload User.
    4. Give that user EnvironmentUser role.

  • You have synchronized your user to the CDP Public Cloud environment that you enabled for DataFlow.

    For information on how to synchronize your user to FreeIPA, see Performing User Sync.

  • You have granted your CDP user the DFCatalogAdmin and DFFlowAdmin roles to enable your user to add the ReadyFlow to the Catalog and deploy the flow definition.
    1. Give a user permission to add the ReadyFlow to the Catalog.
      1. From the Management Console, click User Management.
      2. Enter the name of the user or group you wish to authorize in the Search field.
      3. Select the user or group from the list that displays.
      4. Click Roles > Update Roles.
      5. From Update Roles, select DFCatalogAdmin and click Update.
    2. Give your user or group permission to deploy flow definitions.
      1. From the Management Console, click Environments to display the Environment List page.
      2. Select the environment to which you want your user or group to deploy flow definitions.
      3. Click Actions > Manage Access to display the Environment Access page.
      4. Enter the name of your user or group you wish to authorize in the Search field.
      5. Select your user or group and click Update Roles.
      6. Select DFFlowAdmin from the list of roles.
      7. Click Update Roles.
    3. Give your user or group access to the Project where the ReadyFlow will be deployed.
      1. Go to DataFlow > Projects.
      2. Select the project where you want to manage access rights and click More > Manage Access.
    4. Start typing the name of the user or group you want to add and select them from the list.
    5. Select the Resource Roles you want to grant.
    6. Click Update Roles.
    7. Click Synchronize Users.

For your data ingest target

  • You have created a Snowflake account.

  • Ensure that the Snowflake database and table you are ingesting data to exist. If not, create them.

  • You have set up your destination Snowflake table with the right columns and columns type.

  • Your Snowflake user account has write access to the Snowflake database table.

  • You have the schema of the Snowflake database. The default value is PUBLIC.

  • You have created Snowflake Private and Public Keys and assigned the Public Key to your Snowflake account user.

    For more information, see Key Pair Authentication in Snowflake documentation.

  • You have created your Snowflake Pipe and configured the file format for CSV input.

    For more information, see CREATE FILE FORMAT in Snowflake documentation.