Cloudera Docs

Data Connectors overview

Cloudera Data Engineering provides data connectors to seamlessly integrate your Spark workloads with various internal and external data storage and streaming systems.

What is a Data Connector?

In Cloudera Data Engineering, a data connector is a pre-configured integration mechanism that links your Cloudera Data Engineering Virtual Cluster to a specific data source. When you attach a data connector to a Cloudera Data Engineering job, it automatically injects the required dependencies, libraries, and configuration files into the job's runtime environment. This allows your Spark applications to natively read from and write to these systems without requiring extensive manual setup.

The Need for Data Connectors

Modern data engineering pipelines rarely operate in isolation, they require robust connections to object stores, NoSQL databases, and streaming platforms. Without data connectors, you face several challenges:

  • Dependency management: Manually tracking down, uploading, and maintaining the correct versions of client JAR files for each data source.
  • Complex configuration: Manually passing environment variables, connection strings, and configuration files such as core-site.xml or hbase-site.xml to every Spark submit command.
  • Security and authentication: Handling Kerberos tickets, SSL or TLS certificates, and credentials securely across distributed executors.

Data connectors solve these problems by providing:

  • Simplified job deployment: By selecting a connector during job creation, Cloudera Data Engineering automatically provisions the necessary driver and executor dependencies.
  • Centralized configuration: Administrators can configure the connection details once at the environment or virtual cluster level, and you can reuse them across multiple jobs.
  • Built-in security: Connectors automatically inherit the overarching security context like Kerberos authentication of your Cloudera Base on premises environment, ensuring secure data access out of the box.

Supported Data Connectors in Cloudera Data Engineering

Depending on your Cloudera Data Engineering release version, different data connectors are available to support various workloads:

  • Ozone: Ozone is a highly scalable, distributed, and high-performance object store tailored for big data workloads.
  • HBase (Available from Cloudera Data Engineering 1.5.5 SP4 and later): Allows Spark jobs to read and write large-scale, sparse, and unstructured data natively from HBase tables.
  • Phoenix (Available from Cloudera Data Engineering 1.5.5 SP4 and later): Enables Online Transaction Processing (OLTP) and operational analytics by allowing you to query HBase using standard SQL through Spark.
  • Kafka (Available from Cloudera Data Engineering 1.5.5 SP4 and later): Integrates Cloudera Data Engineering with Kafka for robust stream processing, allowing Spark Structured Streaming jobs to consume and publish real-time event data.