A Flink program may use first- or third-party connectors with required authentication methods which can be Kerberos, SSL/TLS, username/password, and so on.

While meeting the security requirements for various connectors is an ongoing effort, Flink provides first-class support for Kerberos authentication only.

The primary goals of the Flink Kerberos security infrastructure are:
  • to enable secure data access for jobs within a cluster through connectors (for example, Kafka)
  • to authenticate to Hadoop components (for example, HDFS, HBase, Zookeeper)

In a production deployment scenario, streaming jobs usually run for long periods of time. Authentication is mandatory to secure data sources throughout the lifetime of a job. Kerberos keytabs do not expire in that timeframe, unlike a Hadoop delegation token or ticket cache entry. Cloudera recommends using keytabs for long-running production deployments.