Securing Apache Flink jobs

Submitting Flink jobs in a secure environment requires every security parameter for authentication, authorization and other connector related security settings. You should prepare your keystore and keytab files for Flink and for also the chosen connector component.

Running secure Flink Deployments

The following example shows the security parameters that are needed to submit a Flink job:

flink run -d -p 2 \
-yD security.kerberos.login.keytab=test.keytab \
-yD security.kerberos.login.principal=test \
-yD security.ssl.internal.enabled=true \
-yD security.ssl.internal.keystore=keystore.jks \
-yD security.ssl.internal.key-password=`cat pwd.txt` \
-yD security.ssl.internal.keystore-password=`cat pwd.txt` \
-yD security.ssl.internal.truststore=keystore.jks \
-yD security.ssl.internal.truststore-password=`cat pwd.txt` \
-yt keystore.jks \
flink-secure-tutorial-1.0-SNAPSHOT.jar \
--kafkaTopic flink \
--hdfsOutput hdfs:///tmp/flink-secure-tutorial \
--kafka.bootstrap.servers <broker_host>:9093 \
--kafka.security.protocol SASL_SSL \
--kafka.sasl.kerberos.service.name kafka \
--kafka.ssl.truststore.location /etc/cdep-ssl-conf/CA_STANDARD/truststore.jks

The Kerberos and TLS properties are user specific parameters. Generally the cluster administrator provides the Kerberos keytab and TLS certificate to the user. In case you did not receive the keytab and keystore file from the administrator, you can use the following commands:

> ktutil
ktutil: add_entry -password -p test -k 1 -e des3-cbc-sha1
Password for test@:
ktutil:  wkt test.keytab
ktutil:  quit

keytool -genkeypair -alias flink.internal -keystore keystore.jks -dname "CN=flink.internal" -storepass `cat pwd.txt` -keyalg RSA -keysize 4096 -storetype PKCS12

The full explanation of the properties used in the example can be found in the Secure Tutorial. It also includes how to enable security features step-by-step for Flink applications that are running on secured CDP Private Cloud Base environments.

Using SPNEGO authentication with Kerberos for Flink Deployments

By default on a kerberized cluster the Flink dashboard is secured by the YARN proxy, which means direct external access to the Flink deployments should be disabled.

Alternatively, it is possible to directly enable SPNEGO for the Flink Dashboard. To secure a session cluster, use the following configuration options:

flink-yarn-session -d -nm [***CLUSTER-NAME***] \
-Dsecurity.kerberos.login.use-ticket-cache=true \
-Dsecurity.spnego.auth.enabled=true \
-Dsecurity.spnego.auth.keytab=[***SERVICE-KEYTAB***] \
-Dsecurity.spnego.auth.principal=[***HTTP PRINCIPAL***] \
-Dyarn.ship-files=[***SERVICE-KEYTAB***]

To run a Flink job on the previously created session cluster, use the following command:

flink run \
-d -t yarn-session \
-yid [***YARN APPLICATION ID***] \
-yD security.spnego.auth.enabled=true \
-yD security.kerberos.login.keytab=[***USER-KEYTAB***] \
-yD security.kerberos.login.principal=[***USER-PRINCIPAL***] \
[***FULL PATH***][***FILENAME***].jar