Sidecars with pod template

You can extend your FlinkDeployment in case you want to add more containers in your Kubernetes pod using the pod template and sidecars.

The Flink Operator CRD has a minimal set of settings to express the basic attributes of a deployment. For more customization you can use the flinkConfiguration and podTemplate properties.

Pod templates allow customization of the Flink job and task manager pods, to, for example, specify volume mounts, ephemeral storage, sidecar containers and so on.

Pod templates can be layered as shown in the below example. You can define the settings for the pod templates to be applied to both the job and task manager in a common pod template. You can also add another template under the job or task manager to define additional settings that supplement (or override) the common template, for example when using sidecars.

Defining sidecars instruct the Flink Operator to create other containers in the Flink JobManager and TaskManager pods, for example:

to download artifacts (for example, JAR files) before executing the job
to collect metrics and logs from Flink during runtime and analyze/save them.

The following example sets up another container running next to Flink in all the created pods to periodically output the size of the log file:

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: flink-kubernetes-tutorial
spec:
  image: [***REGISTRY HOST***]:[***PORT***]/[***PROJECT***]/flink-kubernetes-tutorial:latest
  flinkVersion: v1_18
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "4"
  serviceAccount: flink
  mode: native
  jobManager:
    resource:
      memory: "2048m"
      cpu: 1
  taskManager:
    resource:
      memory: "2048m"
      cpu: 1
  job:
    jarURI: local:///opt/flink/usrlib/flink-kubernetes-tutorial.jar
    parallelism: 4
    state: running
    upgradeMode: stateless
  podTemplate:
    spec:
      containers:
        - name: flink-main-container
          volumeMounts:
            - mountPath: /opt/flink/log
              name: flink-logs
        - name: sidecar
          image: busybox
          command: [ 'sh','-c','while true; do wc -l /flink-logs/*.log; sleep 5; done' ]
          volumeMounts:
            - mountPath: /flink-logs
              name: flink-logs
      initContainers:
        # Sample sidecar container
        - name: sidecar-init
          image: busybox
          command: [ 'sh', '-c', 'echo initContainer loaded' ]
      volumes:
        - name: flink-logs
          emptyDir: {}

You can use the following commands to create the new deployment:

kubectl -n flink apply -f flink-deployment.yaml

This sidecar creates a new temporary volume called flink-logs in the Flink main container that is mounted to the default log output path, /opt/flink/log. The example also creates a BusyBox sidecar that also mounts the same volume and periodically prints the logs' line count.

The init-container is a type of container that needs to finish running and exit with code 0 before the other containers can start. As an example, this can be used to download artifacts for the Flink jobs.