Use the container as a sidecar container in Kubernetes

Learn how to use the docker container as a sidecar container in Kubernetes.

The MiNiFi container can also be used in a Kubernetes environment, for example, as a logging agent for collecting logs from an application as a sidecar container. In the following example a use case is shown where a NiFi container runs in a pod together with a MiNiFi container as a sidecar collecting NiFi application logs.

For simplicity, the NiFi container is configured with default parameters so that no custom configuration is added for that instance. In this example, MiNiFi collects the NiFi logs, and after every minute it compresses and uploads the NiFi logs to an AWS S3 bucket. For this use case, you can use the following configuration:

Flow Controller:
  name: minifi-logging
Processors:
- id: 94b8e610-b4ed-1ec9-b26f-c839931bf3e2
  name: TailFile
  class: org.apache.nifi.processors.standard.TailFile
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 5 sec
  auto-terminated relationships list: []
  Properties:
    File to Tail: /nifi-logs/nifi-app.log
    Lookup frequency: 1 min
- id: 261e8cf1-71ba-af86-fb2b-bc95764f91f8
  name: MergeContent
  class: org.apache.nifi.processors.standard.MergeContent
  scheduling strategy: EVENT_DRIVEN
  auto-terminated relationships list:
    - original
  Properties:
    Attribute Strategy: Keep Only Common Attributes
    Maximum number of Bins: 100
    Minimum Group Size: 0
    Max Bin Age: 1 min
    Minimum Number of Entries: 1000000
    Maximum Group Size: 1000000
    Maximum Number of Entries: 1000000
    Merge Strategy: Bin-Packing Algorithm
- id: 69335770-ee29-11eb-9a03-0242ac130003
  name: CompressContent
  class: org.apache.nifi.processors.standard.CompressContent
  scheduling strategy: EVENT_DRIVEN
  auto-terminated relationships list:
    - failure
  Properties:
    Compression Level: 6
    Compression Format: gzip
    UpdateFileName: false
- id: fe198bd9-2a1c-316e-0000-000000000000
  name: PutS3Object
  class: org.apache.nifi.minifi.azure.processors.PutS3Object
  scheduling strategy: EVENT_DRIVEN
  auto-terminated relationships list:
    - success
  Properties:
    Bucket: test_bucket
    AWS Credentials Provider service: AWSCredentialsService
Controller Services:
- name: AWSCredentialsService
  id: 2094d776-2006-4d02-9bb9-28eac9d0fc95
  class: org.apache.nifi.minifi.aws.controllers.AWSCredentialsService
  Properties:
    Use Default Credentials: 'true' # Can be used in Amazon EKS to retrieve credentials from metadata otherwise use your AWS Access Key and Secret Key
Connections:
- id: 99f617e7-49a1-6078-8534-26af7d56ca08
  name: TailFile/success/MergeContent
  source name: TailFile
  source relationship names:
  - success
  destination name: MergeContent
- id: 24d6be1e-ee29-11eb-9a03-0242ac130003
  name: MergeContent/merged/CompressContent
  source name: MergeContent
  source relationship names:
  - merged
  destination name: CompressContent
- id: 67ea5c91-446a-393b-6274-b6fae2f475a2
  name: CompressContent/success/PutS3Object
  source name: CompressContent
  source relationship names:
  - success
  destination name: PutS3Object
Remote Process Groups: []

In the pod specification you have to define the application and the MiNiFi containers to mount a shared volume of the logs to be collected. In the following example the NiFi logs are mounted to both NiFi and MiNiFi containers. MiNiFi also mounts the previously detailed configuration from the /root/minifi-config host directory.

apiVersion: v1
kind: Pod
metadata:
  name: log-collection-minifi-pod
  namespace: default
spec:
  containers:
  - name: nifi
    image: apache/nifi:latest
    volumeMounts:
    - name: nifi-logs
      mountPath: /opt/nifi/nifi-current/logs
  - name: sidecar-minifi
    image: container.repo.cloudera.com/cloudera/apacheminificpp:latest
    volumeMounts:
    - name: nifi-logs
      mountPath: /nifi-logs
    - name: minifi-config
      mountPath: /opt/minifi/minifi-current/conf/
  volumes:
  - name: nifi-logs
    emptyDir: {}
  - name: minifi-config
    hostPath:
      path: /root/minifi-config

Another option would be to instead of mounting the configuration files from the host directory they can be configured in a Kubernetes configuration map beforehand and this way they can be mounted individually. A configuration map definition template would look something like this:

apiVersion: v1
data:
  minifi-log.properties: |
    <minifi log properties file content>
  minifi.properties: |
    <minifi properties file content>
  config.yml: |
    <config file content>
kind: ConfigMap
metadata:
  labels:
    k8s-app: minifi-log-collection
  name: minifi-log-collection-config
  namespace: default

With the configuration map defined, you can change the previous pod definition in the following way to use the configuration map volume:

apiVersion: v1
kind: Pod
metadata:
  name: log-collection-minifi-pod
  namespace: default
spec:
  containers:
  - name: nifi
    image: apache/nifi:latest
    volumeMounts:
    - name: nifi-logs
      mountPath: /opt/nifi/nifi-current/logs
  - name: sidecar-minifi
    image: container.repo.cloudera.com/cloudera/apacheminificpp:latest
    volumeMounts:
    - name: nifi-logs
      mountPath: /nifi-logs
    - name: minificonfig
      mountPath: /opt/minifi/minifi-current/conf/config.yml      
      subPath: config.yml
    - name: minificonfig
      mountPath: /opt/minifi/minifi-current/conf/minifi-log.properties
      subPath: minifi-log.properties
  volumes:
  - name: nifi-logs
    emptyDir: {}
  - configMap:
      defaultMode: 420
      name: minifi-log-collection-config
    name: minificonfig

You can find an example of sidecar Kubernetes YAML configuration file here with the previous pod and configuration map definitions to try. The suggested changes are marked with the TODO keyword.