Use the container as a sidecar container in Kubernetes
Learn how to use the docker container as a sidecar container in Kubernetes.
The MiNiFi container can also be used in a Kubernetes environment, for example, as a logging agent for collecting logs from an application as a sidecar container. In the following example a use case is shown where a NiFi container runs in a pod together with a MiNiFi container as a sidecar collecting NiFi application logs.
For simplicity, the NiFi container is configured with default parameters so that no
custom configuration is added for that instance. In this example, MiNiFi collects the NiFi
logs, and after every minute it compresses and uploads the NiFi logs to an AWS S3 bucket. For
this use case, you can use the following
configuration:
Flow Controller: name: minifi-logging Processors: - id: 94b8e610-b4ed-1ec9-b26f-c839931bf3e2 name: TailFile class: org.apache.nifi.processors.standard.TailFile scheduling strategy: TIMER_DRIVEN scheduling period: 5 sec auto-terminated relationships list: [] Properties: File to Tail: /nifi-logs/nifi-app.log Lookup frequency: 1 min - id: 261e8cf1-71ba-af86-fb2b-bc95764f91f8 name: MergeContent class: org.apache.nifi.processors.standard.MergeContent scheduling strategy: EVENT_DRIVEN auto-terminated relationships list: - original Properties: Attribute Strategy: Keep Only Common Attributes Maximum number of Bins: 100 Minimum Group Size: 0 Max Bin Age: 1 min Minimum Number of Entries: 1000000 Maximum Group Size: 1000000 Maximum Number of Entries: 1000000 Merge Strategy: Bin-Packing Algorithm - id: 69335770-ee29-11eb-9a03-0242ac130003 name: CompressContent class: org.apache.nifi.processors.standard.CompressContent scheduling strategy: EVENT_DRIVEN auto-terminated relationships list: - failure Properties: Compression Level: 6 Compression Format: gzip UpdateFileName: false - id: fe198bd9-2a1c-316e-0000-000000000000 name: PutS3Object class: org.apache.nifi.minifi.azure.processors.PutS3Object scheduling strategy: EVENT_DRIVEN auto-terminated relationships list: - success Properties: Bucket: test_bucket AWS Credentials Provider service: AWSCredentialsService Controller Services: - name: AWSCredentialsService id: 2094d776-2006-4d02-9bb9-28eac9d0fc95 class: org.apache.nifi.minifi.aws.controllers.AWSCredentialsService Properties: Use Default Credentials: 'true' # Can be used in Amazon EKS to retrieve credentials from metadata otherwise use your AWS Access Key and Secret Key Connections: - id: 99f617e7-49a1-6078-8534-26af7d56ca08 name: TailFile/success/MergeContent source name: TailFile source relationship names: - success destination name: MergeContent - id: 24d6be1e-ee29-11eb-9a03-0242ac130003 name: MergeContent/merged/CompressContent source name: MergeContent source relationship names: - merged destination name: CompressContent - id: 67ea5c91-446a-393b-6274-b6fae2f475a2 name: CompressContent/success/PutS3Object source name: CompressContent source relationship names: - success destination name: PutS3Object Remote Process Groups: []
In the pod specification you have to define the application and the MiNiFi containers to mount
a shared volume of the logs to be collected. In the following example the NiFi logs are mounted
to both NiFi and MiNiFi containers. MiNiFi also mounts the previously detailed configuration from
the
/root/minifi-config
host
directory.apiVersion: v1 kind: Pod metadata: name: log-collection-minifi-pod namespace: default spec: containers: - name: nifi image: apache/nifi:latest volumeMounts: - name: nifi-logs mountPath: /opt/nifi/nifi-current/logs - name: sidecar-minifi image: container.repo.cloudera.com/cloudera/apacheminificpp:latest volumeMounts: - name: nifi-logs mountPath: /nifi-logs - name: minifi-config mountPath: /opt/minifi/minifi-current/conf/ volumes: - name: nifi-logs emptyDir: {} - name: minifi-config hostPath: path: /root/minifi-config
Another option would be to instead of mounting the configuration files from the host directory
they can be configured in a Kubernetes configuration map beforehand and this way they can be
mounted individually. A configuration map definition template would look something like
this:
apiVersion: v1 data: minifi-log.properties: | <minifi log properties file content> minifi.properties: | <minifi properties file content> config.yml: | <config file content> kind: ConfigMap metadata: labels: k8s-app: minifi-log-collection name: minifi-log-collection-config namespace: default
With the configuration map defined, you can change the previous pod definition in the following
way to use the configuration map
volume:
apiVersion: v1 kind: Pod metadata: name: log-collection-minifi-pod namespace: default spec: containers: - name: nifi image: apache/nifi:latest volumeMounts: - name: nifi-logs mountPath: /opt/nifi/nifi-current/logs - name: sidecar-minifi image: container.repo.cloudera.com/cloudera/apacheminificpp:latest volumeMounts: - name: nifi-logs mountPath: /nifi-logs - name: minificonfig mountPath: /opt/minifi/minifi-current/conf/config.yml subPath: config.yml - name: minificonfig mountPath: /opt/minifi/minifi-current/conf/minifi-log.properties subPath: minifi-log.properties volumes: - name: nifi-logs emptyDir: {} - configMap: defaultMode: 420 name: minifi-log-collection-config name: minificonfig
You can find an example of sidecar Kubernetes YAML configuration file here with the previous pod and
configuration map definitions to try. The suggested changes are marked with the
TODO
keyword.