Cloudera Docs

Initial setup

Scripting allows you to integrate custom code into MiNiFi C++ agents using Python. You can use either the ExecuteScript processor or custom Python processors.

Learn how to install and enable the required Python version on all target systems.

Enabling Python scripting for MiNiFi version 1.23.02 and higher

On Linux

  • To use the Python processors, copy libminifi-python-script-extension.so located in the nifi-minifi-cpp-...-extra-extensions-centos-bXX.tar.gz archive to the extensions/ folder.

  • To use the ExecuteScript processor with Python, copy libminifi-python-script-extension.so and libminifi-script-extension.so located in the nifi-minifi-cpp-...-extra-extensions-centos-bXX.tar.gz archive to the extensions/ folder.

Requirements

Python scripting extension needs the generic Python3 library (libpython3.so) with a minimum version 3.6.

Anaconda

Before starting MiNiFi, set the LD_LIBRARY_PATH environment variable to the lib folder of the installed Python.

export LD_LIBRARY_PATH="${CONDA_PREFIX}/lib"
PyEnv

Before starting MiNiFi, set the LD_LIBRARY_PATH environment variable to the lib folder of the installed Python.

export LD_LIBRARY_PATH="${PYENV_ROOT}/versions/${PY_VERSION}/lib"
RHEL/CentOS
yum install python3-libs
Debian/Ubuntu
apt install libpython3-dev

Debian/Ubuntu does not provide the generic Python3 library (libpython3.so), but the extension works with the specific libraries as well. To use the extension on a system where the generic libpython3.so is not available, patch the extension to use the specific library.

patchelf extensions/libminifi-python-script-extension.so --replace-needed libpython3.so libpython3.9.so

On Windows

The Python extension is part of the normal MiNiFi C++ MSI installer, but it is not enabled by default. You need to enable it during installation if you want to use it.

Requirements

Python scripting extension needs the generic Python3 library (python3.dll) with a minimum version 3.6.

Install Python through the GUI installer on https://www.python.org/downloads/windows, or through winget.

winget install -e --id Python.Python.3.11

Enabling Python scripting for MiNiFi versions 1.22.08 and 1.22.10

To use scripting, you need to install the required Python version on all target systems.

On Linux

Python 3.6 is required, which is available on CentOS 7.

If you are downloading MiNiFi C++ for Linux:

  1. Find the nifi-minifi-cpp-...-extra-extensions-centos-bXX.tar.gz file. This file contains the libminifi-script-extensions.so file.

  2. Copy the libminifi-script-extensions.so file to the extensions/ directory so that the MiNiFi C++ agent can load it on startup.

There is an additional workaround required to make scripting work if you are using MiNiFi version 1.22.08. You need to patch the MiNiFi binary to link to Python:patchelf --add-needed libpython3.6m.so MINIFI_HOME/bin/minifi

On Windows

Python 3.10 and the 64 bit version of the agent are required. The Python extension is already part of the normal MiNiFi C++ MSI installer, but it is not enabled by default. You need to enable it during installation.

Before version 1.23.02, the script-extension was tightly coupled with Python and Lua, so you also need to install the Lua library on all target systems.