Enabling the libgfortran library

This procedure shows how to use Cloudera Manager to enable the libgfortran math library to accelerate Spark ML applications.

By: Zuling Kang, Senior Solutions Architect at Cloudera, Inc.

  1. Enable the libgfortran 4.8 library on every CDH node. For example, in RHEL, run the following command on each node:

    yum -y install libgfortran
  2. Install the GPLEXTRAS parcel in Cloudera Manager, and activate it:

    1. To install the GPLEXTRAS parcel, see Installing the GPL Extras Parcel in the Cloudera Manager documentation.
    2. To activate the package, see Activating a Parcel.
    3. After activating the GPLEXTRAS parcel, in Cloudera Manager, navigate to Hosts > Parcels to confirm that the GPLEXTRAS parcel is activated:

      The GPLEXTRAS parcel acts as the wrapper for libgfortran.

  3. Restart the appropriate CDH services as guided by Cloudera Manager.
  4. As soon as the restart is complete, use the Spark shell to verify that the native library is being loaded by using the following commands:

    scala> import com.github.fommil.netlib.BLAS
    import com.github.fommil.netlib.BLAS
    scala> println(BLAS.getInstance().getClass().getName())
    18/12/23 06:29:45 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
    18/12/23 06:29:45 INFO jni.JniLoader: successfully loaded /tmp/jniloader6112322712373818029netlib-native_ref-linux-x86_64.so

    You might be wondering why there is still a warning message about NativeSystemBLAS failing to load. Don’t worry about this. It is there because we are only setting the native library that Spark uses, not for system-wide use. You can safely ignore this warning.