Enabling the libgfortran library
This procedure shows how to use Cloudera Manager to enable the
libgfortran
math library to accelerate Spark ML applications.
By: Zuling Kang, Senior Solutions Architect at Cloudera, Inc.
-
Enable the
libgfortran
4.8 library on every CDH node. For example, in RHEL, run the following command on each node:yum -y install libgfortran
-
Install the GPLEXTRAS parcel in Cloudera Manager, and activate it:
- To install the GPLEXTRAS parcel, see Installing the GPL Extras Parcel in the Cloudera Manager documentation.
- To activate the package, see Activating a Parcel.
-
After activating the GPLEXTRAS parcel, in Cloudera Manager, navigate to
to confirm that the GPLEXTRAS parcel is activated:
The GPLEXTRAS parcel acts as the wrapper for
libgfortran
.
- Restart the appropriate CDH services as guided by Cloudera Manager.
-
As soon as the restart is complete, use the Spark shell to verify that the native library is being loaded by using the following commands:
scala> import com.github.fommil.netlib.BLAS import com.github.fommil.netlib.BLAS scala> println(BLAS.getInstance().getClass().getName()) 18/12/23 06:29:45 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 18/12/23 06:29:45 INFO jni.JniLoader: successfully loaded /tmp/jniloader6112322712373818029netlib-native_ref-linux-x86_64.so com.github.fommil.netlib.NativeRefBLAS
You might be wondering why there is still a warning message about
NativeSystemBLAS
failing to load. Don’t worry about this. It is there because we are only setting the native library that Spark uses, not for system-wide use. You can safely ignore this warning.