Enabling the libgfortran library
This procedure shows how to use Cloudera Manager to enable the
libgfortran math library to accelerate Spark ML applications.
libgfortran4.8 library on every CDH node. For example, in RHEL, run the following command on each node:
yum -y install libgfortran
Install the GPLEXTRAS parcel in Cloudera Manager, and activate it:
- To install the GPLEXTRAS parcel, see Installing the GPL Extras Parcel in the Cloudera Manager documentation.
- To activate the package, see Activating a Parcel.
After activating the GPLEXTRAS parcel, in Cloudera Manager, navigate toto confirm that the GPLEXTRAS parcel is activated:
The GPLEXTRAS parcel acts as the wrapper for
- Restart the appropriate CDH services as guided by Cloudera Manager.
As soon as the restart is complete, use the Spark shell to verify that the native library is being loaded by using the following commands:
scala> import com.github.fommil.netlib.BLAS import com.github.fommil.netlib.BLAS scala> println(BLAS.getInstance().getClass().getName()) 18/12/23 06:29:45 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 18/12/23 06:29:45 INFO jni.JniLoader: successfully loaded /tmp/jniloader6112322712373818029netlib-native_ref-linux-x86_64.so com.github.fommil.netlib.NativeRefBLAS
You might be wondering why there is still a warning message about
NativeSystemBLASfailing to load. Don’t worry about this. It is there because we are only setting the native library that Spark uses, not for system-wide use. You can safely ignore this warning.