Using Native Math Libraries to Accelerate Spark Machine Learning Applications

Anand Iyer and Vikram Saletore showed in their engineering blog post that native math libraries like OpenBLAS and Intel’s Math Kernel Library (MKL) accelerate the training performance of Spark ML. However, the range of acceleration varies from model to model.

As for the matrix factorization model used in recommendation systems (the Alternating Least Squares (ALS) algorithm), both OpenBLAS and Intel’s MKL yield model training speeds that are 4.3 times faster than with the F2J implementation. Others, like the Latent Dirichlet Allocation (LDA), the Primary Component Analysis (PCA), and the Singular Value Decomposition (SVD) algorithms show 56% to 72% improvements for Intel MKL, and 10% to 50% improvements for OpenBLAS.

However, the blog post also demonstrates that there are some algorithms, like Random Forest and Gradient Boosted Tree, that receive almost no speed acceleration after enabling OpenBLAS or MKL. The reasons for this are largely that the training set of these tree-based algorithms are not vectors. This indicates that the native libraries, either OpenBLAS or MKL, adapt better to algorithms whose training sets can be operated on as vectors and are computed as a whole. It is more effective to use math acceleration for algorithms that operate on training sets using matrix operations.

1. Enable the libgfortran 4.8 library on every CDH node. For example, in RHEL, run the following command on each node:

   yum -y install libgfortran
               

2. Install the GPLEXTRAS parcel in Cloudera Manager, and activate it:

   1. To install the GPLEXTRAS parcel, see Installing the GPL Extras Parcel in the Cloudera Manager documentation.
   2. To activate the package, see Activating a Parcel.

   3. After activating the GPLEXTRAS parcel, in Cloudera Manager, navigate to Hosts > Parcels to confirm that the GPLEXTRAS parcel is activated:

      
      
      
      
      

      The GPLEXTRAS parcel acts as the wrapper for libgfortran.

3. Restart the appropriate CDH services as guided by Cloudera Manager.

4. As soon as the restart is complete, use the Spark shell to verify that the native library is being loaded by using the following commands:

   scala> import com.github.fommil.netlib.BLAS
   import com.github.fommil.netlib.BLAS
   
   scala> println(BLAS.getInstance().getClass().getName())
   18/12/23 06:29:45 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
   18/12/23 06:29:45 INFO jni.JniLoader: successfully loaded /tmp/jniloader6112322712373818029netlib-native_ref-linux-x86_64.so
   com.github.fommil.netlib.NativeRefBLAS
                   

   You might be wondering why there is still a warning message about NativeSystemBLAS failing to load. Don’t worry about this. It is there because we are only setting the native library that Spark uses not for system-wide use. You can safely ignore this warning.

1. Intel provides the MKL native library as a Cloudera Manager parcel on its website. You can add it as a remote parcel repository in Cloudera Manager. Then you can download the library and activate it:

   1. In Cloudera Manager, navigate to Hosts > Parcels.
   2. Select Configuration.

   3. In the section, Remote Parcel Repository URLs, click the plus sign and add the following URL:

      http://parcels.repos.intel.com/mkl/latest
                    

   4. Click Save Changes, and then you are returned to the page that lists available parcels.

   5. Click Download for the mkl parcel:

      
      
      
      
      
   6. Click Distribute, and when it finishes distributing to the hosts on your cluster, click Activate.

2. The MKL parcel is only composed of Linux shared library files (.so files), so to make it accessible to the JVM, a JNI wrapper has to be made. To make the wrapper, use the following MKL wrapper parcel. Use the same procedure described in Step 1 to add the following link to the Cloudera Manager parcel configuration page, download the parcel, distribute it among the hosts and then activate it:

   https://raw.githubusercontent.com/Intel-bigdata/mkl-wrappers-parcel-repo/master/
               

3. Restart the corresponding CDH services as guided by Cloudera Manager, and redeploy the client configuration if needed.

4. In Cloudera Manager, add the following configuration information into the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf:

   spark.driver.extraJavaOptions=-Dcom.github.fommil.netlib.BLAS=com.intel.mkl.MKLBLAS -Dcom.github.fommil.netlib.LAPACK=com.intel.mkl.MKLLAPACK
   spark.driver.extraClassPath=/opt/cloudera/parcels/mkl_wrapper_parcel/lib/java/mkl_wrapper.jar
   spark.driverEnv.MKL_VERBOSE=1
   spark.executor.extraJavaOptions=-Dcom.github.fommil.netlib.BLAS=com.intel.mkl.MKLBLAS -Dcom.github.fommil.netlib.LAPACK=com.intel.mkl.MKLLAPACK
   spark.executor.extraClassPath=/opt/cloudera/parcels/mkl_wrapper_parcel/lib/java/mkl_wrapper.jar
   spark.executorEnv.MKL_VERBOSE=1
               

   This configuration information instructs the Spark application to load the MKL wrapper and use MKL as the default native library for Spark ML.

5. Open the Spark shell again to verify the native library, and you should see the following output:

   scala> import com.github.fommil.netlib.BLAS
   import com.github.fommil.netlib.BLAS
   
   scala> println(BLAS.getInstance().getClass().getName())
   com.intel.mkl.MKLBLAS