API Compatibility changes in 7.1.9 for Spark
Removed or Modified APIs in CDP 7.1.9 for Spark and recommendations for how to handle them.
Apache Base Version of Spark in 7.1.7 SP2 was 2.4.7 and Apache Base Version of Spark in 7.1.9 is 2.4.8
Removed APIs in 7.1.9
The following APIs are no longer available for Spark in CDP 7.1.9
UnsafeInMemorySorter.free
Method Removed
- Package Name
- org.apache.spark.util.collection.unsafe.sort
- Effect
- A client program may be interrupted by NoSuchMethodError exception.
- Reason for change
- SPARK-32901
- Recommendation
- Use UnsafeInMemorySorter.freeMemory method instead of UnsafeInMemorySorter.free.
- Recompilation Required?
- Yes
UnsafeInMemorySorter.reset
Method Removed
- Package Name
- org.apache.spark.util.collection.unsafe.sort
- Effect
- A client program may be interrupted by NoSuchMethodError exception.
- Reason for change
- SPARK-32901
- Recommendation
- Method UnsafeInMemorySorter.reset() removed. Please see its code for alternatives.
- Recompilation Required?
- Yes
ShuffleIndexInformation.getSize
Method Removed
- Package Name
- org.apache.spark.network.shuffle
- Effect
- A client program may be interrupted by NoSuchMethodError exception.
- Reason for change
- SPARK-33206
- Recommendation
- Method getSize removed. Use getRetainedMemorySize() instead.
- Recompilation Required?
- Yes
CompactibleFileStreamLog.compactLogs
Method Removed
- Package Name
- org.apache.spark.sql.execution.streaming
- Effect
- A client program may be interrupted by NoSuchMethodError exception.
- Reason for change
- SPARK-30462
- Recommendation
- Method compactLogs has been removed. Use shouldRetain(log: T) instead
- Recompilation Required?
- Yes
FileStreamSinkLog.compactLogs
Method Removed
- Package Name
- org.apache.spark.sql.execution.streaming
- Effect
- A client program may be interrupted by NoSuchMethodError exception.
- Reason for change
- SPARK-30462
- Recommendation
- Method compactLogs has been removed. Use shouldRetain(log: T) instead
- Recompilation Required?
- Yes
FileStreamSourceLog.compactLogs
Method Removed
- Package Name
- org.apache.spark.sql.execution.streaming
- Effect
- A client program may be interrupted by NoSuchMethodError exception.
- Reason for change
- SPARK-30462
- Recommendation
- Method compactLogs has been removed. Use shouldRetain(log: T) instead
- Recompilation Required?
- Yes
Modified APIs in 7.1.9
The following APIs have been modified for Spark and include a description of the impact of the modifiaction on their use.
UnsafeSorterIterator
Abstract method long getCurrentPageNumber() has been added to this class.
- Package Name
- org.apache.spark.util.collection.unsafe.sort
- Effect
- Recompilation of a client program may be terminated with the message: a client class C is not abstract and does not override abstract method getCurrentPageNumber() in org.apache.spark.util.collection.unsafe.sort.UnsafeSorterIterator.
- Reason for change
- SPARK-32900
- Recommendation
- New abstract method getCurrentPageNumber added to UnsafeSorterIterator. Please recompile your application if you want to use this.
- Recompilation Required?
- Yes
CompactibleFileStreamLog<T>
Abstract method SeqT compactLogs(SeqT) has been removed from this class.
- Package Name
- org.apache.spark.sql.execution.streaming
- Effect
- Recompilation of a client program may be terminated with the message: cannot find method compactLogs(SeqT) in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLogT.
- Reason for change
- SPARK-30462
- Recommendation
- Method compactLogs has been removed. Use shouldRetain(log: T) instead
- Recompilation Required?
- Yes
CompactibleFileStreamLog<T>
Abstract method SeqT compactLogs(SeqT) has been removed from this class.
- Package Name
- org.apache.spark.sql.execution.streaming
- Effect
- A client program may be interrupted by NoSuchMethodError exception.
- Reason for change
- SPARK-30462
- Recommendation
- Method compactLogs has been removed. Use shouldRetain(log: T) instead
- Recompilation Required?
- Yes
