Apache Spark Component Guide
Also available as:
PDF
loading table of contents...

Livy API Reference for Batch Jobs

GET /batches returns all active batch sessions.

Request ParametersDescriptionType
fromStarting index used to fetch sessionsint
sizeNumber of sessions to fetchint
ResponseDescriptionType
fromStarting index of fetched sessionsint
totalNumber of sessions fetchedint
sessionsList of active batch sessionslist

GET /batches/{batchId} returns the batch session information as a batch object.

GET /batches/{batchId}/state returns the state of batch session:

ResponseDescriptionType
idA non-negative integer that represents a specific batch sessionint
stateThe current state of batch sessionstring

GET /batches/{batchId}/log retrieves log records for the specified batch session.

Request ParametersDescriptionType
fromOffset into log fileint
sizeMax number of log lines to returnint
ResponseDescriptionType
idA non-negative integer that represents a specific batch sessionint
fromOffset from start of the log fileint
sizeNumber of log records returnedint
logLog recordslist of strings

POST /batches creates a new batch environment and runs a specified application:

Request BodyDescriptionType
fileFile containing the application to run (required)path
proxyUserUser ID to impersonate when running the jobstring
classNameApplication Java or Spark main classstring
argsCommand line arguments for the applicationlist of strings
jarsJar files to be used in this sessionlist of strings
pyFilesPython files to be used in this sessionlist of strings
filesOther files to be used in this sessionlist of strings
driverMemoryAmount of memory to use for the driver processstring
driverCoresNumber of cores to use for the driver processint
executorMemoryAmount of memory to use for each executor processstring
executorCoresNumber of cores to use for each executorint
numExecutorsNumber of executors to launch for this sessionint
archivesArchives to be used in this sessionlist of strings
queueThe name of the YARN queue to which the job should be submittedstring
nameName of this sessionstring
confSpark configuration propertiesMap of key=val
ResponseDescriptionType
batch object (for more information see "Livy Batch Object")The created batch objectbatch object

DELETE /batches/{batchId} terminates the Batch job.