Sessions example for the CDE CLI
In this example, a Session is created using the Cloudera Data Engineering (CDE) CLI.
> cde session create --name cli-submission-6 --type spark-scala --conf "key=value"
> cde session list
+-------------------+-----------+-------------+-------------+----------------------+----------------------+-------------------------+
| NAME | STATE | TYPE | DESCRIPTION | CREATED | LAST UPDATED | CREATOR |
+-------------------+-----------+-------------+-------------+----------------------+----------------------+-------------------------+
| cli-submission-1 | preparing | spark-scala | | 2023-02-15T08:47:41Z | 2023-02-15T08:47:41Z | csso_surya.balakrishnan |
| cli-submission-19 | preparing | spark-scala | | 2023-02-15T07:50:08Z | 2023-02-15T07:50:08Z | csso_surya.balakrishnan |
+-------------------+-----------+-------------+-------------+----------------------+----------------------+-------------------------+
> cde session describe --name techf-run-1
{
"name": "techf-run-1",
"state": "idle",
"type": "pyspark",
"created": "2023-02-15T13:19:22Z",
"lastStateUpdated": "2023-02-15T13:19:43Z",
"creator": "csso_surya.balakrishnan",
"interactiveSpark": {
"submitID": "95",
"kind": "pyspark",
"driverMemory": "1g",
"driverCores": 1,
"executorMemory": "1g",
"executorCores": 1,
"numExecutors": 1,
"conf": {
"spark.kubernetes.driver.annotation.sidecar.istio.io/inject": "false",
"spark.kubernetes.executor.annotation.sidecar.istio.io/inject": "false",
"spark.pyspark.python": "python3"
}
}
}
> cde session connect --name tech-forum
Starting REPL...
Connected to Cloudera Data engineering...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.3.0
/_/
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 11.0.16.1)
Type in expressions to have them evaluated.
>>> spark.sql("show tables").show()
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
+--------+---------+-----------+
>>> testDF = spark.createDataFrame([(1,"111"), (2,"111"), (3,"222"), (4,"222"), (5,"222"), (6,"111"), (7,"333"), (8,"444")], ["id", "d_id"])
>>> ^C
> cde session delete --name tech-forum
