Sample code to connect to Spark Connect Session
After configuring Spark Connect Sessions, learn how you can run the CLI commands from a remote Python host to connect to a session and execute Spark SQL commands through an example.
You can use the following sample code to connect to the Spark Connect session. Use the spark variable to interact with Spark as you connect to the CDE jobs or sessions.
> python
Python 3.9.13 (main, Jul 29 2022, 12:22:24)
[Clang 13.0.0 (clang-1300.0.27.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from cde import CDESparkConnectSession
>>> spark = CDESparkConnectSession.builder.sessionName('connect-session').get()
>>> spark.version
'3.4.1.1.20.7180.0-33'
>>> spark.sql("use retaildb").show()
++
||
++
++
>>> spark.sql("select * from products_external").show()
+----------+-------------------+--------------------+-------------------+-------------+--------------------+
|product_id|product_category_id| product_name|product_description|product_price| product_image|
+----------+-------------------+--------------------+-------------------+-------------+--------------------+
| 1| 2|Quest Q64 10 FT. ...| | 59.98|http://images.acm...|
| 2| 2|Under Armour Men'...| | 129.99|http://images.acm...|
| 3| 2|Under Armour Men'...| | 89.99|http://images.acm...|
| 4| 2|Under Armour Men'...| | 89.99|http://images.acm...|
| 5| 2|Riddell Youth Rev...| | 199.99|http://images.acm...|
| 6| 2|Jordan Men's VI R...| | 134.99|http://images.acm...|
| 7| 2|Schutt Youth Recr...| | 99.99|http://images.acm...|
| 8| 2|Nike Men's Vapor ...| | 129.99|http://images.acm...|
| 9| 2|Nike Adult Vapor ...| | 50.0|http://images.acm...|
| 10| 2|Under Armour Men'...| | 129.99|http://images.acm...|
| 11| 2|Fitness Gear 300 ...| | 209.99|http://images.acm...|
| 12| 2|Under Armour Men'...| | 139.99|http://images.acm...|
| 13| 2|Under Armour Men'...| | 89.99|http://images.acm...|
| 14| 2|Quik Shade Summit...| | 199.99|http://images.acm...|
| 15| 2|Under Armour Kids...| | 59.99|http://images.acm...|
| 16| 2|Riddell Youth 360...| | 299.99|http://images.acm...|
| 17| 2|Under Armour Men'...| | 129.99|http://images.acm...|
| 18| 2|Reebok Men's Full...| | 29.97|http://images.acm...|
| 19| 2|Nike Men's Finger...| | 124.99|http://images.acm...|
| 20| 2|Under Armour Men'...| | 129.99|http://images.acm...|
+----------+-------------------+--------------------+-------------------+-------------+--------------------+
only showing top 20 rows