Skip to main content

Part of Using Databricks in the NHS England Secure Data Environment

Pulling tables into Python using PySpark

Previous Chapter

Using cells

Current Chapter

Current chapter – Pulling tables into Python using PySpark

View all

Next Chapter

Apache Spark and Spark SQL

PySpark is the Python Application Programming Interface (API) written in Python to support Apache Spark. Apache Spark is the distributed framework used to handle big data analysis.

When pulling tables from Databricks into Python, you need to go via Spark using "PySpark": spark.table('table_name').

Dataframes provide a domain specific language for structured data manipulation. 

The following example shows how to pull a table into Python using PySpark. 

example showing how to pull a table into Python using PySpark

Last edited: 11 January 2024 1:30 pm

Previous Chapter

Using cells

Next Chapter

Apache Spark and Spark SQL

Chapters

  1. Using Databricks in the NHS England Secure Data Environment
  2. Logging in
  3. Activating your databases
  4. Navigating Databricks
  5. Using notebooks and clusters
  6. Using the SDE Package Manager
  7. Using cells
  8. Pulling tables into Python using PySpark
  9. Apache Spark and Spark SQL
  10. Improving code efficiency
  11. Working collaboratively
  12. Copying and pasting into SDE
  13. Output your results
  14. Creating reusable processes
  15. Best practice and further information