sparkmagic

Sparkmagic is a set of tools for interactively working with remote Spark clusters in Jupyter notebooks. It interacts with remote Spark clusters through a REST server.

Supported server implementations:

• Livy: For running interactive sessions on Yarn.
• Lighter: For running interactive sessions on Yarn or Kubernetes (supports PySpark sessions only).
• Ilum: For running interactive sessions on Yarn or Kubernetes.

Key features include:

• Running Spark code in multiple languages via Livy.
• Automatic creation of SparkContext (sc) and HiveContext (sqlContext).
• Using %%sql magic for SparkSQL queries.
• Automatic visualization of SQL queries in PySpark, Spark, and SparkR kernels.
• Accessing Spark application info and logs via %%info magic.
• Capturing SQL query output as Pandas dataframes.
• Sending local files or dataframes to a remote cluster.

The sparkmagic project is licensed under the Modified BSD License (also known as New, Revised, or 3-Clause BSD).

Key conditions for redistribution:

• Source code: Must retain the original copyright notice, the list of conditions, and the disclaimer.
• Binary form: Must reproduce the copyright notice, the list of conditions, and the disclaimer in the documentation and/or other materials provided with the distribution.
• Endorsement: The name of the Jupyter Development Team or its contributors may not be used to endorse or promote derivative products without specific prior written permission.

Disclaimer: The software is provided "as is" without warranty of any kind.

Sparkmagic uses Livy (a REST server for Spark) to execute user code remotely.

Workflow:

1. User code is sent to the remote Spark driver via Livy.
2. Sparkmagic automatically collects the output as plain text or a JSON document.
3. The library parses this data to display results as formatted text or Pandas dataframes on the client side.

Key Concepts:

• Remote Execution: No Spark components are required on the Jupyter server itself.
• Multi-language: Supports Python, Python3, Scala, and R kernels.
• Multiple Endpoints: A single notebook can interact with different remote clusters using different endpoints.
• Data Serialization: Because code runs on a remote driver, all structured data must be serialized to JSON and parsed by Sparkmagic. For client-side data manipulation, you must use Python in %%local mode.

To set up a local development environment for Sparkmagic, follow these steps:

1. Clone the repository:

   git clone https://github.com/jupyter-incubator/sparkmagic

2. Install the local packages in editable mode:

   pip install -e hdijupyterutils 
   pip install -e autovizwidget
   pip install -e sparkmagic

Alternative: Using Poetry If you prefer Poetry, use the following commands:

poetry install
# If numpy or pandas installation fails, run:
# poetry run pip install numpy pandas
# then re-run:
# poetry install

Running Tests Run unit tests using pytest:

# Standard install
pytest

# Poetry install
poetry run pytest

For testing or development, you can use the provided docker-compose.yml to launch a full stack including a Jupyter notebook and a Livy server backed by a local-mode Spark instance.

1. Ensure Docker and Docker Compose are installed.
2. Build and start the services:

   docker compose build
   docker compose up

3. Access Jupyter at http://localhost:8888.
4. Configure a sparkmagic endpoint at http://spark:8998 within the notebook to launch Scala or Python sessions.

To shut down, use Ctrl-C and optionally run docker compose down to remove containers.

If you are developing Sparkmagic and want to test changes in the Docker container without publishing to PyPI, set the dev_mode build argument to true in docker-compose.yml and rebuild.

This installs your local versions of autovizwidget, hdijupyterutils, and sparkmagic using the editable flag (-e). This allows you to make edits directly to the libraries within the Jupyterlab docker service for real-time debugging.

Note: You must re-run docker compose build before spinning up services for changes to take effect.

You can use custom HTTP transport adapters to control request behavior (e.g., adding custom headers, implementing mTLS, or retries) for specific domains.

Add the http_session_config section to your ~/.sparkmagic/config.json to map a URL prefix to a custom adapter class.

{
  "http_session_config": {
    "adapters": [
      {
        "prefix": "http://",
        "adapter": "customadapter.customadapter.CustomaAapter"
      }
    ]
  }
}

When using the PySpark kernel, you do not need to manually create a SparkContext or HiveContext. They are automatically created when you run your first code cell. The following variable names are used:

• sc: SparkContext
• sqlContext: HiveContext

When using Spark kernel notebooks, you do not need to manually create a SparkContext or a HiveContext. They are automatically created when you run your first code cell. The following variable names are used for these contexts:

• sc: SparkContext
• sqlContext: HiveContext