Jupyter Enterprise Gateway

The demo-base image is a Debian-based Miniconda environment designed for demonstrating remote kernel capabilities with Jupyter Enterprise Gateway. It includes:

• Base OS/Python: continuumio/miniconda3:24.1.2-0 (Python 3.11)
• Big Data Stack: Hadoop 3.3.1 and Apache Spark 3.2.1 (controlled by SPARK_VERSION in the top-level Makefile)
• Java: OpenJDK 11 runtime (openjdk-11-jdk-headless)
• R Support: Miniconda with R packages (r-devtools, r-stringr, r-argparse)
• Kernels: Apache Toree 0.5.0-incubating and ipykernel<7 (pinned to ensure compatibility with the Enterprise Gateway process-proxy model)
• Users: jovyan service user (UID 1000) with password-less SSH, plus system users elyra, bob, and alice.
• Storage: HDFS home directories are automatically created for each system user at container startup.

This Docker image enables Jupyter Enterprise Gateway support within Kubernetes or Docker Swarm clusters. It is built on jupyter/minimal-notebook:2023-03-13 and includes Apache Spark 3.2.1.

Key Features:

• Supports launching and distributing Python, R, and Toree kernels across a managed cluster.
• Includes a pinned version of ipykernel<7 to ensure kernel idle/busy transitions function correctly under the process-proxy model.

Note: If you require a YARN-based image, use elyra/enterprise-gateway-demo instead.

The Kernel Image Puller (KIP) is a utility image designed to work within Kubernetes or Docker Swarm clusters. Its primary responsibility is to contact a configured Jupyter Enterprise Gateway instance and automatically pull the required kernel-based images to every node in the cluster.

Key Benefits:

• Automatic Node Provisioning: When adding new nodes to a cluster, KIP ensures kernel images are automatically populated on those nodes.
• Dynamic Kernelspec Support: When you configure new kernelspecs that use different images, KIP ensures those images are pulled to all cluster nodes automatically.

This Docker image provides an IPython kernel (with debugger support) designed to be launched via Jupyter Enterprise Gateway within a Kubernetes cluster. It is built on top of elyra/kernel-py and includes Apache Spark and the OpenJDK 8 runtime (openjdk-8-jdk).

Key Features:

• IPython kernel support including debugger capabilities.
• Pre-installed data science libraries.
• Spark on Kubernetes support directly from within a Jupyter Notebook.
• Bundled Apache Spark (currently version 3.2.1).

Warning: Using this kernel within Spark inside a Docker Swarm configuration may not yield expected results; it is optimized for Kubernetes.

This Docker image provides an IRkernel (R kernel) that can be launched via Jupyter Enterprise Gateway within a Kubernetes cluster. It integrates Apache Spark, allowing you to run Spark on Kubernetes directly from a Jupyter Notebook.

Key Features:

• IRkernel support.
• Spark on Kubernetes support from within Jupyter.
• Bundled OpenJDK 8 runtime (openjdk-8-jdk) required by Apache Spark.

Important Notes:

• The image is built on elyra/kernel-r.
• The bundled Apache Spark version is currently 3.2.1 (controlled by SPARK_VERSION in the top-level Makefile).
• Compatibility Warning: Using this kernel within Spark in a Docker Swarm configuration may not yield expected results; it is designed for Kubernetes.

This Docker image enables Scala support via the Apache Toree kernel when launched through Jupyter Enterprise Gateway in a Kubernetes or Docker Swarm cluster. It is built on top of elyra/spark (currently using Spark version 3.2.1).

Capabilities:

• Scala (Toree) kernel support.
• Spark on Kubernetes support directly from within a Jupyter Notebook.

Note: Using this kernel within Spark in a Docker Swarm configuration may not yield expected results.

The Enterprise Gateway REST API allows developers to author new applications that interact with the gateway. While several endpoints exist, most application logic should focus on two primary endpoints:

• /api/kernelspecs: Used to retrieve a list of available kernel specifications.
• /api/kernels: Used to start, stop, interrupt, and restart kernels.

Note that while the /api/sessions endpoint can be used to manage a kernel's lifecycle, it is often unnecessary for standard application flows. For instance, Jupyter Notebook and JupyterLab use /api/kernelspecs to discover kernels and /api/kernels for lifecycle management, while keeping the 'session' state on the client side.

Jupyter Enterprise Gateway is a headless web server that provides a Kernel as a Service model. It allows remote notebooks to launch kernels on behalf of users within managed-cluster environments (such as Kubernetes, Hadoop YARN, or Docker Swarm).

Unlike the default Jupyter framework which runs kernels locally on the web server, Enterprise Gateway distributes kernels across a compute cluster. This enables:

• Resource Optimization: Better management of limited resources like GPUs and large memory.
• Scalability: Support for a large number of simultaneous users and active kernels.
• High Availability: Support for Disaster Recovery scenarios where a new Gateway server can be spun up to service existing remote kernels.
• Granular Security: Improved security controls for enterprise, scientific, and academic implementations.

A kernel launcher must listen on its communication port for specific JSON messages sent by Enterprise Gateway to manage the kernel lifecycle:

Interrupt Signals

To interrupt current processing (equivalent to SIGINT), Enterprise Gateway sends a signal event:

• Format: {"signum": n}
• Common values: n=2 (for SIGINT).
• Polling: Enterprise Gateway sends {"signum": 0} to perform a poll() check to see if the process is still alive.

Shutdown Requests

When the process proxy is performing final cleanup and needs the launcher to exit, it sends a shutdown request:

• Format: {"shutdown": 1}
• Action: Upon receiving this, the launcher should stop listening on the communication socket and exit.