findspark Documentation

To ensure environment variables are set whenever a new shell is opened, you can use findspark.init() with edit_rc=True. This will attempt to add the necessary configurations to your .bashrc file if it exists.

import findspark
findspark.init('/path/to/spark_home', edit_rc=True)

You can automatically configure an IPython profile to set environment variables and import pyspark every time IPython starts. To do this, set edit_profile=True in the init() call. Note that you should target a specific profile using ipython --profile=<profile_name>.

# First, ensure the profile exists
ipython --profile=myprofile

# Then, initialize with profile editing enabled
import findspark
findspark.init('/path/to/spark_home', edit_profile=True)

Since PySpark is not on sys.path by default, you can use findspark.init() to add it to your path at runtime. By default, it uses the SPARK_HOME environment variable or searches for common installation locations (e.g., /usr/local/opt/apache-spark/libexec on OS X via Homebrew).

import findspark
findspark.init()

import pyspark
sc = pyspark.SparkContext(appName="myAppName")

To check which Spark installation path findspark has identified, call the find() method.

import findspark
print(findspark.find())

If you want to bypass automatic detection, you can pass the specific path to your Spark installation as the first argument to findspark.init().

findspark.init('/path/to/spark_home')