Explorer Documentation

Explorer uses two primary data structures:

1. Series: A one-dimensional array guaranteed to contain items of a single data type (dtype). Nil values are permitted. Supported dtypes include :binary, :boolean, :category, :date, :datetime, :duration, floats ({:f, size}), integers ({:s, size} or {:u, size}), :null, :string, :time, :list, and :struct.

2. Dataframes: A two-dimensional representation consisting of one or more series that all share the same size.

Explorer functions follow an immutable pattern: they always return a new dataframe or series rather than mutating the existing one.

Explorer typically uses precompiled NIFs. To force a local build from source, set the EXPLORER_BUILD environment variable to 1 and include :rustler as a dependency in your mix.exs.

If you need to clean up a previous build before rebuilding, use mix deps.clean explorer.

{:explorer, "~> 0.12.0", system_env: %{"EXPLORER_BUILD" => "1"}},
{:rustler, ">= 0.0.0"}

You can add Explorer to your project using Mix or use it directly in a Livebook script.

In a Mix project

Add {:explorer, "~> 0.12.0"} to your mix.exs dependencies.

In a Livebook or Elixir script

Use Mix.install to load the dependency:

Mix.install([
  {:explorer, "~> 0.12.0"}
])

def deps do
  [
    {:explorer, "~> 0.12.0"}
  ]
end

An Explorer.Backend.QueryFrame is a lazy dataframe used for building query expressions. It is not a standalone data structure you typically manipulate directly; instead, it is provided as an argument within transformation functions like filter_with/3 and mutate_with/3.

Key constraints:

• You cannot perform standard dataframe operations directly on a QueryFrame.
• Its primary purpose is to allow you to access its underlying series within a query context.
• To create a query-backed dataframe, use Explorer.Query.new/1.

The Polars backend allows you to switch from Eager to Lazy execution. This enables query optimization before the data is actually processed.

• Call lazy/0 on a DataFrame to get a LazyFrame.
• Call lazy/1 with an existing DataFrame to convert it to a LazyFrame.
• Use collect/1 on a LazyFrame to execute the plan and return an Eager DataFrame.

Many operations (like select/2, filter_with/3, mutate_with/3, distinct/3, rename/3, explode/3, unnest/3, summarise_with/3, and sql/3) are implemented by internally switching to a LazyFrame, applying the operation, and then calling collect/1.

Explorer uses a backend-based architecture to handle data processing. Each backend is a module that provides its own DataFrame and Series submodules, which must implement the respective Explorer behaviours.

The default backend is determined by the :default_backend configuration key in the :explorer application environment. Currently, the default is an in-memory, eager backend based on Polars.

A TensorFrame is a specialized representation of an Explorer.DataFrame designed for use within Nx defn expressions.

When you pass an Explorer.DataFrame as an argument to an Nx.defn, it is automatically converted into a TensorFrame. The conversion is lazy: tensors are only built out of the specific dataframe fields that are actually accessed during the computation.

Because of this integration, you can also pass DataFrames directly into Nx.stack/2 and Nx.concatenate/1, and they will be automatically converted to tensors.

iex> add_columns(Explorer.DataFrame.new(a: [11, 12], b: [21, 22]))
        #Nx.Tensor<
          s64[2]
          [32, 34]
        >

iex> Nx.concatenate(Explorer.DataFrame.new(a: [11, 12], b: [21, 22]))
        #Nx.Tensor<
          s64[4]
          [11, 12, 21, 22]
        >

Explorer exposes core Polars data structures to Elixir via NIFs. These types are wrapped in Ex prefixed structs to facilitate the interface between Elixir and the Rust backend. The following primary types are available:

• Explorer.PolarsBackend.DataFrame: A wrapper around the Polars DataFrame.
• Explorer.PolarsBackend.Expression: A wrapper around a Polars Expr used for query construction.
• Explorer.PolarsBackend.LazyFrame: A wrapper around a Polars LazyFrame for lazy evaluation.
• Explorer.PolarsBackend.Series: A wrapper around a Polars Series representing a single column of data.

When using TensorFrame inside an Nx.defn, avoid returning the entire TensorFrame. Returning the whole frame forces all columns to be sent to the device (CPU/GPU) and copied back, which is inefficient.

Instead, return only the specific columns that were modified. To integrate these results back into an Explorer.DataFrame, use Explorer.DataFrame.put/4, which accepts tensors as values.

Alternatively, you can use Explorer.Series.from_tensor/1 to explicitly convert a single tensor back into a series.

iex> df = Explorer.DataFrame.new(a: [11, 12], b: [21, 22])
iex> Explorer.DataFrame.put(df, "result", add_columns(df))
#Explorer.DataFrame<
  Polars[2 x 3]
  a s64 [11, 12]
  b s64 [21, 22]
  result s64 [32, 34]
>

Explorer's precompiled artifacts may use modern CPU features that are incompatible with older hardware. If you encounter an Illegal instruction error, enable the legacy artifacts in your application configuration:

config :explorer, use_legacy_artifacts: true

Dataframes can be created using Explorer.DataFrame.new/2 by passing series or lists. To use the powerful query DSL (which allows referring to columns by name and using functions without explicit definition), you must require Explorer.DataFrame.

Example: Filtering data

# Create a dataframe
mountains = Explorer.DataFrame.new(name: ["Everest", "K2", "Aconcagua"], elevation: [8848, 8611, 6962])

# Enable macro features
require Explorer.DataFrame, as: DF

# Filter rows where elevation is greater than the mean elevation
DF.filter(mountains, elevation > mean(elevation))

To view a dataframe as a formatted table in the console, use Explorer.DataFrame.print/2.