Kotlin DataFrame

The dataframe-arrow module enables Kotlin DataFrame to interoperate with Apache Arrow formats. You can use this module to read data from Apache Arrow formats into a DataFrame or to write existing DataFrames into Apache Arrow formats.

For detailed instructions on specific operations, refer to the official documentation on reading and writing Apache Arrow formats.

Kotlin DataFrame supports reading from and writing to a wide variety of data formats and storage systems. This allows you to integrate structured data into your processing workflows seamlessly.

Supported formats and sources include:

• File-based formats: JSON (including OpenAPI), CSV, TSV, Excel, Apache Arrow, and Parquet.
• SQL Databases: PostgreSQL, MySQL, Microsoft SQL Server, SQLite, H2, MariaDB, and DuckDB.
• Extensibility: You can use custom SQL sources or create custom integrations for unsupported data sources.

The dataframe-geo module provides logic and support for working with geographical data within Kotlin DataFrame.

Note: This module is Experimental.

This module targets Java 11 due to restrictions from org.jetbrains.kotlin.jupyter.

Kotlin DataFrame can be integrated into various development environments depending on your use case. Choose the appropriate setup guide based on your project type:

• Application Development: Use Gradle or Maven to add Kotlin DataFrame to your Kotlin applications.
• Mobile Development: Use the Android setup to simplify local data processing and transformation in Android apps.
• Interactive Data Analysis:
  • Kotlin Notebook: For interactive notebooks in IntelliJ IDEA and Android Studio (no configuration required).
  • Datalore: For remote collaboration and sharing in JetBrains Datalore notebooks.
  • Jupyter: For integration into interactive notebooks using the Kotlin Jupyter kernel.

A dataframe is an abstraction for working with structured, 2-dimensional data. It functions like a spreadsheet, a SQL table, or a dictionary of series objects, consisting of labeled columns that can hold different types.

Kotlin DataFrame provides an idiomatic Kotlin DSL for data wrangling—the process of transforming and mapping raw data into formats suitable for analytics and visualization.

Kotlin DataFrame is an official open-source Kotlin framework for working with tabular data on the JVM. It aims to reconcile Kotlin's static typing with the dynamic nature of data using an idiomatic DSL.

Key characteristics:

• Immutability: DataFrame is a completely immutable structure. All operations that modify data return a new DataFrame instance, following a functional programming style.
• Type Support: You can store any Kotlin or Java types within a DataFrame. Built-in support exists for automatic conversion and parsing of Kotlin basic types and kotlinx-datetime types.
• Hierarchical Data: It supports nested structures using FrameColumn (a column of dataframes) and ColumnGroup (a column with nested subcolumns).

A DataFrame is an in-memory, immutable structured data object. While it conceptually resembles a SQL table (rows, columns, and schema), it has key differences:

• In-Memory: Unlike SQL databases, there is no persistent storage engine or transaction log.
• Immutability: Every operation produces a new DataFrame rather than modifying the existing one.
• No Relations: There is no concept of foreign keys or relations between different DataFrames.
• Dynamic Schema: Unlike ORMs (like Hibernate) where mappings are frozen, a DataFrame's schema evolves automatically with your transformations, especially when using the Kotlin Compiler Plugin.
• Purpose: It is an ETL and data analysis tool, not a persistence layer for managing mutable entities.

The expressions-converter is a Kotlin Compiler plugin used by the core module of Kotlin DataFrame. Its primary purpose is to extract intermediate DataFrame expressions from functions annotated with @TransformDataFrameExpressions.

These extracted expressions are used to generate 'explainer dataframe' HTML files, which serve as interactive iFrames within the official Kotlin DataFrame documentation website.