DOCSEARCHbyAlgolia
Submit Docs

Tablesaw Documentation

repository·master·Indexed 26 days ago

https://github.com/jtablesaw/tablesaw

A Java-based dataframe and visualization library for loading, cleaning, transforming, and summarizing data. Tablesaw provides tools for descriptive statistics, data manipulation (joining, filtering, grouping), and visualization via a Plot.ly wrapper. It supports various data formats including CSV, JSON, HTML, Excel, and Apache Arrow, and integrates with Jupyter Notebooks through BeakerX, IJava, and Google Colab.

Tokens
24K
Snippets
68
Records
177
Agent score
86%

What's inside Tablesaw

  1. Overview of Tablesaw capabilities

    master

    Tablesaw is an in-memory, tabular data structure (dataframe) designed for data manipulation, statistical modeling, and visualization. It allows you to work with columns of single datatypes while managing rows of varying types.

    Key capabilities include:

    • Data I/O: Importing and exporting data from text files and databases.
    • Column Manipulation: Adding, removing, and creating new columns via mapping (applying functions to existing columns).
    • Row Manipulation: Adding, updating, removing, sorting, and filtering rows.
    • Data Transformation: Combining tables via appending or joining, and summarizing data via reducing.
    • Analysis: Calculating descriptive statistics.
    • Visualization: Plotting data directly from the dataframe.

  2. Overview of Tablesaw features

    master

    Tablesaw is a Java library for dataframe manipulation and visualization. Key capabilities include:

    Data Processing & Transformation

    • Importing: Supports RDBMS, Excel, CSV, TSV, JSON, HTML, and Fixed Width files (local or remote via http, S3, etc.).
    • Exporting: Supports CSV, JSON, HTML, and Fixed Width files.
    • Manipulation: Append/join tables, add/remove columns or rows, sort, group, filter, edit, and transpose.
    • Operations: Map/Reduce operations and handling of missing values.

    Visualization

    • Provides a wrapper for the Plot.ly JavaScript library to create various chart types (e.g., scatter, histograms, heatmaps, pie charts, etc.).

    Statistics

    • Supports descriptive statistics including mean, min, max, median, sum, product, standard deviation, variance, percentiles, geometric mean, skewness, and kurtosis.

  3. Understand Saw Table Store architecture and limitations

    master

    Saw stores data by streaming each column as a separate file to the file system.

    Technical Details:

    • Data Handling: Streams data to avoid transformation overhead (e.g., Dates are stored as ints rather than being converted to LocalDate or String first).
    • Encoding: StringColumns use dictionary encoding that is stored in its entirety to avoid recalculation during reads.
    • Performance: Uses fast compression algorithms and thread pools for parallel read/write operations.

    Limitations:

    • Thread Safety: Saw is not thread-safe. The table cannot be updated while it is being written.
    • Portability: The format is non-standard, not human-readable, and cannot be used directly by other applications.

  4. Create custom visualizations using Figures, Traces, and Layouts

    master

    For more complex or specific visualizations not covered by the canned plots, you can build custom visualizations. These are assembled using three core components:

    • Figures: The top-level container.
    • Traces: The data representations (e.g., the actual lines or points).
    • Layouts: The configuration for axes, legends, and overall appearance.

    Detailed instructions for custom assembly can be found in the Visualization Customization guide.

  5. Supported Column Types in Tablesaw

    master

    Tablesaw columns are named vectors of a single data type. The following concrete types are available in the api package:

    • Boolean: BooleanColumn (true/false values).
    • Textual:
      • StringColumn: For categorical values (e.g., "New York").
      • TextColumn: For unique text values.
    • Numeric:
      • NumberColumn: Interface for numeric types.
      • ShortColumn: Small integral values.
      • IntColumn: Standard integral type for most cases.
      • LongColumn: Large integral values.
      • FloatColumn: Single-precision floating point.
      • DoubleColumn: Standard 8-byte floating point (usually best for most values).
    • Temporal:
      • DateColumn: Local date (no timezone).
      • DateTimeColumn: Local date and time.
      • TimeColumn: Local time.
      • InstantColumn: Single point in time without timezone reference.

    Note: Mathematical operations typically return DoubleColumn instances.

  6. Create and add columns to a Table

    master

    You can create columns by calling static create() methods on the specific column class. You can initialize them empty or with an array of values. Once created, add them to a Table using addColumns().

    Important: All columns within a table must have unique names. You can retrieve a column's name via name() and its type via type().

  9. Use Unary, Binary, and n-Ary Map Functions

    master

    Map functions transform one or more columns into a new Column of the same length.

    • Unary mappers: Operate on a single column (the receiver). They can optionally take additional non-column parameters (e.g., substring(int)).
    • Binary mappers: Operate on two columns (the receiver and one parameter column).
    • n-Ary mappers: Operate on an array of columns.

    Note: Map functions do not modify the original table. To include the result in your table, you must explicitly call table.addColumn(newColumn).

  12. Customize CSV loading with CsvReadOptions

    master

    If your CSV file uses different delimiters, lacks a header, or uses specific date formats, use CsvReadOptions.builder() to configure the loading process.

    Key options include:

    • .separator(char): Specify a delimiter (e.g., \t for tab-delimited).
    • .header(boolean): Set to false if there is no header row.
    • .dateFormat(String): Provide a format string compatible with java.time.format.DateTimeFormatter.
    • .missingValueIndicator(String): Define a custom string to be treated as missing data (default values include "NaN", "*", "NA", "null", and "").
    • .locale(Locale): Specify a locale to assist with date and number parsing.
    • .sample(boolean): Set to false to perform type inference on the entire dataset instead of a sample.
    CsvReadOptions.Builder builder = 
	CsvReadOptions.builder("myFile.csv")
		.separator('\t')             // table is tab-delimited
		.header(false)               // no header
		.dateFormat("yyyy.MM.dd");   // the date format to use.

CsvReadOptions options = builder.build();

Table t1 = Table.read().usingOptions(options);