DOCSEARCHbyAlgolia
Submit Docs

Frictionless Framework

website·Indexed 36 days ago

https://framework.frictionlessdata.io/

A framework for describing, validating, and transforming data. It provides tools for managing metadata through components like Packages, Catalogs, and Resources, and supports various data formats including SQL. The framework includes a CLI and Python API for data inspection via the `describe` function, as well as integration with portals such as Zenodo and CKAN. Version 5 introduced a distinct boundary between descriptors and the object model for improved state management.

Tokens
63K
Snippets
403
Records
530
Agent score
62%

What's inside Frictionless Framework

  1. Overview of Frictionless Data Transformation

    Transforming data in Frictionless involves modifying data and metadata from a source state (A) to a target state (B). Key characteristics include:

    • Metadata Integration: Unlike many ETL frameworks, Frictionless treats metadata as a first-class citizen, ensuring type information and other metadata are preserved during pipeline evaluation.
    • Data Streaming: Data is streamed rather than loaded entirely into memory, allowing the framework to handle very large files. For operations like sorting, it uses a memory threshold and offloads to the file system when necessary.
    • Lazy Evaluation: Manipulations happen on-demand. For example, if only one table in a package is reshaped, the others are not read. CPU-intensive tasks like type casting only occur if a normalize step is explicitly added.
    • Underlying Technology: Frictionless uses the PETL library for its core transform functions to provide lazy-loading functionality, adding a bridge between PETL processors and Frictionless Package/Resource concepts.

  2. Overview of the Frictionless Framework

    The Frictionless Framework (frictionless-py) is an open-source (MIT) data management framework for Python designed for the DEVT (Describe, Extract, Validate, Transform) workflow. It provides a unified interface to handle tabular data across various sources (HTTP, FTP, S3, Local) and formats (CSV, XLS, JSON, SQL, etc.) while ensuring data quality through the use of Frictionless Standards.

  3. Overview of Frictionless Data Specifications

    Frictionless uses standardized specifications to describe different levels of data organization:

    • Table Schema: Used to describe a data table (fields, types, constraints).
    • Data Resource: Used to describe a single data resource (file path, format, encoding).
    • Data Package: Used to describe a collection of data resources and their relationships.

  4. Overview of Frictionless Framework (frictionless-py)

    frictionless-py is an open-source (MIT) data management framework for Python designed for the DEVT (Describe, Extract, Validate, Transform) workflow of tabular data. It is powered by Frictionless Standards and supports a wide variety of data sources (HTTP, FTP, S3), formats (CSV, XLS, JSON, SQL, etc.), and platform integrations. Key capabilities include:

    • Describe: Infer, edit, and save metadata (descriptions, field types) to ensure data quality.
    • Extract: Read data via a unified tabular interface with schema-guaranteed consistency.
    • Validate: Generate unified validation reports for tables, resources, and datasets.
    • Transform: Clean, reshape, and transfer data using pipelines or low-level interfaces.

  5. Overview of built-in transform step groups

    Frictionless provides over 40 built-in transform steps. These are organized into groups based on the object they operate on, which facilitates discovery via IDE auto-completion. The available groups are: resource, table, field, row, and cell (accessed via steps.<group_name>...).

  6. Overview of describing data in Frictionless

    In the Frictionless Framework, "describing data" refers to the process of creating metadata for data files. Metadata is essential for providing context that raw data files lack, such as field meanings, data types, constraints, and relations. This metadata is primarily used for data validation (identifying problems early in the workflow) and data publication (providing necessary context for external users).

  7. Overview of describing data in Frictionless

    In the Frictionless Framework, "describing data" refers to the process of creating metadata for data files. Metadata is essential for providing context that raw data files lack, such as field meanings, data types, constraints, and data relations. This metadata is primarily used for data validation (identifying problems early in the workflow) and data publication (providing necessary context for users).

  8. Overview of Frictionless extract functions

    Frictionless provides three primary high-level interfaces for data extraction:

    • extract: A general-purpose function that detects the source file type and extracts data.
    • resource.extract: A method on a Resource object that returns a data table.
    • package.extract: A method on a Package object that returns a map of all tables within that package.

  9. Use BooleanField to represent boolean data

    The BooleanField is used to represent data that contains boolean (true/false) values. When data is provided as strings in physical representations, the framework casts them to logical booleans based on defined true_values and false_values lists.
    from frictionless import Schema, extract, fields

# Data containing string representations of booleans
data = [['name'], ['true'], ['false']]

# Define a schema with a BooleanField
rows = extract(data, schema=Schema(fields=[fields.BooleanField(name='name')]))
print(rows)
# Output: {'memory': [{'name': True}, {'name': False}]}

  11. Understand the role of Data Package descriptors

    A Data Package descriptor is a high-level specification that describes an entire dataset. It aggregates individual resource descriptors, including their schemas, dialects, and paths. This allows developers to share and validate complete datasets rather than just individual files.

  12. Infer and access resource statistics

    To obtain statistics about a resource, call the infer() method with stats=True. The resulting statistics are accessible via the resource.stats property, which returns a ResourceStats object.
    from frictionless import Resource

resource = Resource('table.csv')
resource.infer(stats=True)
print(resource.stats)