Frictionless Framework for Python

Frictionless is a data management framework for Python designed to handle the DEVT lifecycle:

• Describe: Infer, edit, and save metadata (descriptions, field types, and tabular details) to ensure data usability.
• Extract: Read data using a unified tabular interface across various formats (CSV, XLS, JSON, SQL, etc.) and schemes (HTTP, FTP, S3).
• Validate: Validate datasets, resources, or individual tables against a schema, generating unified validation reports.
• Transform: Clean, reshape, and transfer data using a pipeline capability or a low-level Python interface.

Key characteristics include a pluggable architecture, low memory consumption for large datasets, and support for custom checks and formats.

Transforming data in Frictionless involves modifying data and metadata from an initial state to a target state (e.g., cleaning a messy Excel file into a structured CSV).

Key characteristics of the Frictionless Transform engine:

• Metadata-First: Unlike many ETL frameworks, Frictionless treats metadata as a first-class citizen, ensuring type and schema information is preserved throughout the pipeline.
• Data Streaming: It uses streaming to handle large datasets, minimizing memory usage by processing data in chunks.
• Lazy Evaluation: Data manipulation happens on-demand. For example, if you reshape a table in a large package, Frictionless will not read the other unrelated tables unless explicitly instructed.
• Modular API: It provides a high-level interface for both imperative (Python code) and declarative (JSON/Descriptor) transformations.

A Dialect is a core Frictionless Data concept representing metadata about a tabular data source. It allows you to manage table headers and format-specific details (like comment characters or row skipping).

Dialect instances are widely used across the library and can be passed to:

• Resource
• describe
• extract
• validate
• and other core functions.

from frictionless import Resource, Dialect

dialect = Dialect(header=True)
with Resource('data.csv', dialect=dialect) as resource:
    print(resource.to_view())

The system object (available as frictionless.system) is the central singleton in the Frictionless Framework. It manages the execution context and serves as the primary factory for creating low-level components like adapters, loaders, parsers, and checks.

Important: Using the system object to instantiate low-level objects is preferred over calling classes directly because the system object ensures that all registered plugins are correctly integrated into the process.

from frictionless import system

# Preferred way to create components to ensure plugin support
adapter = system.create_adapter(source, control=control)
loader = system.create_loader(resource)
parser = system.create_parser(resource)

A Checklist is a collection of validation steps and settings (like skip_errors) that makes validation rules shareable across different files or projects. You can define a checklist in YAML and apply it using the CLI to validate multiple files with the same quality requirements.

frictionless validate table1.csv --checklist checklist.yaml
frictionless validate table2.csv --checklist checklist.yaml

A Data Resource describes a specific data file. Unlike a Table Schema (which is an abstract description of a class of files), a Data Resource is concrete because it includes a path property pointing to an exact file.

It includes the Table Schema for tabular data, but also contains information about the file's format, compression, checksums (hash, bytes, rows), and the Dialect (e.g., delimiter, header rows).

from frictionless import describe

# Describe a specific file
resource = describe("country-2.csv")

# Manually correct dialect and schema if inference fails
resource.dialect.header_rows = [2]
resource.dialect.get_control('csv').delimiter = ";"
resource.schema = "country.schema.yaml"

resource.to_yaml("country.resource-cleaned.yaml")

When working with datasets, choose between describe and list based on whether you need to inspect actual data content:

• describe: If a datapackage.json is not provided, this command will load a sample from every tabular data file in a dataset to infer a schema. It provides deep metadata by touching the actual data files.
• list: A lean and quick command that operates only with available metadata and does not touch the actual data files.

The Detector class is used to tweak how Frictionless detects various aspects of metadata (like encoding, field types, and missing values). You can create a Detector instance and pass it to many core Frictionless classes and functions, including Package, Resource, describe, extract, and validate.

CLI Usage

You can pass many detector options directly via the CLI. For example, to specify missing values:

frictionless extract table.csv --field-missing-values 1,2

from frictionless import Detector, Resource

detector = Detector(field_missing_values=['1', '2'])
resource = Resource('table.csv', detector=detector)
print(resource.read_rows())

A LabelError is a specific type of error in Frictionless used to indicate that a value or metadata does not match a required label or schema constraint. When encountering these errors, you should inspect the error's properties to identify the specific mismatch.

Key properties available on a LabelError include:

• type: The error type identifier.
• title: A short summary of the error.
• description: A detailed explanation of why the label error occurred.
• template: The template used to generate the error message.
• tags: Metadata tags associated with the error for categorization.

The Inquiry class allows you to define arbitrary validation jobs consisting of multiple individual validation tasks. This is useful for complex workflows where you need to group different validation requirements (e.g., validating multiple files or different types of resources) into a single unit of work. When an Inquiry contains more than one task, it automatically utilizes multiprocessing to execute the tasks.

from frictionless import Inquiry

inquiry = Inquiry.from_descriptor({
  'tasks': [
    {'path': 'capital-valid.csv'},
    {'path': 'capital-invalid.csv'},
  ]
})
inquiry.to_yaml('capital.inquiry-example.yaml')