influxdb-client-python

repository·master·Indexed 21 days ago

https://github.com/influxdata/influxdb-client-python

A Python client library for interacting with InfluxDB 2.x and Flux. It supports high-performance data writing via Line Protocol, Data Point objects, and Pandas DataFrames, as well as flexible querying in CSV, Table, and DataFrame formats. The library includes a Management API for buckets, tasks, and authorizations, supports asyncio for non-blocking applications, and provides utilities for InfluxDB Cloud features and AWS Lambda Layer packaging.

Tokens
14.3K
Snippets
43
Records
62
Agent score
73%

What's inside influxdb-client-python

  1. Overview of InfluxDB 2.0 client features

    master

    The influxdb-client-python provides comprehensive support for InfluxDB 2.x and Flux.

    Core Capabilities:

    Querying

    Data can be queried using the Flux language and returned in several formats:

    • CSV (via query_csv)
    • Raw data
    • Table structure (via flux_table)
    • Pandas DataFrame

    Writing

    Data can be written using multiple methods:

    • Line Protocol
    • Data Point objects
    • RxPY Observables
    • Pandas DataFrames

    Management

    The client includes a management API (generated from InfluxDB's Swagger/OpenAPI spec) for:

    • Organizations and Users management
    • Buckets management
    • Tasks management
    • Authorizations
    • Health checks

    Compatibility Note:

  2. Enable Flux profiling for queries

    master

    The Flux Profiler provides performance data for Flux queries. When enabled, the query results include additional tables with the profiler/ measurement. Note that FluxCSVParser automatically excludes these profiler/ measurements from standard results to maintain consistent behavior.

    You can enable profilers in three ways:

    1. Set QueryOptions.profilers in the QueryApi.
    2. Set the INFLUXDB_V2_PROFILERS environment variable.
    3. Set the profilers option in your configuration file.

    To use the profiler via the API, pass a list of profiler names (e.g., ["query", "operator"]) to the QueryOptions object when calling query_api().

    query_api = client.query_api(query_options=QueryOptions(profilers=["query", "operator"]))
    csv_result = query_api.query(query=q, params=p)
  3. Authenticate to InfluxDB

    master

    The InfluxDBClient supports three authentication methods:

    1. Token (Preferred): Pass a token to the constructor. An Authorization header will be sent with the value Token <your-token> (the word Token is case-sensitive).
    2. Username & Password: Pass username and password. This uses HTTP Basic authentication and creates a session. Note that sessions expire based on the InfluxDB TTL (default 60 minutes).
    3. HTTP Basic: Use auth_basic=True and a token (representing the proxy secret) when connecting to an InfluxDB 1.8.x instance protected by a reverse proxy. Do not use this for InfluxDB 2.x.
    # Token authentication (Preferred)
    with InfluxDBClient(url="http://localhost:8086", token="my-token") as client:
        pass
    
    # Username & Password authentication
    with InfluxDBClient(url="http://localhost:8086", username="my-user", password="my-password") as client:
        pass
    
    # HTTP Basic (for 1.8.x behind a proxy)
    with InfluxDBClient(url="http://localhost:8086", auth_basic=True, token="my-proxy-secret") as client:
        pass
  4. Set default tags for all writes

    master

    You can attach static or environment-based tags to every measurement written by the client. This is useful for metadata like hostname or customer_id.

    Via PointSettings API

    Use PointSettings to add tags programmatically. You can use the syntax ${env.VARIABLE_NAME} to pull values from environment variables.

    Via Configuration File

    In an init configuration file (supporting INI, TOML, or JSON), specify tags under a [tags] segment.

    Via Environment Properties

    Set environment variables with the prefix INFLUXDB_V2_TAG_ (e.g., INFLUXDB_V2_TAG_HOSTNAME).

    from influxdb_client import InfluxDBClient, PointSettings
    
    # Via API
    point_settings = PointSettings()
    point_settings.add_default_tag("id", "132-987-655")
    point_settings.add_default_tag("data_center", "${env.data_center}")
    
    write_client = client.write_api(point_settings=point_settings)
    
    # Via Environment Properties
    # Set INFLUXDB_V2_TAG_HOSTNAME=my-host in your shell
    client = InfluxDBClient.from_env_properties()
  5. Manage InfluxDB resources via Management API

    master

    The client allows programmatic management of InfluxDB resources:

    • Buckets: Create, list, and delete buckets.
    • Tasks: Create tasks via the API.
    • Authorizations: Create and manage API tokens/authorizations.
    • Monitoring & Alerting: Create checks with notifications (e.g., Slack).
    • Templates & Stacks: Manage InfluxDB templates and stacks.
  6. Upload and use the Lambda Layer in AWS

    master

    Once you have extracted the python.zip file from the Docker container, follow these steps to use it in your AWS environment:

    1. Upload: Use the AWS CLI or the AWS Management Console to create a new Lambda Layer and upload the python.zip archive.
    2. Attach: Import the newly created Layer into your target Lambda function.
    3. Usage: Once attached, you can import the client in your Lambda function code as usual:
    from influxdb_client import InfluxDBClient, Point
  7. Write data to InfluxDB using WriteApi

    master

    The WriteApi supports synchronous, asynchronous, and batching writes. Data can be provided in several formats:

    1. Line Protocol: A string or bytes formatted as InfluxDB Line Protocol.
    2. Data Point: Using the Point class.
    3. Dictionary: A mapping with keys measurement, tags, fields, and time.
    4. Other structures: NamedTuple, Data Classes, or Pandas DataFrame.
    5. Streams: An Observable stream (using reactivex) that produces any of the above.

    Important: The WriteApi in batching mode (the default) should be treated as a singleton. To ensure all data is flushed, use a with statement or call write_api.close() at the end of your script.

    from influxdb_client import InfluxDBClient, Point, WriteOptions
    
    with InfluxDBClient(url="http://localhost:8086", token="my-token", org="my-org") as _client:
        with _client.write_api(write_options=WriteOptions(batch_size=500)) as _write_client:
            # Write Line Protocol
            _write_client.write("my-bucket", "my-org", "h2o_feet,location=coyote_creek water_level=1.0 1")
            
            # Write Data Point
            _write_client.write("my-bucket", "my-org", Point("h2o_feet").tag("location", "coyote_creek").field("water_level", 4.0).time(4))
            
            # Write Dictionary
            _write_client.write("my-bucket", "my-org", {"measurement": "h2o_feet", "tags": {"location": "coyote_creek"}, "fields": {"water_level": 1.0}, "time": 1})
    
            # Write Pandas DataFrame
            import pandas as pd
            _df = pd.DataFrame(data=[["coyote_creek", 1.0]], columns=["location", "water_level"])
            _write_client.write("my-bucket", "my-org", record=_df, data_frame_measurement_name='h2o_feet', data_frame_tag_columns=['location'])
  8. Create an AWS Lambda Layer using the Docker image

    master

    This Docker image allows you to package the influxdb-client and its dependencies into a .zip file suitable for an AWS Lambda Layer. Using a Layer keeps your main Lambda function code small, allowing you to continue using the AWS Console IDE for editing.

    The resulting zip file is approximately 3.5MB when containing only influxdb-client-python, which is well within the 10MB browser upload limit for Lambda Layers.

    # 1. Build the image
    docker build -t lambdalayer:latest .
    
    # 2. Create a container instance
    docker create --name lambdalayer lambdalayer:latest
    
    # 3. Copy the generated zip file from the container to your local machine
    docker cp lambdalayer:/install/python.zip .
  9. Write data to InfluxDB

    master

    The client provides several patterns for writing data, including importing CSV files, ingesting Pandas DataFrames, and writing structured data using Python NamedTuple or dataclasses (requires Python 3.8+).

    Common write patterns include:

    • CSV Import: Importing data from CSV files, including support for large files via Python Multiprocessing.
    • DataFrame Ingestion: Writing Pandas DataFrames with default tags or handling large DataFrames.
    • Structured Data: Using NamedTuple or dataclasses to define data structures for writing.
    • Batching & RxPY: Using RxPY to prepare batches based on count or maximum byte size.
    • Error Handling: Leveraging HttpHeader information when errors occur during writes.
  10. Initialize InfluxDBClient

    master

    In influxdb-client-python, initialize the client using a url, token, and org. It is best practice to use the client as a context manager to ensure proper resource cleanup.

    from influxdb_client import InfluxDBClient
    
    with InfluxDBClient(url='http://localhost:8086', token='my-token', org='my-org') as client:
        pass
  11. Migrate from influxdb-python to influxdb-client-python

    master
    This guide provides migration patterns for users moving from the legacy influxdb-python library to the modern influxdb-client-python. Key architectural changes include moving from host/port/username/password authentication to URL/token/org authentication, and transitioning from direct client methods to specialized API objects (e.g., buckets_api(), write_api(), query_api()). It is recommended to use the with statement for client lifecycle management.
  12. Run linting and tests

    master

    Before submitting a Pull Request, ensure your changes adhere to the project's formatting and pass all tests.

    • Linting: Uses flake8 to ensure code formatting. Run with make lint.
    • Testing: The built-in tests require a running instance of InfluxDB 2.x. Run all tests with make test.

    To run both sequentially, use:

    make lint test