DOCSEARCHbyAlgolia
Submit Docs

Google BigQuery API client library for Python

repository·main·Indexed 21 days ago

https://github.com/googleapis/python-bigquery

A Python client library for interacting with Google BigQuery, enabling users to run SQL queries against massive datasets using Google's infrastructure. The library includes support for DB-API parameter styles, explicit type information for parameters (including structs), and configurable API-level and job-level retries.

Tokens
22.9K
Snippets
92
Records
115
Agent score
74%

What's inside google-cloud-bigquery

  2. Use DB-API parameter styles in BigQuery

    main

    The BigQuery DB-API supports two parameter styles for queries:

    1. Unnamed/Positional parameters: Uses the qmark style with ? placeholders.
    2. Named parameters: Uses the pyformat style with %(name)s placeholders.

    When using positional parameters, the values are provided in a sequence (like a list) in the order they appear in the query.

    -- Unnamed/Positional style
insert into people (name, income) values (?, ?)

-- Named style
insert into people (name, income) values (%(name)s, %(income)s)

  3. Changes to pandas DataFrame dtypes in 3.x

    main

    The to_dataframe method in version 3.x has updated default mappings for BigQuery types to pandas dtypes:

    BigQuery TypePandas Dtype (3.x)Notes
    BOOLEANbooleanPreviously bool or object
    INT64Int64Previously int64 or float64
    DATEdbdateProvided by db-dtypes. Maps to object if outside pandas Timestamp range (1677-09-22 to 2262-04-11). date_as_object parameter is removed.
    TIMEdbtimeProvided by db-dtypes

    When loading a pandas DataFrame without a schema:

    • Naive datetime64[ns] columns are loaded as DATETIME.
    • Timezone-aware datetime64[ns, UTC] columns are loaded as TIMESTAMP.

  4. Understand the difference between `retry` and `job_retry`

    main

    The BigQuery Python client distinguishes between retrying an API request and re-issuing a query job:

    Featureretry parameterjob_retry parameter
    LayerAPI LayerJob Layer
    GoalRetries the specific REST API call (e.g., jobs.insert) using the same ID to ensure idempotency.Re-issues the entire query by generating a new job/request ID if the previous job failed.
    TriggerOccurs when the API request itself fails (e.g., network error during the call).Occurs when the API call succeeds, but the resulting BigQuery job fails (e.g., backendError).
    RequirementRequires the request to be idempotent (constant jobId or requestId).Only works if the developer has not provided a custom job ID.

  5. Use legacy proto-based types for BigQuery v2 API with caution

    main

    The google.cloud.bigquery_v2.types module provides legacy proto-based types for the Google Cloud BigQuery v2 API.

    Warning: These types are provided for backward compatibility only and are no longer maintained. They may differ from the types supported on the backend. It is strongly advised to migrate to the types found in the standard_sql documentation instead. For migration details, refer to the 3.0.0 Migration Guide.

  6. Core concepts of the BigQuery Python Client

    main

    The BigQuery Python API is organized around several primary abstractions that allow you to interact with Google BigQuery resources:

    • Client (google.cloud.bigquery.client.Client): The central entry point. It manages connections to the BigQuery API and provides methods to run jobs (like query()) and manage BigQuery resources.
    • Dataset (google.cloud.bigquery.dataset.Dataset): Represents a collection of tables.
    • Table (google.cloud.bigquery.table.Table): Represents a single relation (a table) within a dataset.

  7. Thread and process safety for BigQuery clients

    main

    The BigQuery client uses the requests library by default, and the BigQuery-Storage client uses grpcio. Both are safe to share instances across threads.

    Multiprocessing Best Practice: In multiprocessing scenarios, do not share client instances created in a parent process. Instead, create client instances after multiprocessing.Pool or multiprocessing.Process invokes os.fork to ensure stability.

  8. Install BigQuery with Pandas support

    main

    To use BigQuery with Pandas, you must have the pandas library installed. You can install the BigQuery Python client library with the pandas extra to ensure all necessary dependencies are met.

    If you also intend to load data from a DataFrame to a BigQuery table, you must also install pyarrow.

    # Install BigQuery client with pandas support
pip install --upgrade 'google-cloud-bigquery[pandas]'

# Install BigQuery client with pandas and pyarrow support (required for loading DataFrames to BigQuery)
pip install --upgrade 'google-cloud-bigquery[pandas,pyarrow]'

  10. Set up the BigQuery Python development environment

    main

    To run BigQuery Python samples, you must have the BigQuery Admin role assigned to your authenticated account. Follow these steps to set up a local environment:

    1. Authentication: Ensure you have credentials configured following the Google Cloud Authentication guide.
    2. Clone Samples: Clone the python-docs-samples repository.
    3. Create Virtual Environment: Use virtualenv (compatible with Python 3.7+).
    4. Install Dependencies: Install the required packages via pip using the provided requirements.txt.
    $ git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
$ virtualenv env
$ source env/bin/activate
$ pip install -r requirements.txt

  12. Test code samples and snippets

    main

    Code samples are located in the samples/ directory. Each folder containing examples has its own noxfile.py. To test them, navigate to the specific sample directory and run the appropriate nox session.

    Example for samples/snippets:

    • Run all tests in the folder: nox -s py-3.9
    • Run a single sample test: nox -s py-3.9 -- -k <name of test>
    # Run all tests in a folder
$ cd samples/snippets
$ nox -s py-3.9

# Run a single sample test
$ cd samples/snippets
$ nox -s py-3.9 -- -k <name of test>