Tableau Hyper API Samples

The clouddb-extractor is a utility designed to extract data from cloud databases and synchronize it with published Tableau Hyper extracts. It supports appending, updating, upserting, and deleting rows to keep Tableau data sources up to date.

It uses a standard Extractor Interface (implemented via BaseExtractor) which is extended for specific databases like Azure SQL, BigQuery, MySQL, PostgreSQL, and Redshift. For most automation tasks, you will interact with these core methods:

• load_sample: Extracts a sample of rows for testing.
• export_load: Performs an initial full extract of a source table.
• append_to_datasource: Appends rows from a query or table to an existing datasource.
• update_datasource: Updates an existing datasource using a changeset.
• delete_from_datasource: Deletes rows from a datasource based on conditions or primary keys in a changeset.

This repository contains two distinct categories of samples:

1. Tableau-Supported Samples: These are official samples packaged and shipped within the Hyper API itself. They are available for Python, Java, C++, and C#/.NET and are fully maintained by Tableau.
2. Community-Supported Samples: These focus on specific use cases and are Python-only. While written by members of the Tableau development team, they are community-supported and may not be maintained for every new release. They are manually tested and reviewed before publishing.

The query-external-data sample demonstrates how to use the Hyper API to query external data formats, such as Parquet or CSV files, directly. This pattern is useful for ETL (Extract, Transform, Load) workflows where you need to:

• Access multiple files simultaneously.
• Filter data during the read process.
• Create additional calculated columns from external sources.

The Hyper API provides a set of functions to automate interactions with Tableau extract (.hyper) files. It allows developers and administrators to programmatically manage data within these files.

Key capabilities include:

• Creating extract files: Generate new .hyper files for data sources not natively supported by Tableau.
• Data Manipulation: Open existing files to insert, delete, update, or read data.
• ETL Automation: Automate custom Extract, Transform, and Load (ETL) processes, such as implementing rolling window updates or custom incremental updates.
• Data Retrieval: Programmatically retrieve data from an existing extract file.

The utility is built on an Abstract Base Class (ABC) called BaseExtractor. This class provides the core logic for extracting from cloud databases to 'live to hyper' Tableau Datasources.

Specific database implementations (e.g., azuresql_extractor.py, bigquery_extractor.py) extend this base class to handle:

1. Connection Management: Handling database-specific authentication and connectivity.
2. Schema Discovery: Identifying table structures.
3. Query Optimization: Overriding generic query processing to use database-specific optimizations (following DBAPIv2 standards).

Developers can either use the provided extractor_cli.py or import a specific implementation class directly into their own Python scripts for custom automation.

The sample implements an incremental refresh pattern using the Hyper Update REST API. The logic follows two distinct paths based on the state of the Tableau Server/Cloud:

1. Initial Publication: If no datasource with the specified datasource_name_on_server exists on the server, the script creates and publishes a new initial datasource. This establishes the baseline for future updates.
2. Incremental Update: If the datasource already exists, the script uses the Hyper Update REST API to insert only the new data from the local Hyper database into the existing table on the server. This avoids re-publishing the entire dataset and keeps the data fresh efficiently.

When using pantab to transform .hyper databases, keep the following in mind:

1. Datatype Mapping: You must properly map datatypes between Hyper's SQLTypes and pandas' dtypes.
2. Nullability: The NULLABILITY specified in the documentation must match the configuration in your .hyper file to ensure data integrity during the conversion.

In the Hyper 'spheroidal' system, vertex order defines the polygon's interior. Tableau follows an 'interior-left' convention: as you progress through vertices in storage order, the interior of the polygon should be to the left.

Because the system is spheroidal, there is no technically 'incorrect' order, but a reverse order changes the space the polygon encloses. If data is imported with an 'interior-right' winding order, users may encounter:

• Incorrect auto-zoom behavior in Tableau maps.
• Unexpected results during spatial joins.

To run the conversion script, ensure you have Python 3.7 or newer installed on Windows, macOS, or Linux. It is recommended to use a Python virtual environment.

1. Navigate to the directory containing tde_to_hyper.py.
2. Create a virtual environment and install the required Hyper API version (0.0.17537) using the provided requirements.txt.

$ python3 -m venv .venv/
$ .venv/bin/python3 -m pip install -r requirements.txt