ingestr Documentation

ingestr supports a wide range of platform sources categorized by industry and function. You can find specific integration guides for each platform in the /supported-sources/ directory. Categories include:

• Marketing, ads, and attribution: e.g., Adjust, Facebook Ads, Google Ads, TikTok Ads.
• Analytics, feedback, and media: e.g., Google Analytics, Mixpanel, PostHog.
• CRM, sales, support, and success: e.g., HubSpot, Salesforce, Zendesk.
• Commerce, payments, billing, and finance: e.g., Stripe, Shopify, RevenueCat.
• Work management, productivity, and operations: e.g., Airtable, Asana, Jira, Notion, Slack.
• Developer, data, and infrastructure: e.g., Amazon S3, GitHub, Google Cloud Storage, SFTP.
• Sports, games, and public data: e.g., ESPN, Chess.com, Polymarket.

When using the Anthropic source, be aware of the following constraints:

• Scope: This source only tracks Claude Code usage on the Anthropic API (1st party). Usage via Amazon Bedrock, Google Vertex AI, or other third-party platforms is not included.
• Access Level: The source requires organization-level access; it is not available for individual accounts.
• Timezones: All dates and timestamps are provided in UTC.
• Rate Limiting: The Anthropic Admin API has rate limits. The ingestr source handles pagination automatically and respects these limits.

To copy data from a source to a destination using the CLI, use the ingest command with --source-uri, --source-table, --dest-uri, and --dest-table flags.

Example: Copying from Postgres to BigQuery:

ingestr ingest \
    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
    --source-table 'public.some_data' \
    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
    --dest-table 'ingestr.some_data'

OneLake only supports Microsoft Entra ID authentication. ingestr resolves credentials in the following order:

1. SAS token: If sas_token is provided in the URI.
2. Service principal: If tenant_id, client_id, and client_secret are all provided. The service principal requires Contributor (or item-level) access to the workspace, and Fabric admins must allow service principals to use the APIs.
3. DefaultAzureCredential: Fallback to Azure SDK's default credential (environment variables, managed identity, or Azure CLI login).

The SharePoint source supports the --incremental-strategy flag, which defaults to replace.

Supported strategies:

• replace (default)
• append
• merge
• delete+insert

⚠️ CAUTION: SharePoint does not support incremental extraction. Every run reads the entire file or glob. If you use append, you will duplicate rows on every run. It is recommended to use replace unless you have a specific reason to accumulate data.

When using the ingestr CLI, any passwords containing special characters (such as @, :, /, #, ?, etc.) must be URL-encoded within the connection URI.

# Example: encoding 'MyPass@123!' as 'MyPass%40123%21'
ingestr ingest \
  --source-uri "couchbase://admin:MyPass%40123%21@localhost" \
  --source-table "mybucket.myscope.mycollection" \
  --dest-uri "duckdb:///output.db" \
  --dest-table "main.couchbase_data"

After running the ingestion command, you can verify the data using the DuckDB CLI:

1. Open the database: duckdb personio.duckdb
2. List tables: Run .tables; to confirm the destination table exists.
3. Query data: Run SELECT * FROM dest.employees; to view the content.

duckdb personio.duckdb

# Inside DuckDB:
.tables;
SELECT * FROM dest.employees;

To allow ingestr to read from Amazon Kinesis, the IAM user/role must have appropriate permissions.

Option 1: Managed Policy Attach the AWS managed policy: AmazonKinesisReadOnlyAccess.

Option 2: Custom Policy Create a custom policy with the following permissions:

• kinesis:DescribeStream
• kinesis:GetShardIterator
• kinesis:GetRecords
• kinesis:ListShards
• kinesis:ListStreams

Custom queries support incremental loads provided the following conditions are met:

• The incremental key must be a column returned by the query.
• The incremental key must be a datetime or timestamp column.
• You must manually implement filtering in your SQL query using the :interval_start and :interval_end variables.

Example of an incremental load using a join and a timestamp filter:

ingestr ingest \
    --source-uri $POSTGRES_URI \
    --dest-uri "duckdb:///mydb.db" \
    --dest-table "public.output" \
    --source-table "query:select oi.*, o.updated_at from order_items oi join orders o on oi.order_id = o.id where o.updated_at > :interval_start" \
    --incremental-key updated_at \
    --incremental-strategy merge \
    --primary-key id

If you need to discard the existing destination state and rebuild the table from a fresh snapshot, add the --full-refresh flag to your ingestr ingest command.

ingestr ingest \
  --source-uri "mongodb+cdc://user:password@localhost:27017/" \
  --source-table "shop.customers" \
  --dest-uri "duckdb:///warehouse.duckdb" \
  --dest-table "shop.customers" \
  --full-refresh

The orders and subscriptions tables support incremental loads using a date range.

• Provide --interval-start and --interval-end to fetch only records within that window.
• If no interval is provided, ingestr fetches the full history.

ingestr ingest \
  --source-uri 'fastspring://?username=xxx&password=yyy' \
  --source-table 'orders' \
  --dest-uri duckdb:///fastspring.duckdb \
  --dest-table 'dest.orders' \
  --interval-start '2024-01-01' \
  --interval-end '2024-01-31'