DOCSEARCHbyAlgolia
Submit Docs

ckanapi

repository·master·Indexed 42 days ago

https://github.com/ckan/ckanapi

A command line interface and Python module for accessing the CKAN Action API (version 4.11). It provides tools for interacting with remote and local CKAN instances, including specialized classes like RemoteCKAN, LocalCKAN, and TestAppCKAN. The tool is optimized for bulk operations such as dumping, loading, and deleting large amounts of CKAN data (datasets, groups, organizations, and users) with support for parallelism and JSON Lines (.jsonl) files.

Tokens
15.6K
Snippets
52
Records
68
Agent score
47%

What's inside ckanapi

  1. How action shortcuts and call_action work

    master

    The .action object on a RemoteCKAN or LocalCKAN instance provides a dynamic interface to the CKAN Action API.

    1. Action Shortcuts: Calling demo.action.method_name(param=value) automatically maps to a CKAN action. This works for all standard CKAN actions and actions added by plugins (e.g., demo.action.ckanext_showcase_list()).
    2. call_action: The explicit method demo.call_action('method_name', {params}) achieves the same result and is useful when method names are dynamic.

    Both approaches support complex query parameters, such as those used in package_search with SOLR syntax.

    # Action shortcut
packages = demo.action.package_search(q='+organization:sample-org +tags:geojson')

# Explicit call
packages = demo.call_action('package_search', {'q': '+organization:sample-org +tags:geojson'})

  2. Bulk Dumping and Loading data

    master

    You can dump CKAN entities (datasets, groups, organizations, users) to JSON Lines (.jsonl) files and load them back into CKAN.

    Key Features

    • Parallelism: Use the -p flag to run jobs with multiple worker processes.
      • For local CKAN, there is no parallel limit.
      • For remote sites, the default limit is 3 workers.
      • You can override the limit using the CKANAPI_PARALLEL_LIMIT environment variable.
      • You can bypass the limit for specific sites using the CKANAPI_MY_SITES environment variable (comma-delimited list of URLs).
    • Resuming/Splitting: Jobs can be resumed or split by specifying record start and max values.
    • Search vs Dump: ckanapi search is generally faster than dump because it uses package_search and handles pagination automatically.

    Exporting to Data Package format

    To export datasets to the datapackage.json format, use the --datapackages flag.

    ckanapi dump datasets --all -O datasets.jsonl.gz -z -p 4 -r http://localhost

  4. Bulk Delete entities

    master

    The delete command allows for bulk removal of datasets, groups, organizations, or users. It accepts IDs or names via command line arguments or via standard input (piped data).

    Supported Input Formats for Pipe:

    • JSON list: A JSON array of strings containing id or name values.
    • Selective JSON: A JSON object with a results list containing id values.
    • JSON Lines: One JSON object per line, each containing an id or name value.
    • Plain Text: A list of id or name values, one per line.
    ckanapi action package_list -j | ckanapi delete datasets

  5. Use the ckanapi CLI for CKAN Actions

    master

    The ckanapi CLI allows you to interact with the CKAN Action API. You can call any action supported by CKAN or its plugins.

    Connecting to Instances

    • Remote Instances: Use the -r flag to specify the remote CKAN URL. By default, remote actions connect as an anonymous user. Use -a to provide an API KEY.
    • Local Instances: Use the -c flag to specify a CKAN configuration file. If no remote server or config file is provided, the CLI looks for a ckan.ini file in the current directory.
    • User Permissions: Local actions are performed by the site administrator by default. To perform actions as a specific user, use the -u flag with a username.

    Argument Formats

    • String parameters: Use KEY=STRING.
    • JSON parameters: Use KEY:JSON.
    • File uploads: Use KEY@FILE to pass a local file path for upload.
    ckanapi action group_list -r https://demo.ckan.org --insecure

  6. Configure request timeouts

    master

    You can control request timeouts for both the CLI and the Python module to prevent hanging on slow connections.

    Environment Variables

    • CKANAPI_REQUEST_TIMEOUT: Sets the connect timeout (time to establish connection).
    • CKANAPI_REQUEST_READ_TIMEOUT: Sets the read timeout (time to receive a response). If not set, the connect timeout is used.

    Explicitly in Python

    Pass a requests_kwargs dictionary to the call_action method to specify a timeout.

    from ckanapi import RemoteCKAN

demo = RemoteCKAN('https://demo.ckan.org')
# Set a 10-second timeout for this specific call
groups = demo.action.group_list(id='data-explorer', requests_kwargs={"timeout": 10})

  7. Use ActionShortcut for convenient API calls

    master

    The ActionShortcut class provides a syntactic sugar layer for calling CKAN actions. Instead of using the verbose call_action('action_name', {'param': 'value'}) method, you can access actions as attributes on the .action property of a LocalCKAN or RemoteCKAN instance.

    When calling an action via ActionShortcut, the class automatically detects "file-like" objects (objects with a .read() method or specific tuples) and handles them as multipart file uploads. Non-file parameters are sent as standard data.

    # Instead of this:
# pkg = demo.call_action('package_show', {'id': 'adur_district_spending'})

# Use this:
pkg = demo.action.package_show(id='adur_district_spending')

# For file uploads:
# pkg = demo.call_action('resource_update', {'package_id': 'foo'}, files={'upload': open('file.csv')})

# Use this:
pkg = demo.action.resource_update(package_id='foo', upload=open('file.csv'))

  9. Execute batch CKAN actions via CLI

    master

    The batch command allows you to execute a sequence of CKAN actions provided in a JSONL (JSON Lines) file. It supports parallel execution using multiple worker processes to speed up large-scale operations. Each line in the input file should be a JSON object containing an action key and an optional data dictionary for parameters.

    Input Format Example:

    {"action": "package_create", "data": {"name": "my_dataset"}}
{"action": "resource_create", "data": {"package_id": "...", "url": "..."}}

    Key Features:

    • Parallelism: Use --processes to specify the number of concurrent workers.
    • File Support: Use --local-files to allow the worker to open and upload files specified in the files dictionary of the input JSON.
    • Resuming/Slicing: Use --start-record and --max-records to process specific segments of a large input file.
    • Logging: Use --log <file> to write a detailed execution log of all actions, errors, and responses.

  10. Use the ckanapi CLI for remote and local CKAN operations

    master

    The ckanapi CLI allows you to interact with a remote CKAN instance via its API or a local CKAN installation.

    Remote Mode: Use the -r SITE_URL flag to specify the CKAN server URL. You can authenticate using -a APIKEY.

    Local Mode: If no -r flag is provided, ckanapi attempts to run commands through the local ckan CLI. This requires the ckan command to be available in your system path. You can specify a configuration file using -c CONFIG.

    # Example: Remote action
ckanapi action package_search "q=dataset_name" -r https://ckan.example.org -a YOUR_API_KEY

# Example: Local action (requires 'ckan' CLI installed)
ckanapi action package_search "q=dataset_name" -c /path/to/ckan.ini

  11. Import datasets with resources, datastore fields, and resource views

    master

    When loading datasets, you can perform extended operations by including specific flags. Note that for these to work, the input JSON must contain the corresponding data structures (e.g., resources list with resource_views or datastore_fields keys).

    • --upload-resources: If a resource has a url_type of upload, the command will download the file from the provided url and upload it to CKAN using resource_patch.
    • --datastore-fields: Processes datastore_fields within the resource objects to create/update CKAN Datastore tables.
    • --resource-views: Processes resource_views within the resource objects to create/update CKAN Resource Views.

    Note: The order of operations is critical: Datastore fields are loaded first, then resources are uploaded, then resource views are created.

    ckanapi load datasets --upload-resources --datastore-fields --resource-views

  12. Import logos for groups and organizations

    master

    When loading groups or organizations, you can automatically upload logos. If the input object contains an image_url that is a local path (not a remote URL), the command will download the image from image_display_url and upload it to the CKAN instance.

    • --upload-logo: Enables the automatic processing and uploading of logos for groups and organizations.