DOCSEARCHbyAlgolia
Submit Docs

sodapy

repository·main·Indexed 19 days ago

https://github.com/afeld/sodapy

A Python client for the Socrata Open Data API (SODA) that enables developers to programmatically interact with datasets hosted on Socrata platforms. It supports read and write operations, including retrieving data via SoQL queries, managing dataset metadata, and performing upsert, replace, and delete operations on rows and datasets.

Tokens
5.2K
Snippets
23
Records
24
Agent score
61%

What's inside sodapy

  1. Overview of sodapy

    main

    sodapy is a Python client designed to interact with the Socrata Open Data API. It allows you to perform read and write operations on Socrata datasets.

    Note on Write Operations:

    • Use sodapy for direct writing to datasets via the Socrata Open Data API.
    • If your write operations require data transformations via the Socrata Data Management Experience (the web UI), you should use the Socrata Data Management API instead. A dedicated Python SDK for that API is available at socrata-py.

  2. Initialize the Socrata client

    main

    To interact with Socrata, import Socrata from sodapy and instantiate a client.

    • username and password are required for write operations (creating or modifying data).
    • An application token can be None, but providing one helps avoid strict throttling limits.
    • You can use a context manager (with statement) to ensure the session is closed automatically.
    • You can increase the timeout for large requests.
    from sodapy import Socrata

# Full authentication for write access
client = Socrata(
    "sandbox.demo.socrata.com",
    "FakeAppToken",
    username="fakeuser@somedomain.com",
    password="mypassword",
    timeout=10
)

# Bare-bones client for read-only access
client = Socrata("sandbox.demo.socrata.com", None)

# Using a context manager
with Socrata("sandbox.demo.socrata.com", None) as client:
    # do some stuff

  4. Setup sodapy and authentication

    main

    To use sodapy, import the Socrata class. Authentication is optional; you can pass None as the token, but requests will be rate limited. To avoid rate limiting, generate an App Token at https://opendata.socrata.com/signup and set it as an environment variable.

    To set the environment variable in your terminal:

    export SODAPY_APPTOKEN=<token>
    import os
import pandas as pd
import numpy as np
from sodapy import Socrata

socrata_domain = "opendata.socrata.com"
socrata_dataset_identifier = "f92i-ik66"
socrata_token = os.environ.get("SODAPY_APPTOKEN")

client = Socrata(socrata_domain, socrata_token)

  5. Download dataset attachments

    main

    Use client.download_attachments(dataset_identifier, download_dir) to download all files associated with a dataset. It returns a list of the local file paths where the attachments were saved.

    paths = client.download_attachments("nimj-3ivp", download_dir="~/Desktop")
# Output: ['/Users/xmunoz/Desktop/nimj-3ivp/File1.PDF', ...]

  6. Get all rows from a dataset using a generator

    main

    If you need to process all rows in a dataset, use client.get_all(dataset_identifier, ...). This method returns a generator that handles pagination automatically, making it memory-efficient for large datasets.

    # Iterate through all items
for item in client.get_all("nimj-3ivp"):
    print(item)

# Use itertools to grab a specific number of items from the generator
import itertools
items = client.get_all("nimj-3ivp")
first_five = list(itertools.islice(items, 5))

  7. Create and publish a new dataset

    main

    To create a new dataset, use client.create(name, ...).

    Supported keyword arguments:

    • description: String description.
    • columns: A list of field definitions (e.g., [{"fieldName": "f", "name": "N", "dataTypeName": "text"}]).
    • category: The dataset category (must exist in /admin/metadata).
    • tags: A list of tag strings.
    • row_identifier: The field name of the primary key.
    • new_backend: Boolean indicating if it should use the new backend.

    After creation, the dataset is in 'working copy' mode. Use client.publish(dataset_identifier) to make it live.

    columns = [
    {"fieldName": "delegation", "name": "Delegation", "dataTypeName": "text"},
    {"fieldName": "members", "name": "Members", "dataTypeName": "number"}
]
tags = ["politics", "geography"]

# Create
new_ds = client.create(
    "Delegates", 
    description="List of delegates", 
    columns=columns, 
    row_identifier="delegation", 
    tags=tags, 
    category="Transparency"
)

# Publish using the ID returned from create
client.publish(new_ds['id'])

  8. Upsert and Replace data in a dataset

    main

    Use these methods to modify existing data or add new rows.

    • upsert(dataset_identifier, payload): Creates new rows or updates existing ones. The payload can be a list of dictionaries or a file object (like a CSV).
      • To update/delete, include the :id or :deleted: True in the payload.
    • replace(dataset_identifier, payload): Similar to upsert, but it overwrites existing data.

    Both methods accept a file object (e.g., an open CSV file) as the payload.

    # Upsert new rows
data = [{'Delegation': 'AJU', 'Name': 'Alaska', 'Key': 'AL', 'Entity': 'Juneau'}]
client.upsert("eb9n-hr43", data)

# Upsert with updates and deletions
data = [
    {'Delegation': 'sfa', ':id': 8, 'Name': 'bar', 'Key': 'doo', 'Entity': 'dsfsd'}, 
    {':id': 7, ':deleted': True}
]
client.upsert("eb9n-hr43", data)

# Upsert using a CSV file
with open("upsert_test.csv") as f:
    client.upsert("eb9n-hr43", f)

# Replace using a CSV file
with open("replace_test.csv") as f:
    client.replace("eb9n-hr43", f)

  9. Get data from a dataset

    main

    Use client.get(dataset_identifier, ...) to retrieve data.

    • To get a single row, use the format dataset_identifier/row_id.
    • You can filter and query data using SoQL keywords via arguments like where and order.
    • Use exclude_system_fields=False if you need to see system-generated metadata fields.
    • You can also pass specific field names as keyword arguments to filter by value (e.g., region="Kansas").
    # Get a subset of rows
client.get("nimj-3ivp", limit=2)

# Query with SoQL
client.get("nimj-3ivp", where="depth > 300", order="magnitude DESC", exclude_system_fields=False)

# Get a specific row by ID
client.get("nimj-3ivp/193", exclude_system_fields=False)

# Filter by a specific field value
client.get("nimj-3ivp", region="Kansas")

  10. Retrieve datasets in a domain

    main

    Use client.datasets() to retrieve a list of datasets associated with the domain used to initialize the client. You can use limit and offset to paginate through the results.

    >>> client.datasets()
[{"resource" : {"name" : "Approved Building Permits", "id" : "msk6-43c6", ...}, ...]

  11. Delete rows or datasets

    main

    Use client.delete(dataset_identifier, row_id=None) to remove data.

    • To delete a specific row, provide the row_id.
    • To delete the entire dataset, call delete with only the dataset_identifier.
    # Delete an individual row
client.delete("nimj-3ivp", row_id=2)

# Delete the entire dataset
client.delete("nimj-3ivp")

  12. Manage dataset metadata

    main

    You can retrieve and update the metadata for a dataset.

    • get_metadata(dataset_identifier): Returns a dictionary of the dataset's metadata (e.g., license, owner, category).
    • update_metadata(dataset_identifier, update_fields): Overwrites specific metadata keys. update_fields must be a dictionary containing only the keys you wish to change.

    Warning: Invalid payloads to update_metadata can corrupt the dataset or its visualization.

    # Get metadata
metadata = client.get_metadata("nimj-3ivp")

# Update a specific field
client.update_metadata("nimj-3ivp", {"attributionLink": "https://anothertest.com"})