DOCSEARCHbyAlgolia
Submit Docs

DBDB.io Documentation

repository·master·Indexed 20 days ago

https://github.com/cmu-db/dbdb.io

DBDB.io is a 'Database of Databases' platform serving as a registry or catalog for database systems. Built with Django and PostgreSQL, it includes management tools for approving system versions, bulk-updating attributes and features, collecting repository statistics, and computing system embeddings using transformer models.

Tokens
17K
Snippets
63
Records
77
Agent score
69%

What's inside DBDB.io

  3. Regenerate test fixtures from production

    master

    If you need to update local test data with current production data, use manage.py dumpdata.

    Prerequisites:

    • You must be connected via Tailscale to reach db-web.tail5fc291.ts.net.
    • The DATABASE_URL in your .env file must point to the production database.

    This process dumps Feature, FeatureOption, Attribute, and AttributeOption rows into JSON files located in data/fixtures/.

    # Features and feature options
uv run python manage.py dumpdata core.feature core.featureoption \
    --indent 2 -o data/fixtures/core_features.json

# Attributes and attribute options
uv run python manage.py dumpdata core.attribute core.attributeoption \
    --indent 2 -o data/fixtures/core_attributes.json

  4. Load test fixtures into a local database

    master

    Use manage.py loaddata to populate your local database with the standard set of test fixtures (admin users, core features, attributes, etc.).

    uv run python manage.py loaddata adminuser testuser \
    core_features core_attributes core_system core_savedsearch

  5. Run the test suite

    master

    Run tests using manage.py test. The runner automatically creates and destroys a dedicated database named test_dbdb_io for every run.

    Requirements:

    • A local PostgreSQL instance must be running.
    • You must provide a PGUSER environment variable.
    • The dbdb.test_settings module is used to override production settings with local PostgreSQL parameters.
    PGUSER=<your-local-pg-user> uv run python manage.py test dbdb.core \
    --settings=dbdb.test_settings --verbosity=2

  6. Install DBDB.io

    master

    To install the necessary system dependencies and synchronize the Python environment using uv, run the following commands. Note that sudo is required for system package installation.

    sudo apt-get install build-essential libffi-dev libpq-dev python3-dev postgresql-common-dev libcairo2
uv sync

  7. How repository abandonment and disabling works

    master

    The collect_repo_info command manages repository lifecycle through two mechanisms:

    Automatic Disabling

    If a repository fails to provide valid data repeatedly, it is automatically disabled. Specifically, if the last _FAILED_DISABLE_THRESHOLD (set to 3) consecutive snapshots have a status of RepositorySnapshot.Status.FAILED, the RepositoryInfo.enabled flag is set to False.

    Abandonment Checking

    If the --check-abandoned flag is provided, the command runs check_abandoned() for each system. This uses the REPOSITORY_INACTIVITY_DAYS setting to determine if a repository has had no activity for a sufficient period. If it has, the repository is marked as abandoned.

  8. How resurrection scanning works in collect_repo_info

    master

    When the --check-resurrection flag is used, the command changes its behavior:

    1. Resurrection Only Mode: If --check-resurrection is active and --check-abandoned is NOT active, the command skips snapshot collection for enabled repositories and focuses solely on scanning abandoned ones.
    2. Detection: It scans systems tagged as "Abandoned" for signs of recent activity.
    3. Action: If recent activity is found, it creates a pending SystemVersion for administrator review.

    If --check-abandoned is also enabled, the command performs standard snapshot collection and then checks for abandonment as part of the normal workflow.

  9. How keyword matching works in update_citations

    master

    The keyword positional argument determines which CitationUrl records are selected based on its format:

    1. Numeric Input: If the keyword is an integer, the command performs an exact match on the record's primary key (pk).
    2. String Input: If the keyword is not an integer, the command performs a case-insensitive substring match on the URL field (url__icontains).

  10. Migrate attributes using copy_attributes

    master

    The copy_attributes management command is used for data migration to move Tag, License, OperatingSystem, ProgrammingLanguage, and ProjectType rows into the Attribute and AttributeOption models. It also re-links existing SystemVersion Many-to-Many (M2M) relationships to the new attr_* fields.

    Migration Workflow

    To ensure data integrity, follow this specific run order:

    1. Apply initial migrations: Run manage.py migrate to apply migrations 0059 and 0060 (which add the Attribute/AttributeOption models and the sv_field/search_text fields).
    2. Execute attribute copy: Run manage.py copy_attributes (use --dry-run to preview changes).
    3. Apply final migrations: Run manage.py migrate to apply migration 0061 (which removes old fields and renames the attr_* fields).

    Options

    • --dry-run: Prints what would be done without writing any changes to the database. If used, all database operations are rolled back at the end of the command.
    # 1. Apply initial migrations
python manage.py migrate

# 2. Run the copy command (with dry-run to test)
python manage.py copy_attributes --dry-run

# 3. Apply final migrations
python manage.py migrate

  11. Enrich Organization data using the enrich_organization command

    master

    The enrich_organization management command uses an LLM to automatically fill missing fields in Organization records. It can perform two main tasks: general field enrichment (description, stock symbols, etc.) and specific URL extraction (LinkedIn, Crunchbase) from the organization's homepage.

    Workflow:

    1. Identify Targets: Select organizations by keyword (slug or name), by a specific org_type, or by targeting organizations where a specific FIELD is currently empty.
    2. Enrichment Mode: The command can run in enrich mode (filling various fields), extract-urls mode (specifically looking for social/business URLs), or both.
    3. Data Gathering: Optionally crawls existing URLs associated with the organization to provide context to the LLM.
    4. LLM Processing: Sends a prompt and a tool schema to the LLM to suggest values.
    5. Validation & Saving: Validates suggested citations and saves the data directly to the database (note: this bypasses versioning/pending flows).

    Important Note: If the organization type is COMPANY, the command may also attempt to find and update the full legal name (including suffixes like 'Inc.' or 'GmbH') and update the organization's slug accordingly.

    # Example: Enrich organizations of type 'company' that are missing a 'description'
python manage.py enrich_organization --search-type company --missing description

# Example: Extract LinkedIn and Crunchbase URLs for a specific organization by slug
python manage.py enrich_organization my-org-slug --mode extract-urls

  12. Use the enrich_system command to fill missing SystemVersion data

    master

    The enrich_system command uses an LLM to automatically identify and fill missing information for a SystemVersion. It crawls known URLs (like the system's homepage or GitHub repository), passes the text to an LLM along with the project's taxonomy, and validates the results.

    Key Behaviors:

    • Non-destructive: It never overwrites existing data. Instead, it creates a new SystemVersion with approved=False (a pending version) containing the suggested values.
    • Fields Supported:
      • Simple text (description, history)
      • Integers (start_year, end_year)
      • URL Foreign Keys (system_url, docs_url, blog_url, sourcerepo_url, wikipedia_url, twitter_url)
      • M2M Attributes (project_types, licenses, oses, written_in, tags)
      • Features (via Feature slugs)
    • Validation: Suggested URLs are validated for reachability, and LLM-suggested citations are verified via fetch_url_metadata().
    python manage.py enrich_system <slug> [options]