DOCSEARCHbyAlgolia
Submit Docs

Dataverse Documentation

repository·develop·Indexed 21 days ago

https://github.com/iqss/dataverse

An open-source platform for managing, sharing, and preserving research data. This documentation covers the internal user system abstraction (User, GuestUser, AuthenticatedUser), Admin API settings, environment initialization via setup-all.sh, and deployment using Docker and the Dataverse Base Container Image. It also includes guides for the Config Baker container, IntelliJ webapp file copying, and OpenAPI specification validation using the vacuum tool.

Tokens
327K
Snippets
821
Records
1.4K
Agent score
74%

What's inside Dataverse

  1. Overview of Dataverse API capabilities

    develop

    The Dataverse API allows for the automation of researcher and curator workflows. Key task categories include:

    Data Management

    • Depositing Data: Creating collections, creating datasets, uploading files, and publishing collections or datasets.
    • Finding Data: Searching for datasets (e.g., /api/search?q=query), finding recently published content via date ranges or OAI-PMH, and listing collection contents.
    • Downloading Data: Downloading files (individually or by dataset) and exporting dataset metadata in various formats.

    Access Control

    • Managing Permissions: Granting roles, revoking roles, and listing current role assignments (permissions) on Dataverse objects.

    Client Libraries

    If you prefer not to use curl, official client libraries are available for Python, Javascript, R, and Java.

  2. Overview of Dataverse Software APIs

    develop

    Dataverse Software APIs allow users to perform tasks programmatically without using the web interface. Common tasks include:

    • Creating datasets
    • Uploading files
    • Publishing datasets
    • Searching collections, datasets, and files
    • Downloading and subsetting data
    • Managing dataset metadata (via JSON-LD)
    • Migrating datasets from other repositories
    • Transferring large files via S3 or Globus

    APIs enable integrations between Dataverse and other software like OSF (Open Science Framework), OJS (Open Journal Systems), RSpace, or custom scripts (Python, R, etc.).

  3. Overview of Dataverse reporting and metrics tools

    develop

    Several community tools are available for reporting, visualization, and metrics aggregation for Dataverse installations:

    • Matrix: A visualization tool showing collaboration and connectedness between authors and affiliations.
    • Dataverse Installation Web Report: Generates interactive charts from the Dataverse installation Excel Report.
    • Dataverse Installation Reports for Texas Digital Library: A Python 3 tool that uses the native API and database queries to generate and email statistical reports.
    • dataverse-metrics: Aggregates and visualizes metrics for single or multiple Dataverse installations globally.
    • Automated Dataverse Metrics Reports: Uses R-Markdown and GitHub Actions to create customizable reports with minimal effort.

  4. What is Dataverse?

    develop

    Dataverse is an open source web application designed for sharing, preserving, citing, exploring, and analyzing research data. It is organized into a hierarchy where a Dataverse repository hosts one or more Collections.

    Collections can contain:

    • Datasets
    • Sub-collections for further organization

    Each Dataset consists of:

    • Metadata: Descriptive information about the data.
    • Data files: The actual research data.
    • Documentation or code: Supporting files for reproducibility.

  5. Navigate Dataverse documentation by persona

    develop

    Dataverse documentation is organized into specialized guides based on your role and objectives:

    • User Guide: For finding and using data, adding Datasets, administering dataverses/Datasets, and performing dataset exploration or visualizations.
    • Installation Guide: For individuals or organizations intending to host their own Dataverse instance.
    • Container Guide: For deploying Dataverse using container technologies.
    • Developer Guide: For contributing to the Open Source Dataverse project or modifying the source code for custom needs.
    • API Guide: For developers building external applications that need to connect to Dataverse via its APIs.

  7. Understand the Dataverse release process

    develop

    Dataverse releases follow a time-based schedule (quarterly) rather than a feature-based one. This means features that are not ready by the announced release date are deferred to the next release cycle.

    There are two types of releases:

    • Regular releases: Minor (e.g., 6.5) or Major (e.g., 7.0) versions.
    • Hotfix releases: Patch versions (e.g., 6.4.1 or 7.0.1).

    Key lifecycle stages include:

    1. Issue Creation: Running the dv-project-metrics script to generate release issues.
    2. Code Freeze: A period where no new features are merged, allowing focus on stability and testing.
    3. Release Candidate (RC) Building: Generating a .war file via GitHub Actions.
    4. Deployment & Testing: Deploying the RC to Internal and QA environments for curation feedback, performance, and regression testing.
    5. Demo Deployment: Deploying the final RC to the demo server after testing is complete.
    6. Release Notes: Consolidating developer-provided snippets into a final Markdown file.

  8. Branding your Dataverse installation

    develop

    Dataverse installations can be branded using two main approaches:

    1. Simple Branding: Adjust the theme of a Dataverse collection. This allows you to change colors, add a logo, add a tagline, or add a website link to the collection header. These settings are managed via the User Guide under the theme section.

    2. Advanced Customization: This involves replacing specific webpage blocks (Header, Content, and Footer) with custom HTML/CSS files. This method provides complete control over the installation's name, navbar, homepage, and footer.

  9. Core Capabilities of Dataverse

    develop

    Dataverse provides several key functional areas for managing research data:

    Data Management

    • Upload & Structure: Upload data while retaining directory structures to maintain context and reproducibility.
    • Collaboration: Manage datasets by inviting collaborators to work on them before they are published.
    • Access Control: Control access using permissions, configurations, licenses, file restrictions, and guestbooks.
    • Publishing & Downloading: Publish datasets with rich metadata and versioning. Users can download data with clear terms of use and utilize provided citation options.

    Metadata & Discoverability

    • Rich Metadata: Add metadata to datasets before publication, including domain-specific metadata blocks.
    • Harvesting: Metadata can be harvested by web services (e.g., Google Dataset Search) through standardized data descriptions.

    Licensing & Reuse

    • Standardized Licenses: Apply standard licenses to datasets.
    • Custom Terms: Use custom dataset terms for publication scenarios that do not fit standardized licenses.

    Versioning

    • Change Tracking: Track changes via metadata and file updates, categorized into minor and major versions.
    • Version Access: Access and cite specific versions via the version tab on a dataset.
    • Comparison: Compare different versions using the detailed version change overview at the dataset level.

  10. Understand the Dataverse Application Container Image

    develop

    The Dataverse application image is a deployment-ready container that bundles the Dataverse application with the necessary materials to jumpstart it. It runs on top of a base image which provides the underlying application server.

    Key characteristics:

    • Purpose: Bundles the application and configuration glue required to set up the application domain during the application server boot process.
    • Architecture Support: It is a multi-arch image supporting AMD64 (Windows/Linux) and ARM64 (Apple M1/M2).
    • Customization: You can use, extend, or alter this image and host it in your own registry. The build process uses the Maven Docker Plugin.
    • Licensing: Content created by the community is licensed under the Apache License, Version 2.0. Note that the image also contains other software (like Payara Server) which may have different licenses.

  11. Explore Python modules for Dataverse APIs

    develop

    There are several Python libraries available depending on your use case:

    • EasyDataverse: Simplifies dataset management (upload, download, update) using an object-oriented approach and metadata block configurations.
    • python-dvuploader: Implements dv-uploader as a module; supports parallel direct uploads, streaming, and multi-part uploads.
    • pyDataverse: Best for managing collections, datasets, and datafiles, specifically for migrations and DevOps (testing/configuration).
    • UBC's Dataverse Utilities: Console utilities for bulk operations like uploading from TSV spreadsheets, bulk releasing/deleting datasets, and duplicating records.
    • dataverse-client-python: An older library used by the Open Science Framework (OSF) to integrate with Dataverse.
    • Pooch: A general data downloading library that supports Dataverse by allowing users to reference datasets via DOI.
    • idsc.dataverse: Specialized for reading/writing metadata and files to local directories and exporting/importing between Dataverse instances.

  12. Implementation details of the Dataverse backup script

    develop

    The Dataverse backup script is a Python-based utility (tested with v. 2.7.10) designed to perform incremental backups of Dataverse datafiles. It relies on a secondary PostgreSQL database to track backup status and uses various drivers for storage and transport.

    Prerequisites

    The following Python modules are required:

    • psycopg2: PostgreSQL driver for database access.
    • boto3: AWS SDK for S3 storage access.
    • paramiko: SSH client for SFTP transfers.
    • swiftclient: For OpenStack Swift storage (support is incomplete/experimental).

    Core Components

    1. Database Requirements

    • Access: Uses psycopg2 to query the main Dataverse database for changed files and to manage its own tracking database.
    • Tracking Database: A separate database used to maintain the datafilestatus table.
    • Constraint: This tracking database must reside on the same server as the main Dataverse database and be owned by the same PostgreSQL user.
    • Setup: Refer to README_HOWTO.txt for instructions on setting up this database before running the script.

    2. Storage and Backup Logic

    • Storage Access (Reading): Handled via storage_filesystem.py (local) and storage_s3.py (S3). To add new storage types (e.g., Swift), implement an open_storage_object... method that returns a byte stream.
    • Backup Access (Writing): Handled via backup_ssh.py (default) and backup_swift.py (experimental). New backup methods must:
      1. Copy the byte stream to the target storage.
      2. Verify the copy against the Dataverse checksum (MD5 or SHA1).

    3. Backup Status Tracking

    The script uses the datafilestatus table in the backup database to manage incremental backups and error states:

    • OK: File successfully backed up.
    • FAIL_READ: Failed to read the file from Dataverse storage.
    • FAIL_WRITE: Failed to copy or verify the backup copy.
    • Incremental Logic: The script uses the Datafile createdate timestamp to identify files created after the latest timestamp recorded in the backup database.

    4. Notifications

    The script uses the Unix mail command to send status reports. This requires the system to have the mail command installed and configured for command-line email sending.

    If you need to use a remote SMTP server instead, you can modify email_notification.py to use smtplib and email.mime.text as shown in the implementation notes.