DOCSEARCHbyAlgolia
Submit Docs

NASA Cumulus and ORCA Documentation

website·Indexed 30 days ago

https://nasa.github.io/cumulus/

Technical documentation for the Cumulus API (versions 11.0.0, 18.1.0, and 21.0.0) and the ORCA software system. Includes API specifications for authentication and request/response structures, ORCA architectural details regarding database and API containers, developer guides for local debugging with Localstack, and metadata reconciliation between ORCA and Cumulus inventories.

Tokens
2.8M
Snippets
9.8K
Records
19.4K
Agent score
42%

What's inside Cumulus

  1. Overview of the Cumulus API

    v18.4.0
    The Cumulus API provides a web-accessible interface to interact with the Cumulus Framework, allowing developers to monitor status and perform CRUD operations on records. It is the same API that powers the Cumulus dashboard. The API uses HTTPS endpoints and communicates using JSON payloads. All requests to these endpoints require authentication.

  2. Overview of the Cumulus cloud-based data system

    v21.0.0
    Cumulus is a "native" cloud-based data ingest, archive, distribution, and management system designed for Earth Observing System Data and Information System (EOSDIS) data streams. It leverages cloud infrastructure components for processing efficiency and cost reduction and supports data streams from satellite missions, aircraft missions, and field campaigns.

  3. Overview of Cumulus v11.0.0 RDS Integration

    v11.0.0
    Cumulus v11.0.0 introduces integration with RDS, utilizing a PostgreSQL database for archiving data such as granules, files, and executions. During the transition period, the system performs parallel writes to both DynamoDB and PostgreSQL to ensure data redundancy. To move existing data from DynamoDB to PostgreSQL, users must execute specific data migration scripts.

  4. Overview of Cumulus Data Management Types

    v22.2.2

    Cumulus uses several core entities to manage the data lifecycle:

    • Collections: Logical sets of data objects of the same type and version; they provide context for ingest and define where data files are stored.
    • Granules: The smallest independently manageable aggregation of data, always associated with a collection.
    • Providers: Entities that generate and distribute input data (e.g., S3, HTTPS) and tell Cumulus where to retrieve new data.
    • Rules: Logic that associates providers with collections and determines when/how to trigger workflow executions.
    • Workflows: Composed of AWS Lambda Functions and ECS Activities used to discover, ingest, process, manage, and archive data.
    • Executions: Records of a specific workflow run.
    • Reconciliation Reports: Comparisons of data sets used to detect conflicts and ensure agreement.

  5. Overview of the Cumulus cloud-based data system

    v9.9.0
    Cumulus is a "native" cloud-based data ingest, archive, distribution, and management system designed for Earth Observing System Data and Information System (EOSDIS) data streams. It is built to leverage cloud infrastructure for efficiency in processing time and cost, supporting data from satellite missions, aircraft missions, and field campaigns.

  6. Overview of Cumulus Data Management

    v11.0.0
    Cumulus is an open-source set of components designed for NASA's Earth Science data streams to handle cloud-based data ingest, archive, distribution, and management. It is deployed within an AWS account.

  7. Overview of Cumulus Documentation Resources

    v18.2.0

    Cumulus is a cloud-native data ingest, archive, distribution, and management system for Earth Observing System Data and Information System (EOSDIS) data streams. The documentation is split into several key areas:

    • Getting Started: Initial guidance for new users to understand and work in the environment.
    • API Documentation: Technical reference for the Cumulus API.
    • Developer Documentation: Implementation details and READMEs located in the main GitHub repository.
    • Data Cookbooks: Practical guides for data handling.
    • Integrator Guide: Guidance for integrating Cumulus into existing workflows.
    • Operator Docs: Documentation for system operators.

  8. Overview of Cumulus

    v21.2.0
    Cumulus is an open-source set of components designed for NASA's future Earth Science data streams. It provides cloud-based capabilities for data ingest, archive, distribution, and management, primarily deployed within AWS accounts.

  9. Overview of Cumulus execution payload retention

    v18.3.4
    Cumulus automatically stores the initial and final payload values of executions in the originalPayload and finalPayload fields within the RDS database and Elasticsearch. To manage storage costs and requirements, Cumulus uses a CloudWatch rule ({stack-name}-dailyExecutionPayloadCleanupRule) that triggers a cleanExecutions lambda daily to remove payload records older than the configured thresholds.

  10. Overview of the Cumulus workflow module

    v20.3.0
    The Cumulus workflow module allows developers to convert an AWS Step Function definition into a Cumulus workflow. It provides built-in integration for AWS CloudWatch events (to handle database updates for Cumulus objects like executions and granules) and supports the Cumulus throttling feature. Using this module ensures compatibility with future versions of Cumulus.

  11. Overview of the Cumulus API

    v20.1.2
    The Cumulus API provides a web-accessible interface to interact with the Cumulus Framework, allowing developers to monitor status and perform CRUD operations on records (such as providers, collections, granules, and workflows). It is the same API that powers the Cumulus dashboard. The API uses HTTPS endpoints and communicates using JSON payloads. All requests require authentication.

  12. Overview of Localstack for AWS simulation

    Localstack allows developers to simulate an AWS environment locally, removing the need for a remote sandbox during testing. Most services are accessible on port 4566. Note that some features are restricted to the Pro Version and RDS is not supported (a local database should be used instead).