Open Collective API

repository·main·Indexed 19 days ago

https://github.com/opencollective/opencollective-api

The core backend service for Open Collective, providing API functionality and managing collective data. It features a PostgreSQL-backed data layer, GraphQL API, and integrated support for local email testing. The documentation covers local development setup, database migrations using Sequelize, payment provider implementation, backup and restoration processes, and deployment to staging and production via Heroku.

Tokens
22.9K
Snippets
65
Records
88
Agent score
66%

What's inside opencollective-api

  1. Understand PayPal API schema changes from V1 to V2

    main

    When working with the PayPal integration, be aware of the mapping differences between the legacy V1 schema and the current V2 schema. Specifically, the concept of a Transaction in V1 is now represented as a Capture within a Payment object in V2. This Capture contains the payment breakdown and corresponds to the internal Transactions table.

    // V1 Concept
    "Transaction"
    
    // V2 Concept
    "Payment" > "Capture"
  2. Understand the Open Collective backup archive structure

    main

    A full backup archive follows a specific directory structure organized by service and environment. When extracted, the root directory (e.g., backup-YYYY-MM-DD/) contains:

    • heroku/: Contains Heroku app backups. Each app directory (e.g., opencollective-prod-api/, oc-prod-frontend/, oc-metabase/) includes a postgres.dump for the database and a .env file for environment variables.
    • s3-buckets/: Contains the contents of AWS S3 buckets, organized by bucket name (e.g., opencollective-production/).
    • README.md: Documentation for the specific backup instance.
    backup-YYYY-MM-DD/
    ├── README.md                    # This file
    ├── heroku/
    │   ├── opencollective-prod-api/
    │   │   ├── postgres.dump        # Main production database dump
    │   │   └── .env                 # Environment variables
    │   ├── oc-prod-frontend/
    │   │   └── .env                 # Environment variables
    │   └── oc-metabase/
    │       ├── postgres.dump        # Metabase database dump
    │       └── .env                 # Environment variables
    └── s3-buckets/
        ├── opencollective-production/
        └── opencollective-production-us-tax-forms/
  3. Understand the privacy model for organizations

    main

    Private organizations (including collectives, funds, and hosts) are designed to be usable via the dashboard and authorized integrations only. They are not discoverable or readable via public profiles.

    Key constraints of the privacy model:

    • No Visibility Toggle: Privacy is immutable once created. There is no user-facing flow to switch an account from public to private or vice versa. Private accounts are currently created via manual operations.
    • Fiscal Host Inheritance: Privacy is inherited down a fiscal host tree. If a host is private, all its hosted children (projects, events, etc.) are also private. A single fiscal host tree must not mix public and private hosted accounts.
    • Siloing: Privacy ensures that visibility into one private collective does not imply visibility into another, unless specific roles (ADMIN/ACCOUNTANT) grant access.
  4. Business rules and restrictions for private organizations

    main

    Private organizations are subject to specific business logic constraints to prevent accidental data leakage or cross-tree bridging:

    • Cross-host expenses: Private payee accounts are prohibited from submitting expenses to collectives that reside under a different fiscal host (assertPrivateOrganizationNoCrossHostExpense).
    • Adding funds: For a private host, the canAddFundsFromAccount check restricts source accounts to those belonging to that host's specific tree. The trusted-host or allow-all-accounts flags are only applicable to non-private hosts.
    • Feature availability: Many public-facing features are either disabled or marked as UNSUPPORTED for private accounts, including public contribution flows (donate/checkout), public tier marketing, public funding goals, and public profile personalization.
  5. Determine access permissions for private accounts

    main

    Access to a private account is governed by the canSeePrivateAccount logic. A user is authorized to see a private account only if they meet one of the following criteria:

    1. They are a root admin.
    2. They hold an ADMIN or ACCOUNTANT role on the account itself.
    3. They hold an ADMIN or ACCOUNTANT role on the account's fiscal host.
    4. They hold an ADMIN or ACCOUNTANT role on the account's parent (applicable for projects and events).
    5. They are an ADMIN/ACCOUNTANT on a hosted collective belonging to a private host organization (allowing host admins to access the host dashboard).

    Unauthorized users attempting to access private accounts will receive a Forbidden error, which distinguishes the request as "exists but denied" rather than a generic "not found."

  6. Delete Wise webhooks

    main

    When you are finished testing, you should remove the Wise webhook to clean up your environment. Use the scripts/setup-transferwise-webhook.js script with the down command and the specific webhook ID provided when the webhook was created.

    $ npm run script scripts/setup-transferwise-webhook.js down "YOUR_WEBHOOK_ID"
  7. Run admin scripts from the scripts directory

    main

    The repository contains various admin scripts located in the /scripts directory. To run these scripts in a local development environment (without Docker), use npx babel-node to execute the JavaScript files.

    # Example: Running the populate_usernames script
    $ npx babel-node ./scripts/populate_usernames.js
  8. View email templates locally

    main

    You can preview specific email templates locally by running the compilation script. Ensure that the necessary data for the template is provided in scripts/compile-email.js to allow the template to render correctly.

    Run the following command, replacing <template name> with the name of the template you wish to view:

    npm run compile:email <template name>
  9. Prerequisites for restoring an Open Collective backup

    main

    Before attempting to restore a full cold backup created by full-backup.sh, ensure the following tools are installed and available in your environment:

    • 7zip (7z command): Required for extracting the encrypted backup archive.
    • Heroku CLI (heroku command): Required for restoring Heroku database backups.
    • AWS CLI (aws command): Required for restoring S3 bucket contents.
    • PostgreSQL client tools: Required for manual database operations if necessary.
  10. Prevent data leakage when implementing private organization queries

    main

    When adding new GraphQL queries or fields that return an Account (or any concrete account type), you must implement privacy gates to prevent leaking names, slugs, balances, or relationship graphs. Any new read path is considered high risk by default.

    To secure a new field, you should use one of the following established privacy strategies:

    • entry-gate: Use an assertion at the entry point of the query.
    • parent-gate: Gate access based on the parent account's visibility.
    • no-private: Explicitly filter out accounts where isPrivate: true (e.g., in membership lists).
    • admin-only: Restrict access to the organization's administrators.
    • skipped: Mark the field as not yet supporting private account privacy (e.g., certain OpenSearch paths).

    Key implementation patterns:

    • Account Lookups: Use assertCanSeeAccount on V2 account queries.
    • Memberships: Apply a default filter isPrivate: false on joined collectives in memberOf or memberships queries.
    • Orders/Contributions: Use assertOrderAccessibleForPrivateCollective to ensure only host admins, people who can see the collective, or the 'from' collective can access the record.
    • Expenses: Use assertExpenseAccessibleForPrivateCollective after performing host/collective checks.
  11. Register a new model and its associations

    main

    To make the new model available to the application, you must import it and add it to the setupModels function in models/index.ts. If the new model has relationships with existing models, define those associations within setupModels using Sequelize association methods (e.g., .belongsTo(), .hasMany()).

    // Add the table to the map of models
    import MyTable from './MyTable';
    
    export function setupModels() {
      const m = {}; // models
      // ...
      m['MyTable'] = MyTable;
      // ...
    
      // If you want to add associations, you have to do it here:
      m.MyTable.belongsTo(m.Collective, { foreignKey: 'CollectiveId', as: 'collective' });
    }
  12. Set up a PayPal Merchant account for development

    main

    To test PayPal integration in a sandbox environment, you must create a PayPal app and configure the API with the resulting credentials.

    1. Create a PayPal app at https://developer.paypal.com/developer/applications/create.
    2. Set PAYPAL_ENVIRONMENT=sandbox and PAYPAL_APP_ID in your API's .env file using the generated credentials.
    3. Encrypt your PAYPAL_CLIENT_SECRET using the provided encryption script.
    4. Manually insert a record into the ConnectedAccounts database table to link the merchant credentials to a CollectiveId.
    5. Create a buyer test account at https://developer.paypal.com/developer/accounts/create to simulate payments.
    # 1. Set env vars in .env
    PAYPAL_ENVIRONMENT=sandbox
    PAYPAL_APP_ID=your_app_id
    
    # 2. Encrypt the client secret
    npm run script scripts/encrypt.js PAYPAL_CLIENT_SECRET
    
    # 3. Insert into database
    INSERT INTO "ConnectedAccounts" ("service", "clientId", "token", "CollectiveId", "createdAt", "updatedAt")
    VALUES (E'paypal', clientId, clientSecret, hostCollectiveId, NOW(), NOW());