DOCSEARCHbyAlgolia
Submit Docs

DataJoint for Python

repository·master·Indexed 41 days ago

https://github.com/datajoint/datajoint-python

A framework for scientific workflow management based on the Relational Workflow Model. DataJoint uses database schemas to define execution logic and dependencies, ensuring full lineage and reproducibility of scientific data. It includes a Python CLI, support for JSON data types, and a DatabaseAdapter system for implementing custom database backends.

Tokens
16.3K
Snippets
52
Records
81
Agent score
52%

What's inside DataJoint

  1. What is the Relational Workflow Model in DataJoint?

    master

    DataJoint is a framework for scientific data pipelines built on the Relational Workflow Model. In this paradigm, your database schema serves as an executable specification of your workflow. Key principles include:

    • Tables represent workflow steps: Each table in your schema corresponds to a specific step in your pipeline.
    • Foreign keys encode dependencies: Dependencies between steps are managed via foreign keys; a child table cannot be populated until its parent tables are populated.
    • Computations are declarative: You define what needs to be computed, and DataJoint manages when and how to execute those computations based on the schema.
    • Results are immutable: This model ensures full lineage and reproducibility of scientific data.

  3. Run full Docker testing (CI mode)

    master

    To run the full testing suite within a Docker environment (similar to a CI pipeline), use the test profile to bring up the djtest service, which builds the environment and executes the test suite.

    docker compose --profile test up djtest --build

  5. Run tests using testcontainers

    master

    For standard testing, docker-compose is optional. The test suite uses testcontainers to automatically manage the lifecycle of required containers. You can run the tests directly using pytest.

    pytest tests/

  7. Use docker-compose for persistent development environments

    master

    If you want persistent containers that survive between test runs (useful for development and debugging), you can manually start the database and MinIO services using docker compose before running tests.

    docker compose up -d db minio
pytest tests/

  9. Load schemas as virtual modules via the CLI

    master

    When using the dj CLI, you can use the -s or --schemas flag to load database schemas as virtual modules. This maps a schema name to a Python alias, allowing you to access tables directly via the alias in the REPL.

    Format: schema_name:alias

    Once loaded, you can interact with the schema's tables using the alias. For example, if you load my_lab:lab, you can call lab.TableName.

    # Load schema 'my_lab' with the alias 'lab'
dj -s my_lab:lab

# Inside the REPL:
>>> lab.Subject.to_dicts()

  10. Use `dj.U` (Universal Set) for restrictions and aggregations

    master

    The dj.U class represents a universal set of all possible values for specified attributes. It is used to perform advanced query operations that cannot be done with standard tables alone.

    Common Operations with dj.U

    1. Restriction: Use the & operator to find unique combinations of attributes present in a table. dj.U('attr1', 'attr2') & table yields all unique combinations of attr1 and attr2 found in table.

    2. Aggregation: Perform summary calculations over an entire set using .aggr().

      • Total count: dj.U().aggr(expr, n='count(*)')
      • Distinct values: dj.U().aggr(expr, n='count(distinct attr)')
      • Summation: dj.U().aggr(expr, s='sum(attr)')
      • Grouped aggregation: dj.U('attr1', 'attr2').aggr(expr, n='count(*)')

    3. Joins: While the * operator was removed in DataJoint 2.0, dj.U is still used to define the scope of aggregations and restrictions.

    # Enumerate unique combinations of contrast and brightness in stimulus
dj.U('contrast', 'brightness') & stimulus

# Total count using universal set
dj.U().aggr(expr, n='count(*)')

# Count per group
dj.U('attr1', 'attr2').aggr(expr, n='count(*)')

  11. Use the <npy@> codec for NumPy array storage

    master

    The <npy@> codec allows you to store NumPy arrays as standard .npy files using schema-addressed paths. This ensures portability (readable by MATLAB, NumPy, etc.) and enables lazy loading.

    To use a specific storage backend (e.g., an archive), use the @ modifier: <npy@archive>.

    Key Features:

    • Lazy Loading: Metadata like shape and dtype are available without downloading the actual data.
    • Transparent Integration: Using an NpyRef (the object returned when fetching) in NumPy operations like np.mean() or arithmetic triggers an automatic download.
    • Schema-Addressed: Files are stored in a structured path mirroring your database: {schema_prefix}/{schema}/{table}/{pk}/{attribute}_{token}.npy.

    Note: Deleting rows in the database does not automatically delete the files. You must use dj.gc.GarbageCollector to clean up orphaned .npy files.

    @schema
class Recording(dj.Manual):
    definition = '''
    recording_id : int
    ---
    waveform : <npy@>           # default store
    spectrogram : <npy@archive>  # specific store
    '''

# Insert - just pass the array
Recording.insert1({
    'recording_id': 1,
    'waveform': np.random.randn(1000, 32),
})

  12. How QueryExpression operators work

    master

    A QueryExpression object represents a SQL SELECT statement. You can derive new entity sets from existing ones using several core operators. These operators do not always create a subquery; DataJoint optimizes the SQL generation based on whether the operation modifies the heading, applies restrictions to computed attributes, or involves specific subclasses like Join, Aggregation, or Union.

    Core operators include:

    • restrict(restriction): Filters the entity set (equivalent to & for AND or - for NOT).
    • join(other): Combines two entity sets (equivalent to *).
    • proj(*attributes, **named_attributes): Selects or renames attributes.
    • aggr(group, *attributes, ...): Performs grouping and aggregation.
    • union(other): Combines two entity sets (equivalent to +).
    # Example of operator chaining
result = (TableA.restrict(attr == 1) 
          .join(TableB) 
          .proj('attr1', new_attr='attr2 + 1'))