PyNWB: Python API for Neurodata Without Borders

PyNWB is built on a modular architecture designed to separate data representation, specification, and I/O. The four primary abstractions are:

1. Container: In-memory objects that serve as the primary interface for most applications. There is one Container for each neurodata_type in the NWB schema (e.g., pynwb.ecephys, pynwb.ophys). They behave similarly to a table row.
2. Builder: Intermediary objects used for I/O. Backend readers and writers exchange Builders rather than Containers to decouple the in-memory model from the storage format. Common types include GroupBuilder, DatasetBuilder, LinkBuilder, and ReferenceBuilder.
3. Spec (Specification): Python representations of YAML specifications. They define the structure, data types, and dimensions of NWB components. Key classes include NWBAttributeSpec, NWBGroupSpec, NWBDatasetSpec, and NWBLinkSpec.
4. ObjectMapper: The bridge between Containers and Builders. It maintains the mapping between Container attributes and Spec components, allowing for conversion between the in-memory Container and the I/O-ready Builder.

Tutorials in pynwb are generated using sphinx-gallery. Source files are located in docs/gallery/, organized into subfolders representing gallery subsections.

Steps to add a tutorial:

1. Add the Python file: Place a new .py file in the appropriate subfolder within docs/gallery/.

   • To render output (plots/data): Include the prefix plot_ in the filename (e.g., plot_my_tutorial.py).
   • To render code and text only: Use any other filename without the plot_ prefix.

2. Manage output files: If your tutorial script generates files, add their names to the files_to_remove variable within the clean_up_tests() function in test.py. This ensures the test suite cleans up generated artifacts after the sphinx-gallery tests run.

3. Set position (Optional): By default, tutorials are sorted alphabetically. To specify a custom order, update the GALLERY_ORDER variable of the CustomSphinxGallerySectionSortKey class in docs/source/conf.py.

4. Build and Verify:

   • Build the documentation: cd docs && make html
   • View the result: open docs/_build/html/index.html

5. Submit: Create a Pull Request to include your tutorial in the repository.

# Build the documentation
cd docs
make html

# View the documentation
open docs/_build/html/index.html

If you use PyNWB in your research, please use the following BibTeX citation to credit the Neurodata Without Borders ecosystem:

@article {10.7554/eLife.78362,
       article_type = {journal},
       title = {{The Neurodata Without Borders ecosystem for neurophysiological data science}},
       author = {R"ubel, Oliver and Tritt, Andrew and Ly, Ryan and Dichter, Benjamin K. and
                Ghosh, Satrajit and Niu, Lawrence and Baker, Pamela and Soltesz, Ivan and
                Ng, Lydia and Svoboda, Karel and Frank, Loren and Bouchard, Kristofer E.},
       editor = {Colgin, Laura L and Jadhav, Shantanu P},
       volume = {11{,
       year = {2022},
       month = {oct},
       pub_date = {2022-10-04},
       pages = {e78362},
       citation = {eLife 2022;11:e78362},
       doi = {10.7554/eLife.78362},
       url = {https://doi.org/10.7554/eLife.78362},
       keywords = {Neurophysiology, data ecosystem, data language, data standard, FAIR data, archive},
       journal = {eLife},
       issn = {2050-084X},
       publisher = {eLife Sciences Publications, Ltd},
  }

You can create a modified version of an existing NWB file while preserving the original by using the export feature.

Workflow:

1. Open the existing file using NWBHDF5IO in read mode (mode='r').
2. Read the file into an NWBFile object.
3. Modify the NWBFile object or its child objects (e.g., add/remove containers, change attributes).
4. If you changed container attributes, you must call nwbfile.set_modified() before exporting.
5. Open a new NWBHDF5IO in write mode (mode='w') and call export(src_io=read_io, nwbfile=nwbfile).

Important Notes:

• Dataset Modifications: Modifications to h5py.Dataset objects act directly on the source file on disk. To avoid unintentional changes to the source file, ensure the source is opened with mode='r'. If you want to modify a dataset only in the new file, replace the entire object with a new array holding the modified data.
• Object IDs: Exported files will have the same object IDs as the original file.
• Container Movement: Moving containers within the same file is not currently supported via export.

with NWBHDF5IO(self.read_path, mode='r') as read_io:
    nwbfile = read_io.read()
    # ...  # modify nwbfile
    nwbfile.set_modified()  # necessary if modifications are changes to attributes

    with NWBHDF5IO(self.export_path, mode='w') as export_io:
        export_io.export(src_io=read_io, nwbfile=nwbfile)

Releases on conda-forge should ideally be handled by the regro-cf-autotick-bot, which monitors PyPI and automatically creates PRs for new versions. If manual intervention is required, you must first publish to PyPI.

Manual Steps:

1. Fork and clone the pynwb-feedstock repository.
2. Download the source .tar.gz for the new release.
3. Update recipe/meta.yaml with the new version and sha256 values.
4. Push the changes to your fork and create a Pull Request against the main feedstock repository.

# Update version and sha256 in meta.yaml (Linux)
$ sed -i "2s/.*/{% set version = "$release" %}/" recipe/meta.yaml
$ sha=$(openssl sha256 /tmp/pynwb-$release.tar.gz | awk '{print $2}')
$ sed -i "3s/.*/{$ set sha256 = "$sha" %}/" recipe/meta.yaml

# Update version and sha256 in meta.yaml (macOS)
$ sed -i -- "2s/.*/{% set version = "$release" %}/" recipe/meta.yaml
$ sha=$(openssl sha256 /tmp/pynwb-$release.tar.gz | awk '{print $2}')
$ sed -i -- "3s/.*/{$ set sha256 = "$sha" %}/" recipe/meta.yaml