DOCSEARCHbyAlgolia
Submit Docs

DeepDataSpace (DDS)

repository·main·Indexed 42 days ago

https://github.com/idea-research/deepdataspace

An open-source tool for Computer Vision (CV) data visualization, intelligent annotation, and model analysis. It provides a web interface for managing datasets, including features for image re-ranking by flags and asynchronous task tracking via Redis and Celery.

Tokens
52.5K
Snippets
151
Records
190
Agent score
54%

What's inside DeepDataSpace

  1. Understand the deepdataspace-app directory structure

    main

    The project follows a standard UmiJS structure. Key directories include:

    • src/components: Common reusable components.
    • src/hooks: Common React hooks.
    • src/models: Global data models for state management.
    • src/pages: Page-level components, styles, and local models.
    • src/services: API definitions.
    • src/utils: Common utility functions.
    • src/app.tsx: The main application entry point.
    • public/: Static resources accessible to the browser.
    • dist/: The output directory after running the build command.

  2. How data flow works in deepdataspace-app

    main

    The application uses the @umi/max built-in data flow management plugin, which is a lightweight solution based on the hooks paradigm for managing global shared data. It relies on a convention-based directory structure.

    Commenting guidelines for models:

    • No Route Operations: Do not perform route-related operations within a model.
    • Lifecycle Management: Use usePageModelLifeCycle within a model to implement lifecycle events and handle page loading/unloading logic.
    • URL Synchronization: Standardize pageState variables in the model. Use usePageModelLifeCycle to synchronize these variables with the URL to ensure state is preserved during navigation.

  3. Install DeepDataSpace via PyPI

    main

    To install DeepDataSpace (DDS) using the Python package manager, first ensure you meet the prerequisites, then upgrade pip and install the deepdataspace package.

    Prerequisites

    • Python Version: 3.8 - 3.10
    • Supported Platforms:
      • Mac OS: x86/x64, arm64
      • Windows 10: x86/x64
      • Ubuntu LTS (18.04+): x86/x64
      • Docker Compose: x86/x64, arm64
    python3 -m pip install pip --upgrade
python3 -m pip install deepdataspace

  4. Develop and build deepdataspace-app

    main

    To work on the frontend, install the project dependencies, run the local development server, or build the production distribution files using pnpm commands.

    # Install dependencies.
pnpm i

# Run local server to code
pnpm run dev

# Build the dist file for deployment.
pnpm run build

  5. Quick Start DeepDataSpace

    main

    Once installed, you can launch the DDS tool immediately using the --quickstart flag.

    Note: The first launch may take some time as the tool downloads extra dependencies to set up the runtime environment. Upon successful startup, the tool will host a web interface at http://127.0.0.1:8765 containing sample datasets.

    To exit the tool, press Ctrl+C in the terminal.

    dds --quickstart

  6. Set up the environment for deepdataspace-app

    main

    Before developing or building the frontend, ensure you have pnpm and a Node.js environment installed. You can install pnpm using the official shell script and use pnpm env to manage your Node.js version.

    # Install pnpm.
curl -fsSL https://get.pnpm.io/install.sh | sh -

# Install node.js env.
pnpm env use --global lts

  8. Develop dds-components

    main

    Use the following commands to manage the development lifecycle of the dds-components library:

    • Install dependencies: pnpm install
    • Run docs demo: pnpm run dev (develop the library using the documentation demo)
    • Build source code: pnpm run build
    • Build source code in watch mode: pnpm run build:watch
    • Build documentation: pnpm run docs:build
    • Project health check: pnpm run doctor (checks for potential problems in the project)
    pnpm install
pnpm run dev
pnpm run build
pnpm run build:watch
pnpm run docs:build
pnpm run doctor

  9. Install DeepDataSpace from Source

    main

    If you need to build from source, follow these steps to prepare the Node environment, compile the frontend, and install the Python package.

    1. Clone the repository.
    2. Install and use pnpm for the Node environment.
    3. Build the frontend application.
    4. Copy the compiled frontend files to the Python server directory.
    5. Install the Python dependencies and the package via setup.py.
    # clone the source code
git clone https://github.com/IDEA-Research/deepdataspace.git

# prepare the node environment
curl -fsSL https://get.pnpm.io/install.sh | sh -
pnpm env use --global lts

# compile frontend files
pnpm i
pnpm run build:app

# copy frontend files to python package dir
rm -rf deepdataspace/server/static/*
cp -R packages/app/dist/* deepdataspace/server/static/
cp deepdataspace/server/static/index.html deepdataspace/server/templates/

# install the package
python3 -m pip install pip --upgrade
python3 -m pip install -r requirements.txt
python3 setup.py install

  10. Install DeepDataSpace using Docker Compose

    main

    You can run DDS using Docker Compose by setting the dataset directory and the desired port via environment variables.

    Environment Variables

    • DDS_DATASET_DIR: The path to the directory where your datasets are stored.
    • DDS_PORT: The port you wish to use for visiting the DDS web interface (default is often 8765).
    # clone the source code
git clone https://github.com/IDEA-Research/deepdataspace.git

# prepare dataset directory
mkdir -p datasets
export DDS_DATASET_DIR=$PWD/datasets

# choose a visiting port for DDS
export DDS_PORT=8765

# start DDS with docker compose
cd deepdataspace
docker compose up

  11. Understand the PageState interface for URL persistence

    main

    The PageState interface defines the structure of the state that is saved in the URL to allow for deep-linking and state persistence when navigating datasets. It includes dataset identification, pagination settings, filter configurations, and comparison modes.

    Key properties include:

    • datasetId & datasetName: Identifiers for the current dataset.
    • queryMode: Determines if the dataset is accessed via pagination or random mode.
    • filterValues: An object containing active filters such as categoryId, displayAnnotationType, displayOptions, selectedLabelIds, and diffMode.
    • comparisons: Optional configuration for comparing labels, including orderBy, precision, and diffMode.
    • flagTools: Configuration for image flagging tools, specifically flagStatus.
    export interface PageState {
  datasetId: string;
  datasetName: string;
  queryMode: QueryMode;
  offset: number;
  page: number;
  pageSize: number;
  cloumnCount: number;
  previewIndex: number;
  isSingleAnnotation: boolean;
  filterValues: {
    categoryId?: string;
    displayAnnotationType: AnnotationType;
    displayOptions: DisplayOption[];
    selectedLabelIds: string[];
    diffMode: LabelDiffMode;
  };
  comparisons?: Comparisons;
  flagTools?: {
    flagStatus: IMG_FLAG;
  };
}

  12. Understand the DeepDataSpace application layout structure

    main

    The main application layout is a React component that provides the shell for the DeepDataSpace web interface. It manages several core responsibilities:

    • Navigation Sider: A collapsible sidebar containing the application logo, CustomMenu, and user actions. It supports a collapsed state with a width of 88px and an expanded state of 226px.
    • Route-based Visibility: The sidebar can be hidden on specific routes by setting the hideSider property in the route configuration.
    • Authentication Integration: It uses a user model to manage login status, displaying either a user profile dropdown (with a logout option) or a login button.
    • Global State & Loading: It wraps the application in a GlobalLoading component (from dds-components) that responds to the loading and loadingTip states from the global model.
    • Responsive Handling: On mobile devices, it renders a MobileAlert component instead of the standard desktop layout.
    • Localization: Integrates LangSelector and useLocale to support multi-language interfaces.

    When building custom pages, the content will be rendered within the <Outlet /> component provided by the layout.