InternNav

The InternData-N1 dataset follows a specific directory structure for training and evaluation.

Key components:

• vln_pe/raw_data/r2r/: Contains JSON files defining tasks, navigation goals, and dataset splits (e.g., train/, val_seen/, val_unseen/).
• vln_pe/traj_data/: Contains training sample data (trajectories) for different scene types like interiornav/ or r2r/.
• scene_data/mp3d_pe/: Contains the underlying scene assets organized by ID.
• Embodiments/: Contains embodiment-specific data.

data/
├── Embodiments/
├── scene_data/
│   └── mp3d_pe/
│       ├──17DRP5sb8fy/
│       ├── 1LXtFkjw3qL/
│       └── ...
└── vln_pe/
    ├── raw_data/                       # JSON files defining tasks, navigation goals, and dataset splits
    │   └── r2r/
    │       ├── train/
    │       ├── val_seen/
    │       │   └── val_seen.json.gz
    │       └── val_unseen/
    └── traj_data/                      # training sample data for two types of scenes
        ├── interiornav/
        │   └── kujiale_xxxx.tar.gz
        └── r2r/
            └── trajectory_0/
                ├── data/
                ├── meta/
                └── videos/

The DialogAgent (defined in internnav/agent/dialog_agent.py) is driven by the evaluator and uses a Hugging Face Qwen2.5-VL model to process observations and generate actions.

Core Responsibilities

• Sensor Handling: Uses convert_input to filter depth via Habitat's filter_depth and maintains RGB/depth history.
• Prompt Construction: The inference method builds chat-style prompts combining the current instruction, historical observations (as <image> tokens), and prior NPC exchanges.
• Action Decoding: The convert_output method parses the model's language output to identify:
  • <talk>: Triggers a dialog request to the NPC.
  • look down: Triggers a look-down action.
  • Pixel Goal Coordinates: Converts coordinates to world GPS points, validates navigability, and uses Habitat's ShortestPathFollower for motion.
• Step Loop: The step method manages the alternation between model inference and executing queued navigation actions.

InternNav uses a modular architecture consisting of Models, Agents, and Trainers. To add a new navigation algorithm, you must implement these three components.

1. Custom Model

The Model implements the neural network and inference logic. It expects an observation (obs) from the ego-centric camera and returns an action.

Observation Format (obs):

obs = [{
    'globalgps': [X, Y, Z],              # robot location
    'globalrotation': [X, Y, Z, W],      # robot orientation in quaternion
    'rgb': np.array(256, 256, 3),        # rgb camera image
    'depth': np.array(256, 256, 1)       # depth image
}]

Action Format (action): Returns a List[int] representing actions for each environment:

• 0: stop
• 1: move forward
• 2: turn left
• 3: turn right

Configuration: Define a Config class that inherits from PretrainedConfig (see CMAModelConfig in internnav/model/cma/cma_policy.py for reference).

Registration: Register the model in internnav/model/__init__.py using get_policy and get_config.

2. Custom Agent

The Agent wraps the Model and handles environment interaction and preprocessing. Inherit from Agent and implement:

• reset(): Resets internal state (e.g., RNN states).
• inference(obs): Performs preprocessing and calls the model.
• step(obs): The external interface that calls inference.

3. Custom Trainer

The Trainer manages the training loop. Inherit from Base Trainer and implement:

• train_epoch(): Handles batch iteration, forward pass, and parameter updates.
• eval_epoch(): Evaluates on the validation set.
• save_checkpoint() / load_checkpoint(): Manages model weights and state.

To participate in the IROS challenge, you must package your model and code into a Docker image and submit a JSON file to Eval.AI.

1. Build and Test your Docker Image

Ensure your trained weights and code are located at /root/InternNav inside the image. Use the following command to build:

docker build -t my-internnav-custom:v1 .

To test your image locally using a mini split:

docker run --name internnav-test -it --gpus all --network host \
  -e "ACCEPT_EULA=Y" \
  -e "PRIVACY_CONSENT=Y" \
  -e "DISPLAY=${DISPLAY}" \
  --entrypoint /bin/bash \
  -w /root/InternNav \
  -v /tmp/.X11-unix/:/tmp/.X11-unix \
  -v ${PWD}/data:/root/InternNav/data \
  -v ${PWD}/interiornav_data:/root/InternNav/interiornav_data \
  your-registry/internnav-custom:v1 \
  -c "challenge/start_eval_iros.sh --config scripts/eval/configs/challenge_cfg.py --split mini; exec /bin/bash"

2. Submit to Eval.AI

Push your image to a public registry (e.g., Aliyun or Quay) and create a JSON file with the following structure:

{
    "url": "your-registry/internnav-custom:v1",
    "team": {
        "name": "your-team-name",
        "members": [
            {
                "name": "John Doe",
                "affiliation": "University of Example",
                "email": "john.doe@example.com",
                "leader": true
            }
        ]
    }
}

Upload this JSON file to Eval.AI using the 'Upload file' submission type.