Monty Sensorimotor Learning System

Monty uses a custom set of benchmark experiments designed to track research progress and evaluate specific capabilities added to the system. These are not standard AI benchmarks but are tailored to evaluate sensorimotor learning and inference.

Key considerations for users:

• Configuration Location: Experiment configurations can be found in the benchmarks/ folder of the repository.
• Optimization Trade-off: Benchmark parameters are tuned for a balance between speed and accuracy to facilitate rapid iteration. For production use cases requiring higher precision or faster learning, you should adjust the specific learning module parameters.
• Runtime Reporting: Reported runtimes include total experiment overhead (e.g., Habitat initialization, logging, telemetry). The isolated Monty runtime (actual learning or inference time) is typically lower and corresponds to the RUNTIME column in Wandb logs.
• Application Suitability: Monty is a sensorimotor system designed for efficient learning through environmental interaction. It is not designed for large, static datasets. Before applying Monty to external benchmarks, review the application criteria and challenging preconceptions documentation.

The Monty documentation is organized into several key functional areas to help developers navigate the project:

• Overview: High-level vision, architecture (sensor modules, learning modules, cortical messaging), and benchmark experiments.
• How to Use Monty: Practical guides for developers, including getting started, running benchmarks, logging/analysis, customizing Monty (e.g., implementing actions), and a series of tutorials (pretraining, inference, robotics, etc.).
• How Monty Works: Deep dives into the implementation, including the environment-agent relationship, the Monty core, sensor modules (RGBD, Salience, 2D), learning modules (object models, evidence-based learning), and the motor system.
• Contributing: Guidelines for code contributions, pull request flows, documentation, and style guides.
• Community & Theory: Information on governance, licensing, and the underlying neuroscience theory (compositional models, object behaviors).

The Monty repository is organized into several key functional areas. For developers looking to build or extend the system, the most important directories are:

• src/tbp/monty/frameworks/models/: The primary location for modeling code, including Language Models (LMs), Sensorimotor (SM) models, motor systems, and the Cortical Message Protocol (CMP).
• src/tbp/monty/frameworks/: Contains core logic for actions, environment utilities, environments, experiments, loggers, and utilities.
• src/tbp/monty/simulators/: Contains simulator-specific implementations, such as habitat/ and mujoco/.
• src/tbp/monty/conf/: Centralized configuration management, including constants, environment interfaces, experiment settings, and logging configurations.
• src/tbp/monty/ (Root): Contains core runtime components like cmp.py (Cortical Message Protocol), context.py (Runtime context), hydra.py (Hydra configuration), and math.py.

.
|-- benchmarks/                 # CSVs with latest benchmark results
|-- docs/                       # Source files for documentation
|-- rfcs/                       # Merged RFCs
|-- src/tbp/monty/
|   |-- conf/                   # Monty configurations
|   |   |-- constants/          # Constants used in configurations
|   |   |-- env_interface/      # Configurations for environment interfaces
|   |   |-- environment/        # Configurations for environments
|   |   |-- experiment/         # Configurations for experiments
|   |   |-- logging/            # Configurations for logging
|   |   `-- monty/              # Configurations for Monty and its internals
|   |-- experiment/             # Intended future home of all experiment code
|   |-- frameworks/
|   |   |-- actions
|   |   |-- environment_utils
|   |   |-- environments        # Environments Monty can learn in
|   |   |-- experiments         # Current location of experiment classes
|   |   |-- loggers
|   |   |-- models              # LMs, SMs, motor system, & CMP
|   |   `-- utils
|   |-- simulators/
|   |   |-- habitat/            # Habitat simulator code
|   |   |   |-- actions.py
|   |   |   |-- actuator.py
|   |   |   |-- agents.py
|   |   |   |-- environment.py
|   |   |   |-- sensors.py
|   |   |   `-- simulator.py
|   |   `-- mujoco/             # MuJoCo simulator code
|   |-- cmp.py                  # Cortical Message Protocol code
|   |-- context.py              # Runtime context code
|   |-- hydra.py                # Hydra configuration code
|   |-- math.py                 # Math code
|   `-- path.py                 # Filesystem path code
|-- tests/
|-- tools/
`-- README.md

The Cortical Messaging Protocol (CMP) is a standardized communication format used by all components in the Monty system—including Learning Modules (LMs), Sensor Modules (SMs), and motor systems—to ensure interoperability. By adhering to CMP, components can be combined arbitrarily and scaled across different modalities and hierarchical levels.

Core Concept: Features at a Pose A CMP-compliant message fundamentally describes features at a pose.

• Pose: Represents a location in 3D (or 1D/2D) space relative to a common reference frame (e.g., the body or a landmark) and includes the feature's 3D rotation.
• Features: The specific attributes being sensed. These can be pose-dependent (location/rotation) or pose-independent (e.g., color, texture, temperature, or object ID).

Standard Message Package Components:

• Features and Poses: The primary payload.
• Sender ID: Identifies the source (e.g., a specific sensor module).
• Confidence Rating: A measure of the reliability of the sensed information.

Key Constraints:

• No Structural Sharing: LMs do not communicate their internal structural models or memory contents. They only communicate the results of their processing (features/poses).
• Lateral Votes: When LMs communicate via lateral votes, they send unions of possible poses and objects. They do not send the low-level features that were used to arrive at those hypotheses.
• Reference Frames: All communication occurs in a common reference frame (typically relative to the body). This allows LMs to calculate displacements between consecutive poses to detect objects independently of their current pose.

The Global Interval Timer is a proposed mechanism designed to provide Large Models (LMs) with information about the time elapsed since the last significant event. It functions like a stopwatch that broadcasts discrete temporal representations rather than a continuous signal.

Key Characteristics

• Discrete Representations: Instead of a continuous "time-since" value, the timer uses a sequence of discrete neural states (similar to time cells). As time passes, different neurons become active in a sequence (e.g., clockwise).
• Reset Mechanism: Significant events detected by LMs or SMs reset the timer to its initial state.
• Variable Speed: During inference, the cortical column can signal the timer to spin faster or slower. This allows the model to associate the same neural temporal input with different absolute time intervals.
• Input to L1: The ID of the currently active time neuron serves as the input to Layer 1 (L1) of the cortical column.
• Temporal Resolution: The system can be designed to have higher resolution (more distinct neural representations) for shorter durations compared to longer ones.

Use Cases

• Melody Learning: Each note acts as a significant event that resets the timer. Input features (notes) are associated with the specific L1 input received at that moment, representing the time passed since the previous note.
• Object Behavior Modeling: Allows representing features at different offsets after a significant event (e.g., a stapler reaching its lowest position). This enables the model to store discrete states in a sequence, where each state can store features at different locations and different temporal offsets.