H-Net

The core logic resides in the hnet/ directory:

• hnet/models/: Contains the model definitions.
  • config_hnet.py: Defines the H-Net configuration.
  • hnet.py: The core sequence model with signature (B, L, D) -> (B, L, D).
  • mixer_seq.py: A wrapper that converts the H-Net sequence model into a language model.
• hnet/modules/: Contains architectural components.
  • dc.py: Implementation of the dynamic chunking mechanism.
  • isotropic.py: Non-hierarchical components.
• generate.py: Inference and text generation script.

To install H-Net, clone the repository and install the package in editable mode. Ensure you have PyTorch >= 2.5.1 installed.

It is strongly recommended to build the mamba_ssm package from the latest source to ensure compatibility.

git clone https://github.com/goombalab/hnet
cd hnet
pip install -e .

# Recommended: build mamba_ssm from source
git clone https://github.com/state-spaces/mamba
cd mamba
pip install .

Use the generate.py script to perform inference with pretrained checkpoints. You must provide the path to the model checkpoint and the corresponding configuration file.

python generate.py --model-path hnet_2stage_XL.pt --config-path configs/hnet_2stage_XL.json --max-tokens 1024 --temperature 1.0 --top-p 1.0

H-Net models are available on Hugging Face via cartesia-ai. The models are categorized by scale and training stage:

English Models

• hnet_1stage_L (Large scale, matched to GPT-3 Large)
• hnet_2stage_L (Large scale, matched to GPT-3 Large)
• hnet_1stage_XL (XL scale, matched to GPT-3 XL)
• hnet_2stage_XL (XL scale, matched to GPT-3 XL)

Specialized Models

• hnet_2stage_XL_chinese: Trained on FineWeb-Edu Chinese V2.1.
• hnet_2stage_XL_code: Trained on Pile Github.

The HNetForCausalLM class is the primary entrypoint for using H-Net models for causal language modeling tasks (such as text generation). It is imported from hnet.models.mixer_seq and is exposed at the package level.

from hnet import HNetForCausalLM
# Use HNetForCausalLM for causal language modeling tasks