DOCSEARCHbyAlgolia
Submit Docs

NuPIC Core

repository·master·Indexed 42 days ago

https://github.com/numenta/nupic.core-legacy

C++ source code for the Numenta Platform for Intelligent Computing, focusing on Hierarchical Temporal Memory (HTM) algorithms. It provides standalone C++ libraries and Python SWIG-based bindings for the Algorithms and Network API. Includes build instructions for Unix-like systems and Windows using MinGWPy GCC or MSVC 2015.

Tokens
11.4K
Snippets
33
Records
71
Agent score
55%

What's inside NuPIC Core

  1. Overview of the APR external dependency

    master

    The nupic.core project uses the Apache Portable Runtime (APR) unix sources for both Unix-like (Linux, Mac OS X) and Windows builds. The external package in nupic.core wraps the APR build process within a CMake module to ensure compatibility across platforms.

    Key build characteristics:

    • Unix/Linux/Mac OS X: Uses standard configure-based builds.
    • Windows: Uses CMakeLists.txt provided by APR.

    Note that the APR sources are patched by the project to:

    1. Disable the libapr-1 shared library target on Windows to avoid unnecessary builds.
    2. Modify APR_VERSION_STRING_CSV in apr_version.csv to ensure compatibility with the mingwpy toolchain used for Windows builds.

  2. Fast C++ CSV Parser Features

    master

    The library provides several high-performance features for CSV processing:

    • Column Rearrangement: Automatically rearranges columns based on the header line.
    • Multi-threading: Overlaps Disk I/O and CSV-parsing using threads for improved efficiency.
    • Compile-time Optimization: Features like escaped string parsing can be enabled/disabled via templates to minimize performance overhead.
    • Large File Support: Capable of reading multi-GB files efficiently.
    • Customization: Supports custom column separators (e.g., Tab-separated values), quote-escaped strings, and automatic space trimming.
    • Compatibility: Works with Unix/Linux and Windows newlines; automatically ignores UTF-8 BOMs.
    • Error Handling: Provides exception classes with context; .what() returns user-friendly error messages.

  3. Overview of zlib integration in NuPIC

    master
    NuPIC uses a patched version of zlib-1.2.8 (sourced from http://www.zlib.net/). The zlib sources are modified to prevent the building of shared libraries (.dll/.so) and dependent executables. This modification is necessary to allow NuPIC to hide symbols within the nupic.bindings extension shared libraries, ensuring only functions with explicit visibility are exposed. Because the original zlib sources do not set explicit visibility attributes on API functions, they would otherwise fail to link when NuPIC is compiled with C_FLAGS that hide visibility by default. NuPIC only requires the static library from zlib.

  4. Parse customized types from CSV

    master
    To parse custom types, read the column as a char* instead of a std::string. The library handles column reordering and quote escaping, providing a pointer that points directly into the memory buffer. This avoids the memory copy overhead associated with std::string. You are responsible for the actual parsing of the string into your custom type.

  5. Understand the NuPIC Core repository structure

    master

    The NuPIC Core repository is divided into two primary components:

    1. Main C++ core library: Located in the src directory.
    2. Python SWIG-based bindings: Located in bindings/py. These provide access to the Algorithms and Network API.

    If you are developing new bindings, they should be placed in the bindings folder, organized by language name.

  6. Use NuPIC Math components for domain, indexing, and sets

    master

    The NuPIC math library provides fundamental abstractions for mathematical modeling, including:

    • nupic::Domain: Represents a mathematical domain.
    • nupic::Index: Provides indexing capabilities.
    • nupic::Set: Implements set operations.
    • nupic::NearestNeighbor: Provides nearest neighbor search functionality.

  7. Work with sparse matrices and tensors

    master

    NuPIC provides several specialized structures for handling sparse data efficiently:

    • nupic::SparseTensor: A sparse representation of a tensor.
    • nupic::SparseRLEMatrix: A sparse matrix using Run-Length Encoding (RLE).
    • nupic::SparseMatrixAlgorithms: A collection of algorithms designed for sparse matrices.
    • nupic::SparseMatrixConnections: Manages connections within sparse matrix structures.
    • nupic::SegmentMatrixAdapter: An adapter for segment-based matrix operations.

  8. Use CSVReader with template policies

    master

    The io::CSVReader is a highly configurable template class that uses policies to define parsing behavior. This allows for zero-overhead customization of how columns are trimmed, quoted, and how overflows are handled.

    Template Parameters

    • unsigned column_count: The number of columns to read (does not have to match the actual file column count).
    • class trim_policy: Defines characters to ignore at the start/end of columns. Default: trim_chars<' ', '\t'>.
    • class quote_policy: Defines escaping and the column separator.
      • no_quote_escape<sep>
      • double_quote_escape<sep, quote>
    • class overflow_policy: Defines behavior for integer overflows.
      • throw_on_overflow (default)
      • ignore_overflow
      • set_to_max_on_overflow
    • class comment_policy: Defines how to skip lines.
      • no_comment (default)
      • empty_line_comment
      • single_line_comment<com1, com2, ...>
      • single_and_empty_line_comment<com1, com2, ...>

    Examples

    // Reads 4 columns, trims spaces, and handles double-quoted strings
CSVReader<4, trim_chars<' '>, double_quote_escape<',','"'> > reader("file.csv");

// Reads 3 columns from a tab-separated file, ignores lines starting with #
CSVReader<3, trim_chars<' '>, no_quote_escape<'\t'>, single_line_comment<'#'> > reader("file.tsv");

  9. Define mathematical topology and neighborhoods

    master

    For spatial or structural relationships, NuPIC provides topology abstractions:

    • nupic::math::topology::Neighborhood: Defines a general neighborhood concept.
    • nupic::math::topology::WrappingNeighborhood: A specific implementation of a neighborhood that supports wrapping (often used for periodic boundary conditions).

  10. Understand UTF-8 support in the CSV parser

    master
    The library provides basic UTF-8 support by ensuring it does not break when passing UTF-8 strings through it. If you read a char*, you receive a pointer to the UTF-8 string, which you must decode manually. Note that the library's internal control characters—the separator, quoting, and commenting characters—must be ASCII characters.

  11. Configure Cap'n Proto for NuPIC Core

    master

    To use Cap'n Proto, download version 0.5.3 and extract it into %NUPIC_CORE%\build. Use the following CMake configuration within a Visual Studio Command Prompt to generate the necessary project files.

    cd %NUPIC_CORE%\build\capnproto-c++-win32-0.5.3\capnproto-c++-0.5.3
vcvarsall.bat
cmake -G "Visual Studio 14 2015 Win64" \
	-DCAPNP_LITE=1 \
	-DEXTERNAL_CAPNP=1 \
	-DCAPNP_INCLUDE_DIRS=.\src \
	-DCAPNP_LIB_KJ=.\lib \
	-DCAPNP_LIB_CAPNP=.\lib \
	-DCAPNP_EXECUTABLE="..\capnproto-tools-win32-0.5.3\capnp.exe" \
	-DCAPNPC_CXX_EXECUTABLE="..\capnproto-tools-win32-0.5.3\capnpc-c++.exe"