reedsolo

The algorithm's capacity is constrained by the Galois Field exponent (c_exp). The maximum message length (including ECC symbols) and the maximum value of any single symbol is (2^c_exp) - 1.

• Default (GF(2^8)): Symbols are limited to values between 0 and 255. This is ideal for binary streams. The maximum total length (message + ECC) is 255.
• Chunking: If your message exceeds the length limit of the chosen field, the RSCodec class automatically applies chunking by splitting the message into smaller chunks and encoding/decoding them separately. This is transparent to the user.
• Performance Note: Using higher fields (larger c_exp) will slow down the algorithm because it cannot use the optimized bytearray structure and must use array.array('i', ...) instead. Additionally, Reed-Solomon complexity is quadratic, so longer messages increase processing time quadratically.

For significantly faster encoding (up to 14.3 MB/s on specific hardware), you can use the Cython extension creedsolo.

Requirements:

• pip install cython==3.0.0b2
• A C++ compiler (e.g., Microsoft Visual C++ 14.x for Windows).

Setup Steps:

1. Navigate to the directory containing creedsolo.pyx.
2. Run the following command to compile the extension in-place:

   python setup.py build_ext --inplace --cythonize

Usage Differences:

• Import: Use from creedsolo import RSCodec instead of from reedsolo import RSCodec.
• Data Type: You must feed bytearray() objects to the RSCodec object. The Cython implementation is optimized specifically for bytearray and will not work with other types.
• Limitation: The Cython implementation only supports Galois Fields up to 8 (values up to 255) because it relies on bytearray.

# After compiling creedsolo.pyx
from creedsolo import RSCodec

codec = RSCodec(nsym=10)
# Use bytearray exclusively for speed
data = bytearray("Hello World", "UTF-8")
encoded = codec.encode(data)

The RSCodec class provides a high-level API for Reed-Solomon error correction. You can initialize it with the number of error correction symbols (ECC) you want to use.

Encoding

RSCodec.encode() accepts lists of numbers, bytearray objects, or byte strings.

• Recommendation: Use bytearray for performance and predictable sizing, especially if you need to store or send the data in a fixed-size field.
• Transparent Chunking: The codec automatically handles strings longer than the Galois field size using transparent chunking.

Decoding (Repairing)

RSCodec.decode() repairs corrupted data.

• If the number of errors exceeds the codec's capacity, it raises a reedsolo.ReedSolomonError.
• You can provide erase_pos (a list of indices) to specify known error locations. Providing erasures allows the codec to correct up to twice as many errors compared to unknown error locations.

Return Values of decode()

For versions 1.0 and later, decode() returns a tuple of three values:

1. decoded_msg: The corrected message.
2. decoded_msgecc: The corrected message combined with the corrected ECC symbols.
3. errata_pos: A bytearray containing the positions of errors/erasures. Convert this to a list to get integer indices.

from reedsolo import RSCodec, ReedSolomonError

# Initialize with 10 ECC symbols
rsc = RSCodec(10)

# Encoding
encoded = rsc.encode(b'hello world')

# Decoding
try:
    decoded_msg, decoded_msgecc, errata_pos = rsc.decode(encoded)
    print(list(errata_pos))  # Get error positions as integer indices
except ReedSolomonError:
    print("Too many errors to correct")

Upgrading from v1.x to v2.x involves several changes to build requirements, packaging, and API usage:

Build Requirements

• Cython: Requires Cython>=v3.0.0b2 to cythonize creedsolo.pyx.
• Packaging: The project now uses a src-layout and is PEP 517 compliant, supporting build isolation by default.

API Changes

• Imports: The standard import API remains similar:
  • Pure Python: import reedsolo as rs; codec = rs.RSCodec(10)
  • Cython extension: import creedsolo as crs
• Cython C-Imports: If you use the fast Cython cimport system, you must change the path:
  • Old: cimport creedsolo as crs
  • New: cimport creedsolo.creedsolo as crs

Data Type Requirements (Breaking Change)

• creedsolo (Cython extension): You must always supply a bytearray object for both data and erasures_pos. Providing list or string objects will result in errors.
• reedsolo (Pure Python): Still supports list objects as it transparently converts them to bytearray internally (specifically within RSCodec).

Compatibility

• Python Versions: Support for Python 2.7 and Python <= 3.6 has been dropped. For these versions, use reedsolo v1.7.0.

# Standard import for pure python
import reedsolo as rs
codec = rs.RSCodec(10)

# Standard import for cython extension
import creedsolo as crs

# New cimport path for Cython users
cimport creedsolo.creedsolo as crs

# CRITICAL: When using creedsolo, always use bytearray
data = bytearray([1, 2, 3, 4])
# erasures_pos = bytearray([0, 2])

By default, pip installs the pure-python reedsolo module. To build the speed-optimized Cython/C extension (creedsolo), you must force pip to use the source distribution (sdist) instead of wheels and pass the --cythonize configuration setting. This requires cython>=3.0.0b2 and a C compiler.

If you encounter installation issues, use the --no-binary="reedsolo" and --no-cache flags to ensure a fresh build from source.

# Compile from the latest stable release with Cythonization
pip install --upgrade reedsolo --no-binary "reedsolo" --no-cache --config-setting="--build-option=--cythonize" --use-pep517 --isolated --verbose

# Compile from the latest development release with Cythonization
pip install --upgrade reedsolo --no-binary "reedsolo" --no-cache --config-setting="--build-option=--cythonize" --use-pep517 --isolated --pre --verbose

# Compile from cutting-edge code with Cythonization
pip install --upgrade "reedsolo @ git+https://github.com/tomerfiliba-org/reedsolomon" --no-binary "reedsolo" --no-cache --config-setting="--build-option=--cythonize" --use-pep517 --isolated --pre --verbose