bstr

bstr provides extension traits for &[u8] and Vec<u8> that allow them to be treated as byte strings.

Unlike the standard library's String and str types, byte strings are not required to be valid UTF-8. They may be fully or partially valid UTF-8. Use byte strings when it is inconvenient or incorrect to require strict UTF-8 validation, such as when processing raw data streams that might contain invalid sequences but still require string-like operations (e.g., searching, splitting, or case conversion).

To add bstr to your Rust project, use cargo add:

cargo add bstr

The BufReadExt trait extends std::io::BufRead to provide convenient and efficient APIs for working with lines and records as byte strings (&[u8] or Vec<u8>). It is useful when you need to process text-like data without the overhead or UTF-8 requirements of standard string types.

use bstr::io::BufReadExt;

// Any type implementing std::io::BufRead now has access to BufReadExt methods
let mut reader: Box<dyn std::io::BufRead> = ...;
reader.byte_lines();

The bstr crate extends standard byte collections with string-oriented methods. It provides two primary traits:

• ByteSlice: Extends the [u8] type with methods for searching, splitting, and iterating over bytes as if they were a string.
• ByteVec: Extends the Vec<u8> type with similar string-oriented capabilities.

These traits allow you to perform operations like substring searching, trimming, and case conversion directly on raw byte slices and vectors, even if they contain invalid UTF-8.

use bstr::ByteSlice;

let s = b"foo bar foo foo quux foo";
for start in s.find_iter("foo") {
    // ...
}

In Unicode, a grapheme cluster is an approximation of a single user-visible character. A single grapheme can be composed of multiple codepoints (e.g., a base character followed by combining marks, or emoji sequences).

When performing text processing tasks that are oriented toward what a user sees (like cursor movement, backspacing, or character counting), you should prefer using graphemes (via bstr::ByteSlice::graphemes) over standard Rust Chars. The Chars iterator yields one codepoint at a time, which may split a single user-visible character into multiple pieces.

For storing or passing byte strings, the crate provides two concrete types that deref to standard byte collections:

• BStr: A byte string slice, analogous to str. It provides a convenient std::fmt::Debug implementation that prints bytes as a string (using escape sequences for invalid UTF-8) rather than a sequence of integers.
• BString: An owned, growable byte string buffer, analogous to String. (Requires alloc feature).

Conversions between these types and [u8]/Vec<u8> are zero-cost.

use bstr::ByteSlice;

let mut bytes = Vec::from("hello β");
bytes[0] = b'\xFF';

// Prints as "\xFFello β" instead of [255, 101, 108, 108, 111, ...]
println!("{:?}", bytes.as_bstr());

The ByteVec trait extends Vec<u8> with string-oriented methods, allowing you to treat byte vectors as if they were strings. This includes methods for conversion to BString, String, OsString, or PathBuf, as well as character-based manipulation.

use bstr::ByteVec;

let mut v = vec![b'a', b'b', b'c'];
// Use ByteVec methods on Vec<u8>
v.push_str(b"def");

Byte strings in bstr are conventionally UTF-8, meaning they may contain invalid UTF-8 sequences.

When performing Unicode-aware operations (like iterating over codepoints or graphemes), bstr uses the 'substitution of maximal subparts' strategy. It replaces invalid byte sequences with the Unicode replacement codepoint U+FFFD (�).

To access the original raw bytes during iteration, use the char_indices() method, which returns the byte offsets of the original data.

use bstr::ByteSlice;

// Example of substitution
let bs = b"a\xFF\xFFz";
let chars: Vec<char> = bs.chars().collect();
assert_eq!(vec!['a', '\u{FFFD}', '\u{FFFD}', 'z'], chars);

// Example of accessing original raw bytes via indices
let bs = b"a\xE2\x98z";
let chars: Vec<&[u8]> = bs.char_indices()
    .map(|(s, e, _)| &bs[s..e])
    .collect();
assert_eq!(chars, vec![B("a"), B(b"\xE2\x98"), B("z")]);

You can use bstr::io::BufReadExt to efficiently iterate over lines in a reader (like stdin) using for_byte_line_with_terminator. This is useful for processing data line-by-line without requiring the entire stream to be valid UTF-8.

use std::{error::Error, io::{self, Write}};
use bstr::{ByteSlice, io::BufReadExt};

fn main() -> Result<(), Box<dyn Error>> {
    let stdin = io::stdin();
    let mut stdout = io::BufWriter::new(io::stdout());

    stdin.lock().for_byte_line_with_terminator(|line| {
        if line.contains_str("Dimension") {
            stdout.write_all(line)?;
        }
        Ok(true)
    })?;
    Ok(())
}