shlex Rust Crate

The shlex crate provides a shell-like lexical analyzer for splitting strings into tokens. It mimics the behavior of the Python shlex.split function with default settings that follow the POSIX shell specification.

Key characteristics:

• POSIX Compliance: Mimics POSIX shell splitting behavior.
• Performance: Optimized by working directly on bytes, making it oblivious to UTF-8 high bytes.
• Input Types: Supports both standard Rust strings and byte strings (via the bytes module).
• Differences from Python: Unlike the Python shlex module, this implementation does not support customization to maintain speed, and it does not treat \r specially, which is intended to be more compliant.

The byte \xa0 (U+00A0 NO-BREAK SPACE) can be incorrectly treated as a shell word separator by Bash on macOS when using the default UTF-8 locale and providing invalid UTF-8 input.

To mitigate this risk and similar issues with other non-ASCII characters, rust-shlex always uses quotes for any argument containing non-ASCII bytes. While this may be less friendly to non-English users, it ensures maximum compatibility and prevents word-splitting vulnerabilities across different systems and locales.

$ echo -e 'ls a\xa0b' | bash
ls: a: No such file or directory
ls: b: No such file or directory

The characters ! (history expansion) and ^ (Quick Substitution) have special meanings in interactive shell contexts and can be expanded even inside double-quoted strings.

To prevent unintended expansion, rust-shlex uses the following strategies:

• For !: The crate uses single-quoted strings, as these characters do not expand within them.
• For ^: Due to a Bash bug where ^ can be partially expanded even in single-quoted strings, the crate forces ^ to appear immediately after an opening single quote. For example, the string "^ is quoted as "'"'"^"' instead of "'"^"'.

This ensures that these characters are treated as literals and do not trigger shell history features.

Passing extremely long strings to a shell can result in several failures:

• Argument list too long: The shell may fail to execute the command (e.g., bash: /bin/echo: Argument list too long).
• Line length limits: Some shells (like Busybox Ash) using a pty may impose a line length limit and ignore all input past that limit.
• Kernel buffer drops: When using a pty in cooked mode, if the input fills the kernel's internal buffer, the kernel may drop bytes instead of blocking, leading to loss of input data.

Future versions of shlex may include a Quoter option to check string length.

By default, shlex is configured with the std feature enabled. If you are working in a no_std environment, you can disable the std feature. Note that even in no_std mode, the crate requires the alloc crate and a global allocator to be available.

[dependencies]
shlex = { version = "1.3.0", default-features = false }

Versions of rust-shlex prior to 1.3.0 failed to properly quote the following characters:

• { (left brace)
• } (right brace)
• \xa0 (non-breaking space)

Ensure you are using version 1.3.0 or later to mitigate these issues (see GHSA-r7qv-8r2h-pg27 / RUSTSEC-2024-0006).

In Unix-like systems, nul bytes (value 0) cannot be used in command arguments, environment variables, or filenames because system calls rely on nul-terminated C strings. Most shells also cannot handle them.

By default, shlex convenience functions return QuoteError::Nul when encountering a nul byte. If you need to allow them (for example, when tunneling commands over SSH), you must use Quoter::allow_nul or bytes::Quoter::allow_nul to prevent an error.

std::process::Command::new("ssh")
    .arg("myhost")
    .arg("--")
    .arg(Quoter::new().allow_nul(true).join(my_cmd_args).unwrap())

If you want to consume the entire byte string at once, use the split function. It returns Some(Vec<Vec<u8>>) containing the words, or None if the input was syntactically erroneous.

use shlex::bytes::split;

let input = b"word1 'word 2' word3";
if let Some(words) = split(input) {
    // words is Vec<Vec<u8>>
    println!("{:?}", words);
} else {
    println!("Parsing error");
}

For more control, use the Shlex struct to create an iterator over the words in a string. Shlex implements Iterator<Item = String>. This is useful for processing large inputs or handling words one by one.

use shlex::Shlex;

let mut shlex = Shlex::new("foo bar 'baz'");
for word in shlex {
    println!("{}", word);
}