repository·master·Indexed 12 days ago
https://github.com/xampprocky/tokeiA high-performance code statistics tool that counts files, lines, code, comments, and blanks across over 150 programming languages. Available as a CLI tool and a Rust library, version 14.0.0 supports multiple output formats including JSON, YAML, and CBOR, and respects .gitignore and .ignore files by default.
Tokei is available as an alpine-based Docker image. You can build it using earthly +docker.
To analyze a specific path:
docker run --rm -v /path/to/analyze:/src tokei .To analyze the current directory (Linux):
docker run --rm -v $(pwd):/src tokei .Tokei uses libFuzzer via cargo-fuzz for fuzz testing. To begin fuzzing, you must first install the cargo-fuzz tool.
Install cargo-fuzz using cargo:
cargo install cargo-fuzzTokei respects .gitignore and .ignore files by default. To exclude additional files or directories, use the --exclude flag, which follows .gitignore syntax, or define exclusions in a .tokeignore file.
tokei ./foo --exclude *.rsBy default, Tokei sorts languages alphabetically. Use the --sort flag to sort by specific columns:
fileslinesblankscodecommentstokei ./foo --sort codeTokei can merge statistics from a previous run into the current run using the --input flag. The input can be a file path, a format name, or stdin.
tokei ./foo --input ./stats.jsonTo build and install Tokei from source, you must have the latest stable Rust compiler installed.
cargo install --git https://github.com/XAMPPRocky/tokei.git tokeilanguages.json. To add support for a new language, you must define it in that file. Detailed instructions for adding and testing a new language can be found in the project's CONTRIBUTING.md file.While the default output is human-readable for terminals, you can use the --output flag to export data in machine-readable formats.
Note: You must compile Tokei with the appropriate feature flags to enable serialization support.
cargo install tokei --features allcargo install tokei --features cborcargo install tokei --features yaml--output json--output yaml--output cbortokei ./foo --output jsonTo execute a fuzzing job, use the cargo +nightly fuzz run command followed by the target name. The process will continue running until interrupted with Ctrl+C.
--jobs flag to specify the number of parallel jobs.-max_len via the -- separator.parse_from_slice_panic and parse_from_slice_total use identical input formats, you can share a common corpus directory to improve efficiency.# Run a specific target
cargo +nightly fuzz run <target>
# Run with 6 cores
cargo +nightly fuzz run <target> --jobs=6
# Run with a maximum input length of 200
cargo +nightly fuzz run <target> -- -max_len=200
# Run both parse targets using a shared corpus
cargo +nightly fuzz run parse_from_slice_{panic,total} fuzz/corpus/commonTokei is used to report code statistics (files, lines, code, comments, and blanks) for specified directories.
tokei ./footokei ./foo ./bar ./bazTokei can be installed on various platforms using standard package managers.
apk add tokeipacman -S tokeisudo dnf install tokeisudo zypper install tokeisudo xbps-install tokeinix-env -i tokeipkg install tokeipkgin install tokeicargo install tokeiconda install -c conda-forge tokeibrew install tokeisudo port install tokeiwinget install XAMPPRocky.tokeiscoop install tokei# Example for Alpine Linux
apk add tokei
# Example for Homebrew
brew install tokei
# Example for Winget
winget install XAMPPRocky.tokeiTokei uses a SyntaxCounter to manage the state of a file during analysis. The counter operates in three primary modes to ensure accurate counting of code, comments, and strings:
plain mode: The default state. Blanks are counted as blanks, and the parser looks for triggers to enter string or comment modes.string mode: Triggered when the parser enters a string literal. While in this mode, comments are ignored and cannot trigger a transition to comment mode.comment mode: Triggered when the parser enters a comment block. While in this mode, strings are ignored and cannot trigger a transition to string mode.This state machine allows Tokei to correctly distinguish between actual code and text that looks like code but is actually inside a comment or a string literal.
/// - `plain` mode: This is the normal state, blanks are counted as blanks,
/// string literals can trigger `string` mode, and comments can trigger `comment` mode.
/// - `string` mode: This when the state machine is current inside a string
/// literal for a given language, comments cannot trigger `comment` mode while
/// in `string` mode.
/// - `comment` mode: This when the state machine is current inside a comment
/// for a given language, strings cannot trigger `string` mode while in
/// `comment` mode.