DOCSEARCHbyAlgolia
Submit Docs

askalono

repository·master·Indexed 42 days ago

https://github.com/jpeddicord/askalono

A fast and accurate library and CLI tool for detecting license texts by matching input text against a database of known licenses using Sørensen–Dice scoring. It features a Rust library, a CLI for identifying single files or crawling directories, and an experimental WASM build. Version 0.5.0 supports SPDX license data loading, JSON/text output formats, and optimized analysis for license text embedded within source code.

Tokens
7.2K
Snippets
27
Records
37
Agent score
54%

What's inside askalono

  1. How askalono matches license text

    master

    askalono uses Sørensen–Dice scoring to match input text against a database of known licenses. The process follows these steps:

    1. Normalization: Text is normalized (Unicode, whitespace, quoting styles) and lines that change frequently (like variable Copyright lines) are removed.
    2. Tokenization: Normalized text is converted into a set of bigrams (word pairs).
    3. Parallel Comparison: The bigram set is compared in parallel against the known license database.
    4. Scoring: The match with the highest Sørensen–Dice coefficient is returned.

    The license database is stored in a MessagePack'd and zstd-compressed cache file to optimize startup speed.

  2. Build and run the askalono WASM demo

    master

    To run the WASM demo, you must first build the library. Then, navigate to the demo directory to build the frontend or start a development server.

    Production Build:

    1. Build the library.
    2. cd demo
    3. npm run build (outputs to dist)

    Development Server:

    1. Build the library.
    2. cd demo
    3. npm start (runs webpack-dev-server)
    # Build library first
wasm-pack build --out-name askalono

# Then for demo
cd demo
npm run build
# OR
npm start

  5. Use the Store struct to manage license collections

    master

    The Store struct is the primary entry point for matching text against a database of known licenses. It acts as a collection of LicenseEntry objects. You can create an empty store using Store::new() or, more commonly, load a pre-populated cache from disk using Store::from_cache (though the implementation of from_cache is not in this file, it is the recommended way to initialize a store for analysis). Once a store is populated, you can use it to analyze text to find the closest license match.

    use askalono::{Store, TextData};
use std::fs::File;

// Loading a store from a cache file
let store = Store::from_cache(File::open("askalono-cache.bin.zstd")?)?;

// Analyzing text against the store
let result = store.analyze(&TextData::from("what's this"));

  6. Understand ScanResult and ContainedResult

    master

    When calling ScanStrategy.scan(), you receive a ScanResult which describes the findings:

    • score: The confidence of the overall match (0.0 to 1.0).
    • license: An Option<IdentifiedLicense> representing the primary license identified for the entire text. If the score is below the confidence_threshold, this is None.
    • containing: A Vec<ContainedResult> containing licenses discovered inside the text. This field is only populated if .optimize(true) was used.

    If a ContainedResult is present, it includes:

    • score: The confidence of that specific sub-match.
    • license: The IdentifiedLicense found.
    • line_range: A (usize, usize) tuple representing the 0-indexed (inclusive, exclusive) line numbers where the license was located.

  7. Compare and locate licenses using TextData

    master

    The TextData struct is the core component for license matching. It handles text normalization, n-gram generation, and scoring. It can be used to calculate similarity scores between two texts or to locate the specific line range where a license exists within a larger body of text.

    Basic Scoring

    Use match_score to get a similarity value (0.0 to 1.0) between two TextData instances. Pre-processing automatically strips out identifiable copyright statements to improve matching accuracy.

    use askalono::TextData;

let license = TextData::from("My First License");
let sample = TextData::from("copyright 20xx me irl\n\n //  my   first license");
assert_eq!(sample.match_score(&license), 1.0);

    Locating License Bounds

    If you have a high match score but need to know exactly where the license is located in a file, use optimize_bounds. This returns a new TextData instance with a restricted lines_view and the calculated score.

    # use std::error::Error;
# use askalono::TextData;
# fn main() -> Result<(), Box<dyn Error>> {
let license = TextData::from("My First License");
let sample = TextData::from("copyright 20xx me irl\n// My First License\nfn hello() {\n ...");
let (optimized, score) = sample.optimize_bounds(&license);

// Check the line range (start, end) where the match was found
assert_eq!((1, 2), optimized.lines_view());
assert!(score > 0.99f32);
# Ok(())
# }
    use askalono::TextData;

let license = TextData::from("My First License");
let sample = TextData::from("copyright 20xx me irl\n\n //  my   first license");
assert_eq!(sample.match_score(&license), 1.0);

  8. Choose between Elimination and TopDown ScanModes

    master

    The ScanMode determines how the scanner traverses text to find licenses:

    • ScanMode::Elimination: A fast, general-purpose strategy. It iteratively locates the highest license match in a file, then the next, until no more strong matches are found.
    • ScanMode::TopDown: Intended for attribution documents or files containing multiple licenses and little else. It is more accurate than Elimination but significantly slower. It uses a step_size to scan through the text.

    Use Elimination for most general use cases and TopDown when high accuracy is required for files specifically structured as license collections.

    // Example of setting the mode
let strategy = ScanStrategy::new(&store).mode(ScanMode::TopDown).step_size(1);

  9. Understand the CLI output data formats

    master

    The askalono CLI produces results in two primary formats: Text and JSON.

    • Text format: Follows Unix conventions. Successful license identifications are printed to stdout, while errors are printed to stderr. The output can optionally include the file path.
    • JSON format: All results (both successes and errors) are printed to stdout to ensure consistent parsing for downstream tools.

    When a file is processed, the result is encapsulated in a FileResult structure which contains either the identification data or an error message.

  10. Use askalono as a Rust library

    master
    For programmatic usage, askalono exposes the Store and LicenseContent types. For implementation details and usage patterns, refer to the examples/basic.rs file in the repository or the official documentation on docs.rs.

  12. Optimize license identification for source code

    master

    If askalono fails to identify a file that contains license text embedded within source code (such as in a file header or footer), use the --optimize flag to perform a deeper analysis.

    askalono id --optimize <filename>