rippkgs

repository·main·Indexed 41 days ago

https://github.com/replit/rippkgs

A high-performance search CLI utility for indexed nixpkgs expressions, providing a fast local alternative to nix-env -q or web-based search services. Includes rippkgs-index for creating SQLite databases from nixpkgs channels, directories, or registry JSON files, and a search CLI supporting exact and fuzzy matching with JSON or table output.

Tokens
3.5K
Snippets
14
Records
14
Agent score
51%

What's inside rippkgs

  1. Generate a rippkgs index

    main

    The rippkgs-index CLI is used to create a SQLite database of nixpkgs expressions. You can index the default nixpkgs channel, a specific directory, or a registry generated via a flake.

    Indexing the default nixpkgs channel

    rippkgs-index nixpkgs -o $XDG_DATA_HOME/rippkgs-index.sqlite

    Indexing a specific nixpkgs directory

    If you have a custom channel, pass the path explicitly:

    rippkgs-index nixpkgs -o $XDG_DATA_HOME/rippkgs-index.sqlite ~/.nix-defexpr/channels/my-very-special-nixpkgs-channel

    Indexing via a registry (to avoid recursive-nix problems)

    First, generate a registry JSON using nix eval, then pass it to rippkgs-index:

    nix eval -L .#lib.genRegistry --apply 'f: f (import <nixpkgs> { })' --impure --json > registry.json
    rippkgs-index registry -o rippkgs-index.sqlite registry.json
    rippkgs-index nixpkgs -o $XDG_DATA_HOME/rippkgs-index.sqlite
  2. Install rippkgs and rippkgs-index

    main

    You can install the rippkgs CLI and the rippkgs-index utility using nix profile install or by adding them to your NixOS configuration.

    Using nix profile

    nix profile install nixpkgs#rippkgs nixpkgs#rippkgs-index
    # Or via flake directly
    nix profile install github:replit/rippkgs/v1.1.0

    Using NixOS configuration

    If you are using a flake to configure your system, add the packages to your environment.systemPackages list:

    environment.systemPackages = [
      pkgs.rippkgs
      pkgs.rippkgs-index
      # If using as a flake input:
      inputs.rippkgs.packages.${system}.rippkgs
      inputs.rippkgs.packages.${system}.rippkgs-index
    ];
    nix profile install nixpkgs#rippkgs nixpkgs#rippkgs-index
  3. Search for packages with rippkgs

    main

    Once an index has been generated, use the rippkgs CLI to search for package expressions.

    Note: You must specify the index file created by rippkgs-index using the -i flag.

    # Basic search
    rippkgs -i path/to/index.sqlite rustc
    rippkgs -i nixpkgs.sqlite rustc
  4. Perform fuzzy search for packages

    main

    The search function allows for fuzzy matching of package names within a SQLite database. It calculates a score for each package based on how well its name matches the provided query string and returns a list of Package objects.

    Parameters

    • query_str: The string pattern to match against package names.
    • db: A reference to the rusqlite::Connection containing the package database.
    • num_results: The maximum number of results to return.
    • filter_built:
      • If true: The function filters the results to only include packages that are currently installed (i.e., their Nix store path exists at /nix/store/<out>). It also populates the present field on the Package struct.
      • If false: The function returns all matching packages regardless of whether they are installed, and the present field is not updated.
    pub fn search(
        query_str: &str,
        db: &Connection,
        num_results: u32,
        filter_built: bool,
    ) -> eyre::Result<Vec<Package>>
  5. Perform exact match search for a package

    main

    The search function allows for an exact match lookup of a package in the database using a specific attribute string.

    If a package is found, the function returns an Option<Package>. Note that the following cases will return None even if a row is found:

    • The package is stdenv (not installable).
    • The package is part of bootstrapping.
    • The package does not have an out path in its store_paths (meaning it is not installable).

    If the package is found and installable, the present field in the returned Package struct is automatically updated to reflect whether the package's out path currently exists in /nix/store/.

    // Signature of the search function
    pub fn search(query_str: &str, db: &Connection) -> eyre::Result<Option<Package>> {
        // ...
    }
  6. PackageInfo data structure schema

    main

    The PackageInfo struct defines the schema for indexed package information. When consuming data generated or used by rippkgs, the following fields are available (note that fields are serialized using camelCase in JSON):

    • pname: The package name.
    • version: The package version.
    • meta: A PackageMeta object containing metadata.
    • store_paths: A mapping of store path identifiers to their actual paths.
    • propagated_build_inputs: A list of propagated build inputs.
    • propagated_native_build_inputs: A list of propagated native build inputs.
    {
      "pname": "example-pkg",
      "version": "1.0.0",
      "meta": {
        "description": "A description",
        "available": true
      },
      "storePaths": {
        "out": "/nix/store/...-example-pkg"
      },
      "propagatedBuildInputs": ["dep1"],
      "propagatedNativeBuildInputs": ["dep2"]
    }
  7. Package data structure schema

    main

    The Package struct represents the metadata for a single package retrieved from the search index. It is serializable and deserializable via serde and contains the following fields:

    • attribute: The primary attribute identifier (String).
    • name: The package name (Option<String>).
    • version: The package version (Option<String>).
    • description: A short description of the package (Option<String>).
    • homepage: The package's homepage URL (Option<String>).
    • long_description: Detailed package documentation (Option<String>).
    • store_path: The path to the package in the store (Option<String>).
    • score: A numerical score used for ranking (Option<f64>).
    #[derive(Debug, Serialize, Deserialize)]
    pub struct Package {
        pub attribute: String,
        pub name: Option<String>,
        pub version: Option<String>,
        pub description: Option<String>,
        pub homepage: Option<String>,
        pub long_description: Option<String>,
        pub store_path: Option<String>,
        pub score: Option<f64>,
    }
  8. PackageMeta data structure schema

    main

    The PackageMeta struct contains additional metadata for a package. Fields are serialized using camelCase in JSON:

    • available: Boolean indicating if the package is available.
    • broken: Boolean indicating if the package is broken (defaults to false).
    • description: A short description of the package.
    • homepage: A single string or a list of strings representing the package homepage.
    • insecure: Boolean indicating if the package is insecure (defaults to false).
    • license: A JSON value representing the package license.
    • long_description: A detailed description of the package.
    • unfree: Boolean indicating if the package is unfree (defaults to false).
    • unsupported: Boolean indicating if the package is unsupported (defaults to false).
    {
      "available": true,
      "broken": false,
      "description": "Short desc",
      "homepage": "https://example.com",
      "insecure": false,
      "license": { "type": "MIT" },
      "longDescription": "Longer desc",
      "unfree": false,
      "unsupported": false
    }
  9. Generate an index from a registry JSON file

    main

    Use the registry subcommand to import a registry that was previously generated (e.g., via genRegistry from the flake library) and convert it into a SQLite index.

    Arguments:

    • registry: The path to the registry JSON file to import.

    Options:

    • -o, --output <PATH>: The location to write the saved index to. Defaults to rippkgs-index.sqlite.
    rippkgs-index registry <PATH_TO_REGISTRY> --output <OUTPUT_PATH>
  10. Use the rippkgs search CLI

    main

    The search command allows you to query the rippkgs index for packages. You can perform either exact or fuzzy searches, filter results by whether they are already built in the Nix store, and choose between a human-readable table or JSON output.

    Search Modes

    • Exact Search: Triggered by the --exact flag. It looks for an exact match of the query against attributes.
    • Fuzzy Search: The default mode. It uses fuzzy matching logic to find relevant packages.

    Filtering

    • Built Filter: When using fuzzy search, the --filter-built flag filters results to only include packages where the /nix/store path already exists.

    Output Formats

    • Table (Default): Prints a formatted table with columns for attribute, version, and description.
    • JSON: Use the --json flag to output the results as a JSON array of package objects.
    # Example: Fuzzy search for a package and output as JSON
    rippkgs search "python" --max-results 10 --json
    
    # Example: Exact attribute search
    rippkgs search "nixpkgs" --exact
    
    # Example: Search for built packages only
    rippkgs search "rust" --filter-built
  11. Generate an index from Nixpkgs

    main

    Use the nixpkgs subcommand to evaluate a Nixpkgs expression directly and build a SQLite index. This command uses nix eval under the hood.

    Arguments:

    • nixpkgs: (Optional) The location of Nixpkgs on-disk to index. If provided, it passes the -I nixpkgs=<PATH> flag to the nix command.

    Options:

    • -r, --save-registry <PATH>: (Optional) Save the evaluated registry as a JSON file before indexing.
    • -a, --nixpkgs-arg <STRING>: (Optional) A JSON string to pass as the config argument to the Nixpkgs import. Defaults to {}.
    • -o, --output <PATH>: The location to write the saved index to. Defaults to rippkgs-index.sqlite.
    rippkgs-index nixpkgs [-I <PATH>] [-r <SAVE_PATH>] [-a <ARG_JSON>] [-o <OUTPUT_PATH>]
  12. Generate an index using the `rippkgs-index` CLI

    main

    The rippkgs-index command is used to create a SQLite database index of packages. This index can then be used by the rippkgs CLI for searching and comparison. You can generate an index from two different sources: an existing registry JSON file or directly from a Nixpkgs expression.

    By default, the index is saved to rippkgs-index.sqlite in the current directory.

    # Example: Generate index from a Nixpkgs expression
    rippkgs-index nixpkgs --output my-index.sqlite
    
    # Example: Generate index from a pre-generated registry JSON
    rippkgs-index registry my-registry.json --output my-index.sqlite