DOCSEARCHbyAlgolia
Submit Docs

UPM (Universal Package Manager)

repository·main·Indexed 22 days ago

https://github.com/replit/upm

A tool providing a consistent, unified CLI interface for managing packages across multiple programming languages by wrapping existing language-specific package managers such as pip, npm, yarn, poetry, and others. UPM abstracts language-specific CLI quirks and provides additional features like dependency guessing, machine-parseable specfile/lockfile listings, and a unified set of commands for adding, removing, locking, and installing dependencies.

Tokens
5.3K
Snippets
19
Records
35
Agent score
72%

What's inside upm

  1. Overview of UPM (Universal Package Manager)

    main

    UPM is a Universal Package Manager that provides a unified interface for managing packages across multiple programming languages. Instead of implementing package management itself, UPM acts as a wrapper that runs existing language-specific package managers (like pip, npm, yarn, poetry, etc.).

    Key benefits include:

    • Unified Interface: Use the same commands regardless of the language.
    • Abstraction of Quirks: Eliminates the need to remember language-specific CLI quirks, argument ordering, or incompatible behaviors between different managers (e.g., NPM vs Yarn).
    • Extra Features: Provides dependency guessing and machine-parseable specfile and lockfile listing.

  2. How to use gen_pypi_map to generate pypi_map.gen.go

    main

    The gen_pypi_map module is used to generate the pypi_map.gen.go file. This file provides three critical mappings for the Python backend:

    1. module -> package mapping: Used for guessing which package provides a specific module.
    2. package -> modules mapping: Used to detect if a module is already installed.
    3. package -> download count mapping: Used for statistics and sorting search results.

    To run this program correctly, ensure your current working directory (CWD) is set to internal/backends/python.

  3. Supported languages and command capabilities in UPM

    main

    UPM supports various languages and package managers, categorized by the types of operations they support:

    • Core operations: upm add, upm remove, upm lock, upm install, upm list
    • Index operations: upm search, upm info
    • Guess operations: upm guess
    Language/ManagerCoreIndexGuess
    python-python3-uvyesyesyes
    python-python3-pipyesyesyes
    python-python3-poetryyesyesyes
    nodejs-yarnyesyesyes
    nodejs-pnpmyesyesyes
    nodejs-npmyesyesyes
    ruby-bundleryesyes
    elisp-caskyesyesyes
    dart-pub.devyesyes
    rlangyesyes
    javayesyes
    rustyesyes
    dotnetyesyes
    phpyesyes

  4. Use UPM for multiple languages

    main

    UPM provides a unified interface for different language ecosystems by using the -l (language) flag. This allows you to perform the same operations (like info) across different package managers.

    Supported language examples:

    • Node.js: upm -l nodejs info <package>
    • Ruby: upm -l ruby info <package>
    • Emacs Lisp: upm -l elisp info <package>
    # Get info for a Node.js package
$ upm -l nodejs info express

# Get info for a Ruby package
$ upm -l ruby info jekyll

  5. How UPM information flows and manages packages

    main

    UPM follows a unidirectional information flow for package management:

    add/remove $\rightarrow$ specfile $\rightarrow$ lockfile $\rightarrow$ installed packages

    When you run upm add or upm remove, UPM automatically performs the entire pipeline: it modifies the specfile, generates a new lockfile, and installs/uninstalls the necessary packages. While you cannot skip steps in the automatic flow, you can manually trigger later stages using upm lock and upm install.

    Language Detection

    UPM autodetects your project's language based on files in the current directory. You can override this using the -l or --lang flag. You can provide specific language strings (e.g., python3-poetry) or simpler identifiers (e.g., python, poetry). If a simple identifier is used, UPM will evaluate all matching languages and select the best fit. Use upm which-language -l <identifier> to test this logic.

    # Example of overriding language detection
$ upm add left-pad -l nodejs-npm

  6. Manage UPM caching and force updates

    main

    UPM uses a JSON cache in the .upm subdirectory to improve performance by skipping redundant lockfile generation, re-installation, or code analysis during upm guess.

    To bypass the cache and force an update, use the following flags:

    • For upm add and upm remove: use --force-lock.
    • For upm lock: use --force-lock.
    • For upm install: use --force.

    To completely reset the cache, delete the .upm directory in your project.

  7. Run UPM Python regression tests

    main

    Navigate to the regression_tests directory to execute the test suite using the test.py script.

    Test a specific package

    Pass the name of the Python package you wish to test as an argument:

    python3 test.py <package_name>

    Run full suite

    To run the regression tests against the top 10,000 packages, execute the script without arguments. Note that this process can take up to 30 minutes.

    python3 test.py

    Understanding test outputs

    • <pkg> ok: The output of the old version matches the new version (or a manual override was applied).
    • <pkg> failed: The output of the old version does not match the new version.
    • <pkg> added: The old version did not have an entry for this package, but the new version does.
    • <pkg> no-guess: The old version had no entry, and the new version could not find a guess using its heuristic algorithm.
    • <pkg> test-errored: The old version had no entry, and the new version encountered an error during the test.
    • <pkg> missing: Neither the old nor the new version has an entry for this package.
    python3 test.py flask

  8. Update PyPI download stats using BigQuery

    main

    Download counts are used for heuristics in the guessing algorithm and for sorting search results in UPM. These stats are stored in download_stats.json. To update this file, fetch data from the public PyPI BigQuery table using the bq command.

    Note: You can use any Replit GCP project name because the source table bigquery-public-data.pypi.file_downloads is public.

    go run ./gen_pypi_map bq -gcp <gcp-project-name>

  10. Test PyPI packages for module mappings

    main

    The test command verifies which modules are added when a package is installed. It compares the state of pkgutil before and after installation.

    Key Options:

    • -threshold <number>: Limits testing to the top N packages (default is 10,000). Use this to test a larger or smaller subset.
    • -package <name>: Tests a single specific package.
    • -force: Forces a re-test of a package even if its version has already been tested.
    • -cache <path>: Specifies the directory where test results are stored (default: ./cache/).

    Examples:

    Test the top 50,000 packages:

    go run ./gen_pypi_map/ test -threshold 50000

    Test a single package:

    go run ./gen_pypi_map test-one -package replit-object-storage

  11. Set up UPM Python regression tests

    main

    To run regression tests, you must prepare a local environment containing both the current version of upm and an older version renamed to upm-old. This allows the test suite to compare the guessing behavior of the new implementation against the legacy one.

    1. Install the legacy version (upm-old)

    From the top of the project directory, checkout the specific legacy commit, rebuild, and copy the binary:

    git checkout abe5cc5
rm internal/backends/python/pypi_map.gen.go
make upm
go install ./cmd/upm
cp $(which upm) $(dirname $(which upm))/upm-old

    2. Restore the current version

    Switch back to the active development branch and rebuild the current environment:

    git checkout th-python-guess-modules-2
make clean-gen
make upm
make install
    git checkout abe5cc5
rm internal/backends/python/pypi_map.gen.go
make upm
go install ./cmd/upm
cp $(which upm) $(dirname $(which upm))/upm-old

git checkout th-python-guess-modules-2
make clean-gen
make upm
make install