DOCSEARCHbyAlgolia
Submit Docs

Pursuit

repository·master·Indexed 41 days ago

https://github.com/purescript/pursuit

A web application that hosts API documentation for PureScript packages, enabling users to search for packages, modules, functions, and type signatures.

Tokens
2.7K
Snippets
7
Records
15
Agent score
51%

What's inside Pursuit

  1. Understand the Pursuit database structure

    master

    Pursuit's filesystem-based database consists of two main directories: cache/ and verified/.

    cache/ directory

    Mirrors the URL structure of the web application. It contains static files (like index.html and documentation files) that can be served directly without hitting the Yesod application.

    verified/ directory

    Stores uploaded packages. Each package has its own directory, containing a JSON file for every version. These JSON files contain a serialized Package GithubUser type.

    Example Structure:

    /
  cache/
    packages/
      purescript-prelude/
        0.1.0/
          index.html
          docs/
            Prelude/
              index.html
  verified/
    purescript-prelude/
      0.1.0.json
      0.1.1.json

  2. Understand Kind Signatures in Pursuit

    master

    Kind signatures describe the kinds of data, newtype, type synonym, and type class declarations. They can be:

    • Explicit: Defined by the developer using the :: syntax.
    • Inferred: Automatically determined by the compiler.

    When documentation comments exist for both the kind signature and the declaration itself, Pursuit merges them into a single block separated by a newline.

    Display Logic (Interesting vs. Uninteresting Kinds): To assist learners, Pursuit only displays "interesting" kind signatures by default.

    • Uninteresting Kinds: Signatures where all type parameters have the kind Type. These are hidden. Examples include Type, Constraint, or Type -> Type (where the parameter is just Type).
    • Interesting Kinds: Signatures that have at least one type parameter whose kind is something other than Type (e.g., a polymorphic kind k, or a higher-kinded type like Type -> Type). These are displayed.
    -- Explicit kind signature
data ExplicitFoo :: forall k. k -> Type
data ExplicitFoo a = ExplicitFoo

{- Kind signature inferred by the compiler for the below type:
kind signature inferred by the compiler: forall k. k -> (Type -> Type) -> Type -}
data ImplicitFoo a f = ImplicitFoo (f Int)

  3. How Anubis bot challenge proxy works

    master

    Anubis protects the backend from scraper fleets by requiring a JavaScript proof-of-work challenge for requests with browser-like user agents.

    • Admitted Traffic: Once a challenge is solved, the browser carries a cookie for one week. Non-browser clients (like curl, pursuit uploads, editor tooling, and GitHub's badge fetcher) and known-good search crawlers pass through without challenges.
    • Routing: Nginx routes only cache misses under /packages/ (and Anubis's own /.within.website/ endpoints) through Anubis. These are the expensive routes that trigger decoding. Cache hits, /search requests, and POST /packages uploads go directly to the backend.
    • Deployment: It is deployed via remote.sh using a pinned .deb package. Configuration is stored in /etc/anubis/pursuit.env and policies are managed via deploy/anubis.botPolicies.yaml.

  4. How to mark a package as deprecated

    master

    To signal that a package is no longer supported, you can mark it as deprecated. Once marked, the package's contents will be excluded from Pursuit search results (except for searches matching the exact package name).

    To deprecate a package:

    1. Add the keyword pursuit-deprecated to the keywords section of your bower.json file.
    2. Publish a new version of the package.

  5. Search for declarations, modules, and packages in Pursuit

    master

    Pursuit's search feature returns three distinct types of results based on your query:

    • Declarations: The most common result. Includes types, values (including functions), data constructors, type synonyms, type classes, and kinds. You can search by name (including prefixes) or by type signature.
    • Modules: The basic unit of PureScript code. Search by the full module name or a prefix (e.g., Data.String returns Data.String.Base64).
    • Packages: Collections of related modules. Search by the package name or a prefix. You do not need to include the purescript- prefix (e.g., lazy matches purescript-lazy).

    Search Behaviors:

    • Name Search: Matches if the query is a prefix of the name (e.g., con matches const).
    • Type Search: If you search for a type signature (e.g., a -> b -> a), Pursuit returns matching declarations. Type variable names are ignored (e.g., x -> y -> x also matches const).
    • Inference: Pursuit automatically infers whether you are searching by name or by type.

    Limitations:

    • Searching for visible type applications (e.g., forall @a. a -> a) is not supported.
    • Documentation comments are not currently included in the search index.

  6. Run the Pursuit web server

    master

    To start the web server locally on http://localhost:3000, use the stack exec command.

    Note: If you have not added any packages to the database (see Database), the server will start but you will not be able to browse any packages. The database is regenerated from the data source before the server begins listening.

    stack exec pursuit

  7. Build Pursuit for development or production

    master

    You can build Pursuit using stack. Use the standard build command for development, or pass a flag to build in production mode.

    • Development mode: Standard build.
    • Production mode: Build with the pursuit:-dev flag disabled.
    # Build in development mode
stack build

# Build in production mode
stack build --flag pursuit:-dev

  9. Server environment and maintenance requirements

    master

    The Pursuit server runs on a DigitalOcean droplet with specific environmental requirements:

    • OS Compatibility: The server runs Ubuntu 24.04. The CI runner in .github/workflows/ci.yml must build on the same Ubuntu release to ensure glibc requirements are satisfied. Always update both simultaneously.
    • Swap: 4GB /swapfile is used, with vm.swappiness=10 configured in /etc/sysctl.d/99-swappiness.conf.
    • Backups: A daily cron job runs deploy/backup.sh at 13:00 UTC. It copies the database (data/verified/) to /var/www/pursuit-backups and pushes it to the purescript/pursuit-backups GitHub repository. Note that the page cache (data/cache/) is not backed up.
    • TLS: Managed via certbot with a systemd timer. The webroot is located at /var/www/letsencrypt-webroot.
    • Cache Management: A weekly cron job (/etc/cron.weekly/pursuit-cache-eviction) deletes cache files not accessed in 90 days to prevent disk exhaustion.

  10. Iterate quickly using GHCi

    master

    For rapid development, you can use the GHCi REPL. After loading the development entry point, you can use the update command to refresh the web server with your changes.

    1. Start the REPL: stack ghci
    2. Load the development module: :l DevelMain
    3. Apply changes: update
    $ stack ghci
> :l DevelMain
> update

  11. Configure the Pursuit database directory

    master

    Pursuit uses the filesystem as a database. The location of the data directory is controlled by an environment variable.

    • Environment Variable: PURSUIT_APPROOT (Note: The README mentions PURSUIT_APPROOT as an example, but refers to src/Settings.hs for the full list).
    • Default: The default directory is data.

    To use sample packages, you can clone the pursuit-backups repository and copy the desired packages into the verified/ directory of your data folder.

  12. Emergency bypass for Anubis bot challenge proxy

    master

    If the Anubis proxy (which handles JavaScript proof-of-work challenges for browser-like user agents) is misbehaving or down, you can bypass it to route traffic directly to the backend.

    Note: This is a temporary fix. The next deployment will restore the standard routing configuration.

    1. Edit the Nginx configuration file at /etc/nginx/sites-enabled/pursuit.conf.
    2. Locate the @anubis location block.
    3. Change the proxy_pass directive to point directly to the backend at http://127.0.0.1:3000.
    4. Reload Nginx using systemctl reload nginx.
    # Step 1: Edit config
edit /etc/nginx/sites-enabled/pursuit.conf

# Step 2: Change proxy_pass to http://127.0.0.1:3000 inside the @anubis block

# Step 3: Reload
systemctl reload nginx