DOCSEARCHbyAlgolia
Submit Docs

arXiv Search Service

repository·develop·Indexed 41 days ago

https://github.com/arxiv/arxiv-search

A search service for arXiv papers featuring a Flask web interface and APIs, utilizing Elasticsearch for indexing. It includes an indexing agent that consumes Kinesis streams to maintain the search index, and provides tools for index creation, population, auditing, and reindexing.

Tokens
21.3K
Snippets
49
Records
72
Agent score
51%

What's inside arXiv Search

  1. Understand the arXiv Search Interface requirements and constraints

    develop

    The arXiv search application is designed to replace the legacy search system (Milestone H1). It provides simple search (title, author, abstract), advanced search (boolean combinations, filtering by primary classification and submission date), and author name search.

    Key Functional Capabilities

    • Simple Search: Search by title, author, or abstract via the site header.
    • Advanced Search: Boolean queries on title, author, and abstract. Results can be filtered by primary classification and submission date (prior year, specific year, or date range).
    • Author Search: Search for papers by author name from abstract pages or search results.
    • Result View: Supports pagination and ordering by publication date or relevance.

    Technical Constraints

    • Stack: Implemented in Python/Flask; deployable behind Apache as a Python/WSGI application.
    • State: The application must be stateless.
    • Search Engine: Must connect to an arbitrary Elasticsearch cluster, configurable via settings.
    • Data Ingestion: New content notifications are delivered via a Kinesis notification broker.

  2. How cross-list categories are handled in search

    develop

    Cross-list (secondary) categories are treated similarly to primary categories to ensure users can search by them:

    • All-fields search: Includes cross-list/secondary categories.
    • Advanced search interface: Includes a dedicated cross-list field and includes cross-list classification in the classification filter (with an option to exclude them).

  3. Note UI constraints and design decisions

    develop

    The current interface design is constrained by a goal to minimize JavaScript and AJAX calls. This results in several specific behaviors:

    • Sorting: A 'Go' button is required to trigger results sorting.
    • Field Toggles: Related fields (such as dates or the Physics subject dropdown) require manual checkbox toggles.
    • Error Messages: Error messages remain persistent on the screen until the page is reloaded, even if the user has corrected the input.

  4. Structure of the Classic arXiv API Feed

    develop

    The Classic arXiv API returns results in an Atom feed format. The feed is composed of two main types of elements: feed elements (metadata about the query itself) and entry elements (metadata for each individual article returned).

    ### Feed Elements
- `<title>`: The title of the feed containing a canonicalized query string.
- `<id>`: A unique ID assigned to this query.
- `<updated>`: The last time search results for this query were updated (set to midnight of the current day).
- `<link>`: A URL to retrieve this feed via a GET request.
- `<opensearch:totalResults>`: The total number of search results for this query.
- `<opensearch:startIndex>`: The 0-based index of the first returned result in the total results list.
- `<opensearch:itemsPerPage>`: The number of results returned.

### Entry Elements (Article Metadata)
- `<title>`: The title of the article.
- `<id>`: A URL in the format `http://arxiv.org/abs/id`.
- `<published>`: The date that version 1 of the article was submitted.
- `<updated>`: The date that the retrieved version of the article was submitted.
- `<summary>`: The article abstract.
- `<author>`: Contains one or more `<name>` child elements.
- `<link>`: Up to 3 URLs associated with the article.
- `<category>`: The arXiv, ACM, or MSC category.
- `<arxiv:primary_category>`: The primary arXiv category.
- `<arxiv:comment>`: Author comments, if present.
- `<arxiv:affiliation>`: Author affiliation (subelement of `<author>`).
- `<arxiv:journal_ref>`: Journal reference, if present.
- `<arxiv:doi>`: A URL for the resolved DOI to an external resource.

  5. How search results and queries are represented in data

    develop

    The project uses different data structures for queries and search results to balance type safety and performance:

    • Queries: Represented using dataclasses. Since queries are "low-volume" structs, the performance overhead of dataclasses is acceptable.
    • Search Results: Represented using TypedDicts. Because search results are high-volume, TypedDicts are used to provide type-checking machinery while avoiding the significant performance cost (up to two orders of magnitude) of casting between dataclasses and native Python dicts.

    This approach aligns with the data domain concept in the arXiv service architecture.

  7. Understand the UI design documentation types

    develop

    The UI documentation in this repository uses two primary types of reference files to describe the interface:

    • Workflow documents: Visualizations of specific screens and user actions required to complete a task or user story.
    • Wireframe documents: Simple layout structures used to define page regions, overall structure, and high-level overviews to inform prototyping.

  8. How paper versions are indexed and searched

    develop

    To support searching for specific versions of a paper, the system indexes each individual version of a paper as a separate document. The behavior of search interfaces differs based on the search type:

    • Simple and Author Search: Results are filtered to exclude any versions that are not the most recent version of a paper.
    • Advanced Search: No versions are excluded. To prevent multiple versions of the same paper from clustering together in relevance-sorted results, current versions are boosted.

  9. Understand the arXiv API Atom response format

    develop

    The arXiv API returns results in Atom format (XML). A typical response includes metadata about the query, total results, and a list of <entry> elements. Each entry contains details such as <id>, <published>, <title>, <summary>, <author>, and links to the abstract and PDF versions of the paper.

    Key XML namespaces used:

    • http://www.w3.org/2005/Atom (Standard Atom)
    • http://a9.com/-/spec/opensearch/1.1/ (OpenSearch metadata like totalResults)
    • http://arxiv.org/schemas/atom (arXiv-specific fields like comment, journal_ref, and primary_category)
    <?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns:arxiv="http://arxiv.org/schemas/atom">
  <opensearch:totalResults>1000</opensearch:totalResults>
  <entry xmlns="http://www.w3.org/2005/Atom" xmlns:arxiv="http://arxiv.org/schemas/atom">
    <title>Multi-Electron Production...</title>
    <summary>Multi-electron production is studied...</summary>
    <arxiv:comment>23 pages, 8 figures and 4 tables</arxiv:comment>
  </entry>
</feed>

  10. Search for exact phrases using quotes

    develop

    Quotes are used to define phrases where words must be found together in the specified sequence.

    Key Behaviors:

    • Phrase Search: ti:("%22hawaii+pineapple%22) finds the exact sequence "hawaii pineapple" in the title.
    • Combining Phrases and Terms: ti:("%22hawaii+pineapple%22+AND+matrix) finds the phrase and the word "matrix".
    • Quotes as OR: Multiple quoted terms without an operator act as an OR (e.g., ti:"pineapple" "apices" searches for ti:"pineapple" OR ti:"apices").
    • Limitation: Adding quotes does not always prevent partial matches (e.g., "pineapple" might still match pineappl).
    curl -is https://export.arxiv.org/api/query\?search_query\=ti:%28%22hawaii+pineapple%22+AND+matrix%29 | gi '<title' | sort

  11. Understand the Search UI routing and controllers

    develop

    The Search UI service uses search.routes.ui to route incoming request parameters to specific controllers. The system separates logic into two main controllers to allow for independent evolution of the user interface:

    • search.controllers.simple: Handles basic search use cases.
    • search.controllers.advanced: Handles complex or advanced search queries.

    Each controller module is responsible for its own form handling and validation logic.

  12. How the arXiv search system architecture works

    develop

    The arXiv search system is composed of three primary layers: a search index, a search application, and an indexing agent.

    1. Search Index: An Elasticsearch cluster (managed via AWS Elasticsearch Service) that stores searchable documents.
    2. Search Application: A Python/Flask application that provides user-facing interfaces and a REST API. It retrieves data from the index to answer queries.
    3. Indexing Agent: A background process that keeps the index up to date. It listens to a Kinesis stream for notifications about new content, retrieves metadata from the docmeta endpoint, transforms it into search documents, and pushes them to Elasticsearch.

    Data flows from the core metadata repository through the indexing agent into Elasticsearch, where it is then served by the search application to users and API clients.