Quepid

Quepid's frontend is currently split between two distinct stacks. The Core case UI (handling queries, ratings, and Solr JSONP) is an AngularJS 1.8 Single Page Application (SPA) located at /case/:id and /case/:id/try/:try_number. The rest of the application (teams, books, scorers, etc.) runs on Rails + Stimulus (using ERB and Turbo Streams).

A migration plan is in progress to completely remove AngularJS. The strategy is to migrate isolated, lower-risk pieces first (like toolbar Stimulus twins and management modals) and tackle the high-complexity live query/search/score state last. The backend (Rails 8.1, MySQL, REST API) remains stable and is not part of the frontend stack migration.

A Scorer is a JavaScript formula that transforms a set of graded judgements into a single relevance score for a query (for example, calculating metrics like nDCG@10 or Precision@10).

Quepid provides two types of scorers:

• Communal Scorers: Built-in scorers available to all users (e.g., nDCG@10, AP@10). These are owned by the "System".
• Custom Scorers: Scorers created by users. These can be private or shared with a team.

You can access the Scorers management page via the Sidebar icon (list-task) or by navigating to /scorers.

The Core Workbench is the primary interface for relevance-tuning workflows in Quepid. It is accessed via the URLs /case/:id or /case/:id/try/:try_number.

In the Workbench, you can:

• Add and manage queries.
• Run queries against your search endpoint.
• Rate documents to provide ground truth.
• Monitor real-time score changes.
• Tweak query and engine settings via the 'Tune Relevance' drawer.

A Book is Quepid's offline relevance-judgement workflow. It consists of a set of query/document pairs (Query Doc Pairs) that one or more people (or AI Judges) rate independently (Judgements). These judgements can later be synced back into a Case's ratings.

Every book page includes a tab strip for managing its lifecycle and data:

• Overview: Summary of the book, including AI judge status, teams, and related cases.
• Query/Doc Pairs: The list of pairs to be judged.
• Judgement Stats: Statistical overview of ratings.
• Judgements: The individual ratings provided by users or AI.
• Import: Tools to bring in new books or additional data via JSON.
• Share book: Controls for visibility.
• Export: Tools to download book data as JSON or CSV.
• Settings: Configuration for the book, including rating scales and danger-zone maintenance tools.

A Search Endpoint is a configuration in Quepid that defines how to communicate with a search engine or API. It allows Quepid cases to query the engine and retrieve results.

Supported engine types include:

• Solr
• Elasticsearch
• OpenSearch
• Vectara
• Algolia
• Static File
• Search API (for bespoke/custom APIs requiring custom JavaScript translation)

The Case is the central entity for search experiments. It acts as a connector for all elements related to a specific search setup.

Core Components:

• Case: The primary container for an experiment. Identified by a name.
• Query: A representation of a search term within a Case. It includes query_text and can have associated notes or options. Queries can be re-ordered within a case.
• Rating: Users assign ratings to search results. A rating is linked to a Query via query_id and to a specific search result via doc_id.

Note: Quepid does not store the full search results or the search engine's response; it only stores the doc_id of the results being rated.

Before starting the final phase of the live query-state migration, the following architectural questions (decision lenses) must be addressed:

• Incremental shipper: What can be shipped without a feature freeze (e.g., Toolbar Stimulus twins, DOM utilities)?
• Search/IR domain: Will live tuning continue to search customer engines from the browser (using a proxy when needed)?
• UX fidelity: Can the rating experience remain instant (currently achieved via client-side scoreAll() on every rating)?
• Security: What will replace browser eval() for scorers (e.g., Web Workers)?
• Performance: How to handle large cases (1,000+ queries) where scoreAll() is currently O(n queries).
• A11y: Ensuring scores are not communicated via color alone and implementing real ARIA on badges and rating controls.

Quepid connects to Solr using one of two methods depending on your hosting environment:

1. JSONP (Default/Cross-Domain): Used when Quepid and Solr are hosted on different domains. This avoids CORS issues. When using JSONP, Solr responses are wrapped in a dynamically generated JavaScript function name provided via the json.wrf parameter.
2. Traditional GET (Same-Domain): Used if Quepid and Solr are hosted on the same domain. This returns standard JSON responses without the function wrapper.

If you see a json.wrf parameter in your request, you must treat the resulting response as a JavaScript function call rather than raw JSON.

// Example of a JSONP wrapped response
angular.callbacks._2({
  "responseHeader":{ ... },
  "response":{ ... }
});

// Example of a standard GET JSON response
{
  "responseHeader":{ ... },
  "response":{ ... }
}

Quepid tracks configuration changes and search engine connectivity through Try and SearchEndpoint models.

Configuration History (Try)

To allow developers to tweak search engine configurations and test results, Quepid uses the Try model. Every time a configuration is changed, a new Try record is created. This maintains a history of changes, making it easy to revert to previous settings.

Connecting to Search Engines (SearchEndpoint)

Each Try connects to a search engine via a SearchEndpoint.

• Standard Endpoints: Support engines like Solr or OpenSearch.
• SearchAPI Endpoint: A flexible variation that allows you to provide custom JavaScript to map any search response format into the specific format Quepid expects.