ranx

ranx is a high-speed ranking evaluation library for Information Retrieval and Recommender Systems. It leverages Numba for fast vector operations and automatic parallelization.

Key capabilities include:

• Metrics: High-speed implementation of standard IR metrics (NDCG, MAP, MRR, etc.).
• Statistical Tests: Built-in support for Paired Student's t-Test, Fisher's Randomization Test, and Tukey's HSD Test.
• Fusion: Various fusion algorithms (RRF, BordaFuse, etc.) and automatic fusion optimization.
• Data Loading: Easy integration with ir-datasets for Qrels and ranxhub for pre-computed runs.

Rank Norm

Transforms scores based on their position in the ranking. The top-ranked result gets a score of 1, and the bottom-ranked result gets a score of $1/|r|$, where $|r|$ is the number of results. This results in a uniform distribution. norm='rank'

Borda Norm

Transforms scores similarly to how BordaFuse assigns points. It accounts for whether a candidate is present in the ranked list. norm='borda'

Min-Max Norm

Scales scores between 0 and 1, where $s_{min}$ is 0 and $s_{max}$ is 1. norm='min-max'

Max Norm

Scales scores such that the maximum score ($s_{max}$) is 1. norm='max'

Sum Norm

Scales the minimum score ($s_{min}$) to 0 and ensures the sum of all scores is 1. norm='sum'

ZMUV Norm

Scales scores to have a zero mean and unit variance (zero-mean, unit-variance). norm='zmuv'

Normalization transforms result list scores into a comparable range, which is required for many fusion methods in ranx. You can specify the normalization strategy using the norm parameter in the fuse and optimize_fusion functions. By default, ranx uses min-max normalization.

# Example usage (conceptual)
# fuse(runs, norm='min-max')
# optimize_fusion(runs, norm='zmuv')

The ranx library is specifically designed for ranking tasks, not classification tasks.

While some metrics like precision and recall are used in both domains, ranx treats relevance scores as values used to sort results. You should not use ranx to evaluate predicted class labels. In ranx, relevance scores are used to determine the order of results before metrics are computed, regardless of the actual numerical value of the score.

You can control the visual output of a Report (both when printing to the console and when generating LaTeX tables) by adjusting the following parameters on the Report instance:

• rounding_digits (int): Controls the number of decimal places shown.
• show_percentages (bool): Controls whether results are displayed as percentages.

When using the compare function to evaluate the difference between two or more runs, you can perform statistical significance testing by passing the stat_test argument. ranx currently supports two types of tests:

1. Fisher's Randomization Test: Pass stat_test="fisher".
2. Two-sided Paired Student's t-Test: Pass stat_test="student".

These tests help determine if the observed differences in performance metrics between runs are statistically significant.

# Example usage pattern for the compare function
# (Note: exact function signature depends on the compare API implementation)
results = compare(run1, run2, stat_test="fisher")
# OR
results = compare(run1, run2, stat_test="student")

If your relevance scores are computed using distance metrics (where a lower score indicates higher relevance), use min-max-inverted. This scales scores between 0 and 1 such that the minimum score ($s_{min}$) becomes 1 and the maximum score ($s_{max}$) becomes 0.

# Use when lower scores mean higher relevance
# fuse(runs, norm='min-max-inverted')

You can learn how to use ranx through a series of interactive Google Colab notebooks that cover the entire workflow, from basic setup to advanced analysis:

• Overview: General introduction to the library.
• Qrels and Run: How to create and manage relevance judgments (qrels) and ranking results (runs).
• Evaluation: Performing ranking evaluation.
• Comparison and Report: Comparing different runs and generating reports.
• Fusion: Using fusion algorithms for metasearch.
• Plot: Visualizing results.
• ranxhub: How to share your runs with ranxhub.

https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/1_overview.ipynb

A run card is a YAML file containing metadata that improves discoverability and enables filtering on the ranxhub. It is divided into four main sections:

1. Run metadata: Computation details, author, and indexing tags.
2. Benchmark metadata: Name, dataset, split, version, and ir-datasets ID.
3. Paper metadata: Title, authors (with ORCID), publication date, DOI, and DBLP URL.
4. Model metadata: Model name, description, tags, and publication details (similar to paper metadata).