repository·master·Indexed 22 days ago
https://github.com/castorini/anseriniA toolkit built on top of Apache Lucene designed for reproducible information retrieval (IR) research. Anserini provides tools for indexing custom collections in MS MARCO JSONL format and performing retrieval, with a Python interface available via Pyserini. The toolkit supports Lucene 10.4.0 (v2.0.0) and maintains readability for Lucene 9 indexes.
To measure the quality of a retrieval system, you compare its output against relevance judgments (also known as qrels).
Relevance Judgments (qrels):
These are artifacts that map queries to documents with a relevance score. They are typically represented as triples:
[query_id] [document_id] [relevance_score]
Example format:
q1 doc23 0
q1 doc452 1
q1 doc536 0
q2 doc97 0(In this example, doc452 is relevant to q1, while doc23 and doc536 are not).
Metrics: Metrics quantify the 'goodness' of a ranked list. A common example is Precision at 10 (P@10), which calculates the fraction of the top 10 documents that are relevant according to the qrels. The final system performance is typically the average of these scores across all queries.
To facilitate the optimization process, metadata (such as model_type, num_heads, and hidden_size) should be embedded into the ONNX model. This is done by loading the exported ONNX model and adding properties to metadata_props before saving.
# Example of adding metadata to an ONNX model
model_type = model.config.model_type
num_heads = model.config.num_attention_heads
hidden_size = model.config.hidden_size
onnx_model = onnx.load(onnx_path)
meta = onnx_model.metadata_props.add()
meta.key, meta.value = 'model_type', model_type
meta = onnx_model.metadata_props.add()
meta.key, meta.value = 'num_heads', str(num_heads)
meta = onnx_model.metadata_props.add()
meta.key, meta.value = 'hidden_size', str(hidden_size)
onnx.save(onnx_model, onnx_path)Anserini was upgraded to Lucene 10.4.0 as part of v2.0.0.
Anserini provides an end-to-end framework for reproduction experiments via the io.anserini.reproduce.ReproduceFromDocumentCollection driver. This driver automates the entire pipeline from raw data to evaluated results.
When running a reproduction, the driver performs these steps in order:
Documentation pages for these experiments are automatically generated from templates located in src/main/resources/reproduce/from-document-collection/docgen.
bin/run.sh io.anserini.reproduce.ReproduceFromDocumentCollection --index --verify --search --config cacmIn Anserini, the SourceDocument class distinguishes between raw() and contents() to ensure collections handle their own parsing logic:
raw(): Returns the raw JSON data of the document.contents(): Returns the extracted, parsed content (e.g., article text) intended for indexing.This separation ensures that 'empty document checks' are performed on the actual parsed content rather than the JSON wrapper, providing a more accurate count of valid documents. When implementing or extending collections, ensure parsing logic resides within the collection implementation to satisfy this principle.
272565). The features described in this documentation are no longer available. This guide is retained for historical purposes only.e71df7aee42c7776a63b9845600a4075632fa11c. This upgrade changes the retrieval results compared to the previous Lucene 6 version. When comparing effectiveness metrics (like AP, NDCG, or P10) against official TREC runs or older reproductions, ensure you are aware of which Lucene version the Anserini build is using, as it directly impacts the scores.The SpladeExportWrapper is a PyTorch nn.Module used during the conversion process to add a 'layer' of logic that transforms raw model outputs into sparse vectors compatible with Anserini.
It performs the following steps in its forward pass:
input_ids and attention_mask to the underlying SPLADE model.ReLU and log1p to the logits.attention_mask to zero out padding.torch.max across the sequence dimension to produce a sparse vector.class SpladeExportWrapper(nn.Module):
def __init__(self, splade_model):
super().__init__()
self.splade = splade_model
def forward(self, input_ids, attention_mask, token_type_ids):
# Use token_type_ids in a no-op way to force ONNX to retain it
dummy = token_type_ids.sum() * 0.0
outputs = self.splade(input_ids=input_ids, attention_mask=attention_mask)
logits = outputs.logits
relu = torch.relu(logits)
log1p = torch.log1p(relu)
sparse_vec = torch.max(log1p * attention_mask.unsqueeze(-1), dim=1).values
nonzero = sparse_vec.nonzero(as_tuple=True)
values = sparse_vec[nonzero]
return nonzero[1], values + dummyWhen working with MS MARCO in Anserini, the following data structures are used:
collection.tsv or JSONL)docid (unique identifier) and the passage text.queries.tsv)qid (unique query identifier) and the query text.qrels.tsv)Standard format for relevance judgments:
qid: Query identifier.0: Historical artifact (almost always 0).docid: Document/passage identifier.relevance: The judgment (e.g., 0 for not relevant, 1 for relevant).In information retrieval (search), the goal is to solve the retrieval problem: given an information need expressed as a query ($q$), return a ranked list of $k$ documents ($d_1, d_2, ext{...}, d_k$) from a finite collection (or corpus) $C$ that maximizes a specific metric (e.g., nDCG, AP).
Key components include:
docs/ref-reproduce-from-prebuilt-indexes.md.