DOCSEARCHbyAlgolia
Submit Docs

text-dedup

repository·main·Indexed 20 days ago

https://github.com/chenghaomou/text-dedup

A collection of high-performance text deduplication tools supporting near-duplicate detection via MinHash and SimHash, as well as exact deduplication using Bloom Filters and Suffix Arrays. Version 0.4.1 provides scripts for processing text data with configurations managed via TOML files and includes a comprehensive benchmark suite for evaluating algorithms on the CORE and NEWS-COPY datasets.

Tokens
9.8K
Snippets
36
Records
45
Agent score
73%

What's inside text-dedup

  1. Overview of text-dedup algorithms

    main

    The text-dedup repository provides several ready-to-use scripts for different deduplication needs:

    • MinHash + MinHashLSH: For near-duplicate detection.
    • SimHash: 64 or 128 bit hashing for near-duplicate detection.
    • SuffixArray: Substring exact deduplication.
    • Bloom Filter: Exact deduplication.

    All algorithms are configured via TOML files.

  2. Understand benchmark datasets and metrics

    main

    The benchmark suite evaluates algorithms on two primary datasets with different evaluation goals:

    1. CORE Dataset (pinecone/core-2020-05-10-deduplication)

    • Purpose: Academic paper deduplication.
    • Ground Truth: Labeled duplicate pairs.
    • Metrics:
      • Precision (Duplicates/Non-Duplicates)
      • Recall (Duplicates/Non-Duplicates)
      • Macro F1
      • Accuracy

    2. NEWS-COPY Dataset (chenghao/NEWS-COPY-eval)

    • Purpose: News article near-duplicate detection.
    • Ground Truth: Cluster labels.
    • Metric:
      • ARI (Adjusted Rand Index): Measures similarity between predicted and true clustering (Range: [-1, 1], where 1 is perfect clustering).

  3. Run benchmarks manually via Python module

    main

    If you are not using just, you can execute the benchmark suite directly using python -m benchmarks.run_benchmark. Use the --dataset and --algorithms flags to control the execution scope.

    # All datasets, all algorithms
python -m benchmarks.run_benchmark --dataset all --algorithms all

# Specific dataset
python -m benchmarks.run_benchmark --dataset core --algorithms minhash,simhash
python -m benchmarks.run_benchmark --dataset news --algorithms minhash

# Multiple datasets
python -m benchmarks.run_benchmark --dataset core news --algorithms all

  5. Run deduplication scripts

    main

    After configuring your config.toml, run the corresponding module using the python -m command.

    # MinHash
python -m text_dedup.minhash

# SimHash
python -m text_dedup.simhash

# Bloom Filter
python -m text_dedup.bloom_filter

# Suffix Array
python -m text_dedup.suffix_array

  6. Run benchmarks using Just

    main

    The fastest way to execute benchmarks is using the just command. You can run all benchmarks, specific datasets, or specific algorithm/dataset combinations using the following commands:

    # Run all benchmarks (both datasets, all algorithms)
just benchmark-all

# Run only CORE dataset benchmarks
just benchmark-core

# Run only NEWS-COPY dataset benchmarks
just benchmark-news

# Run specific algorithm on specific dataset
just benchmark-core-minhash
just benchmark-core-simhash
just benchmark-news-minhash
just benchmark-news-simhash

  7. Add a new algorithm to the benchmark suite

    main

    To include a new algorithm in the benchmarking process, follow these steps:

    1. Implement the benchmark logic: Create a run_<algorithm>_benchmark() function in the relevant dataset file (benchmark_core.py or benchmark_news.py).
    2. Create a configuration: Add a new .toml file in configs/ following the pattern benchmark_<dataset>_<algorithm>.toml.
    3. Register the algorithm: Update run_benchmark.py to include the new algorithm in the execution logic.
    4. Update the CLI: Add a corresponding command in the justfile for easy access.

    Example implementation in benchmark_core.py:

    # In benchmark_core.py
def run_bloom_benchmark(config: Config, labels: dict, id_to_core_id: dict) -> tuple[dict, float]:
    timer = Timer()
    with timer("BloomFilter"):
        bloom_main(config)
    # ... evaluation logic
    return metrics, timer.timings["BloomFilter"]

  10. Configure Bloom Filter exact deduplication

    main

    To use Bloom Filter for exact deduplication, edit the config.toml file. Key parameters include:

    • error_rate: Desired false positive rate.
    • expected_elements: The expected number of elements in the filter.
    [input]
input_type = "local_files"
file_type = "parquet"

[input.read_arguments]
path = "data/your_data"
split = "train"

[algorithm]
algorithm_name = "bloom_filter"
text_column = "text"
error_rate = 1e-5
expected_elements = 100000

[output]
output_dir = "output"
clean_cache = false

[debug]
enable_profiling = false

  11. Configure Suffix Array substring exact deduplication

    main

    To use Suffix Array for substring exact deduplication, edit the config.toml file. Key parameters include:

    • google_repo_path: Path to the third-party deduplication tool.
    • merge_strategy: Strategy for merging (e.g., longest).
    • length_threshold: Minimum length for deduplication.
    [input]
input_type = "local_files"
file_type = "parquet"

[input.read_arguments]
path = "data/your_data"
split = "train"

[algorithm]
algorithm_name = "suffix_array"
text_column = "text"
google_repo_path = "third_party/deduplicate-text-datasets"
merge_strategy = "longest"
length_threshold = 100
cache_dir = ".cache"

[output]
output_dir = "output"
clean_cache = false

[debug]
enable_profiling = false

  12. Configure benchmark settings via TOML

    main

    Benchmark configurations are stored as TOML files in the configs/ directory. You can customize hyperparameters by modifying these files. Available configurations include:

    • benchmark_core_minhash.toml: MinHash on CORE dataset
    • benchmark_core_simhash.toml: SimHash on CORE dataset
    • benchmark_news_minhash.toml: MinHash on NEWS-COPY dataset
    • benchmark_news_simhash.toml: SimHash on NEWS-COPY dataset