repository·main·Indexed 22 days ago
https://github.com/inukshuk/anystyleA fast, machine-learning-based parser for bibliographic references that extracts structured data from citation strings. It provides a Ruby API and a command-line interface (anystyle-cli) for parsing and finding references. Features include support for Latin and Cyrillic scripts, custom model training, and multiple dictionary adapters (Ruby, Hash, GDBM, Redis, LMDB). It can process raw strings, PDF, TTX, and TXT files, and export data to RIS format.
AnyStyle is optimized for English-language bibliographies but supports references written in any Latin script (e.g., most European languages, Indonesian, Malaysian, romanized Arabic, Chinese, and Japanese). It also supports non-Latin alphabets like Cyrillic.
Limitations: AnyStyle is not compatible with languages that do not use white space to separate tokens (e.g., Chinese, Japanese, Arabic, and Indian languages in their native scripts).
To use AnyStyle as a library in your Ruby projects, install the anystyle gem. You can then use the static AnyStyle.parse and AnyStyle.find methods to process bibliographic strings.
require 'anystyle'
pp AnyStyle.parse 'Derrida, J. (1967). L’écriture et la différence (1 éd.). Paris: Éditions du Seuil.'require 'anystyle'
pp AnyStyle.parse 'Derrida, J. (1967). L’écriture et la différence (1 éd.). Paris: Éditions du Seuil.'To use AnyStyle from the command line, install the anystyle-cli gem via RubyGems.
$ [sudo] gem install anystyle-cliAfter training, you can evaluate your model's performance against a curated dataset (like the gold dataset) using the check command. This reports sequence and token error rates.
$ anystyle -P x.mod check ./res/parser/gold.xml$ anystyle -P x.mod check ./res/parser/gold.xmlYou can train custom Finder and Parser models using your own datasets. Use the anystyle train command followed by your training data file and the desired model name.
$ anystyle train training-data.xml custom.modTo use your custom model instead of the default, use the -P or --parser-model flag for the parser, and -F or --finder-model for the finder.
$ anystyle train training-data.xml custom.modOnce installed, you can use the anystyle command to parse bibliographic references or find them. Use --help to see available commands and subcommands like find and parse.
$ anystyle --help
$ anystyle help find
$ anystyle help parseAnyStyle uses a feature dictionary for statistical analysis. You can change the :adapter in AnyStyle::Dictionary.defaults to manage memory and disk usage:
:ruby: Uses a persistent Ruby hash (default). Slower startup, no extra dependencies.:hash: Uses an in-memory dictionary. Fast, but uses no disk space.:gdbm: Uses GDBM (requires require 'anystyle/dictionary/gdbm').:redis: Uses Redis (requires redis and redis/namespace gems and require 'anystyle/dictionary/redis').AnyStyle::Dictionary.defaults[:adapter] = :ruby
AnyStyle::Dictionary.defaults[:adapter] = :hash
require 'anystyle/dictionary/gdbm'
AnyStyle::Dictionary.defaults[:adapter] = :gdbm
AnyStyle::Dictionary.defaults[:adapter] = :redis
require 'anystyle/dictionary/redis'
AnyStyle::Dictionary::Redis.defaults[:host] = 'localhost'
AnyStyle::Dictionary::Redis.defaults[:port] = 6379When initializing AnyStyle::Finder, you can pass an options hash to configure the underlying parser and model behavior.
Default configuration values include:
model: Path to the finder model (default: SUPPORT/finder.mod).pattern: Path to the pattern file (default: SUPPORT/finder.txt).compact: Boolean (default: true).threads: Number of threads (default: 4).format: Default output format (default: :references).training_data: Directory containing training data .ttx files.layout: Boolean (default: true).pdftotext: Command for PDF text extraction (default: 'pdftotext').pdfinfo: Command for PDF info extraction (default: 'pdfinfo').dictionary: A dictionary object for word features.When initializing a AnyStyle::Parser, you can pass an options hash to override default behaviors.
Common configuration keys include:
model: Path to a specific Wapiti model file.format: The default output format (e.g., :bibtex, :ris, :hash).training_data: Path to training data for the model.separator: Regex used to split input (default: /(?:\r?\n)+/).delimiter: Regex used to split tokens (default: /(\s|\p{Space_Separator})+|([\uFF01-\uFF64]|。|、)/).dictionary: A dictionary object for the Dictionary feature.To clean or standardize bibliographic fields, you can create a subclass of AnyStyle::Normalizer. The base class provides utility methods for manipulating bibliographic items (represented as Hashes), but you must implement the normalize method in your subclass.
normalize(item, **opts): (Must be implemented) The entry point for your normalization logic. It should take a bibliographic item (Hash) and return the modified item.append(item, key, value): Safely adds a value to a key in the item. If the key already exists, it appends the value to the existing array; otherwise, it initializes a new array with that value.map_values(item, keys = keys_for(item)): Transforms values for specific keys. The block receives key and value. It automatically flattens results and rejects nil or empty values.each_value(item, keys = keys_for(item)): Iterates over values for the specified keys, yielding both the key and the value to the block.skip: A boolean flag that can be set to skip normalization steps.The map_values method allows you to apply transformations to specific fields within a bibliographic item. It handles the boilerplate of checking for key existence, mapping the values, flattening the resulting arrays, and removing nil or empty strings.
If no keys are provided to map_values, it defaults to the keys defined in the class (or all keys in the item if the class has no keys defined).
The parse method is the primary entrypoint for converting bibliographic text into structured data. It supports several output formats via the format argument.
Supported formats:
:hash (default):bibtex:citeproc:csl:ris:wapitiArguments:
input: The bibliographic string or array of strings to parse.format: The desired output format (defaults to options[:format]).**opts: Additional options passed to the underlying labeling and formatting logic.