DOCSEARCHbyAlgolia
Submit Docs

Ingredient Parser

repository·master·Indexed 41 days ago

https://github.com/strangetom/ingredient-parser

A Python package that uses a sequence labelling NLP model to extract structured information—such as amounts, units, names, and preparation steps—from unstructured recipe ingredient sentences. It includes the `parse_ingredient` function for converting sentences into `ParsedIngredient` objects and provides a regex-based alternative via `parse_ingredient_regex`.

Tokens
42.1K
Snippets
108
Records
161
Agent score
50%

What's inside Ingredient Parser

  2. Overview of the Ingredient Parser package

    master
    The ingredient_parser_nlp package is a Python-based library designed to parse structured information from natural language recipe ingredient sentences. It is intended for developers who need to extract specific data points (such as quantities, units, and ingredient names) from unstructured text.

  3. Capabilities of the Webtools interface

    master

    The webtools interface provides several key workflows for interacting with the ingredient parser library:

    • Parser Testing: Directly use the trained parser (equivalent to calling parse_ingredient).
    • Data Management: Browse the ingredient sentence training database and edit labels for training entries.
    • Model Training & Tuning:
      • train_single: Trains a model with specific hyper-parameters and outputs accuracy statistics.
      • train_multiple: Trains multiple models using different random seed values and outputs aggregated accuracy statistics.
      • grid_search: Trains multiple models across different hyper-parameter sets and outputs accuracy statistics for each set.

  5. How features work in the Ingredient Parser

    master

    Features are properties of a token that are deterministically calculated without the parser model. The model learns weights for these features to determine the likelihood of a token having a specific label.

    Key characteristics of features:

    • Data Types: Features are always primitive types (str, bool). Numeric types (int, float) are converted to str.
    • Contextual Window: To provide context, features are calculated not just for the target token, but also for up to three surrounding tokens.
    • Contextual Naming: Features for surrounding tokens are distinguished from the current token's features by a prefix indicating their distance from the target token.
    • Token Normalization: To reduce complexity, all numeric tokens are replaced with the placeholder !num.

  6. How approximate amounts and the APPROXIMATE flag work

    master

    If an ingredient sentence uses words like approximately, about, or roughly, the parser sets the APPROXIMATE flag to True within the corresponding IngredientAmount object.

    >>> parse_ingredient("4 pork chops, about 400 g each")
ParsedIngredient(name=[IngredientText(text='pork chops', 
                                          confidence=0.9961, 
                                          starting_index=1)],
                     size=None,
                     amount=[IngredientAmount(quantity=Fraction(4, 1), 
                                              quantity_max=Fraction(4, 1), 
                                              unit='', 
                                              text='4', 
                                              confidence=0.999876, 
                                              starting_index=0, 
                                              unit_system=<UnitSystem.NONE: 'none'>, 
                                              APPROXIMATE=False, 
                                              SINGULAR=False, 
                                              RANGE=False, 
                                              MULTIPLIER=False, 
                                              PREPARED_INGREDIENT=False),
                             IngredientAmount(quantity=Fraction(400, 1), 
                                              quantity_max=Fraction(400, 1), 
                                              unit=<Unit('gram')>, 
                                              text='400 g', 
                                              confidence=0.995449, 
                                              starting_index=5, 
                                              unit_system=<UnitSystem.METRIC: 'metric'>, 
                                              APPROXIMATE=True, 
                                              SINGULAR=True, 
                                              RANGE=False, 
                                              MULTIPLIER=False, 
                                              PREPARED_INGREDIENT=False)],
                     preparation=None,
                     comment=None,
                     purpose=None,
                     foundation_foods=[],
                     sentence='4 pork chops, about 400 g each')

  7. Navigate the Web Tool code hierarchy

    master

    The project follows a WYSIWYG directory structure organized by feature and domain:

    • components/: Contains UI components grouped by feature:
      • labeller: "browse & label" UI.
      • parser: "try the parser" UI.
      • train: "train the model" UI.
      • shared: Reusable components.
    • domain/: Core logic and data:
      • api: Endpoint resolvers.
      • collections: Static data lists or JSON files.
      • store: Zustand store for application state.
      • types: TypeScript interfaces.
    • styles/: Custom CSS and Mantine-specific styling.
    • assets/: Images and static assets.
    • Root Files:
      • app.py: The basic API server entry point.
      • app.sockets.py: The web socket server entry point.

  8. Understand the Ingredient Parser model type and workflow

    master

    The ingredient parser uses a Conditional Random Fields (CRF) model implemented via pycrfsuite to label tokens in English recipe ingredient sentences.

    To use the model correctly, you must follow a three-step pipeline:

    1. Pre-processing: Perform text normalization on the raw sentence (see preprocess.py in the ingredient_parser/en/ directory).
    2. Model Inference: Pass the normalized sentence to the CRF model (model.en.crfsuite) to obtain token labels.
    3. Post-processing: Interpret the model's token and label output to generate a structured representation of the ingredient.

  9. How the Ingredient Parser model works

    master

    The library uses a sequence tagging natural language model to parse ingredients. The process follows a specific pipeline that transforms raw text into a structured ParsedIngredient object.

    Parsing Pipeline

    When parsing a sentence, the system follows these four steps:

    1. Sentence Normalisation: Transforms the input sentence into a standardized form to assist the model and post-processing.
    2. Feature Generation: Splits the sentence into tokens and generates a set of features for each token (based on the token itself or its context).
    3. Model Labeling: The model assigns a label to each token in the sequence.
    4. Post-processing: Interprets the sequence of tokens and labels to determine the different parts of the ingredient (e.g., quantity, unit, name) by grouping tokens with the same label or identifying specific patterns.

    Training Pipeline

    When training the model, the process involves:

    1. Sentence Normalisation
    2. Feature Generation
    3. Training: Using a large dataset of example sentences with known correct label sequences to train the model.
    4. Evaluation: Testing the trained model against example sentences with known labels that were not used during training.

  10. The mechanism for improving NLTK part of speech tags

    master

    To improve the accuracy of NLTK's part of speech tagging for ingredient sentences, the library employs a two-step augmentation strategy:

    1. Automated Augmentation: The library uses Spacy's English transformer model to identify high-confidence tags. Any token that appears at least 20 times in the training data with the same tag at least 90% of the time is added to a new ingredient_tagdict.
    2. Manual Augmentation: The library's distributed tagdict is further refined by manually inserting entries for unambiguous words that the automated process or standard NLTK fails to tag correctly in real-world ingredient sentences.

    When the pos_tag function is called, it loads the NLTK tagger and updates its internal tagdict with this specialized ingredient_tagdict before processing tokens.

    def pos_tag(tokens: list[str]) -> list[tuple[str, str]]:
    tagger = _get_tagger("eng")
    ingredient_tagdict = load_ingredient_tagdict()
    tagger.tagdict.update(ingredient_tagdict)
    return _pos_tag(tokens=tokens, tagset=None, tagger=tagger, lang="eng")

  11. Handling ingredients with different amounts (Limitation)

    master

    The library assumes an ingredient sentence does not contain different ingredients with different amounts. If a sentence contains multiple phrases specifying different ingredients in different amounts, the parser will return the first identified ingredient and place everything following the first phrase into the comment field.

    >>> parse_ingredient("1 cup peeled and cooked fresh chestnuts (about 20), or 1 cup canned, unsweetened chestnuts")
ParsedIngredient(name=[IngredientText(text='fresh chestnuts', 
                                      confidence=0.977679, 
                                      starting_index=5)],
                 size=None,
                 amount=[IngredientAmount(quantity=Fraction(1, 1), 
                                          quantity_max=Fraction(1, 1), 
                                          unit=<Unit('cup')>, 
                                          text='1 cup', 
                                          confidence=0.999549, 
                                          starting_index=0, 
                                          unit_system=<UnitSystem.US_CUSTOMARY: 'us_customary'>, 
                                          APPROXIMATE=False, 
                                          SINGULAR=False, 
                                          RANGE=False, 
                                          MULTIPLIER=False, 
                                          PREPARED_INGREDIENT=True),
                         IngredientAmount(quantity=Fraction(20, 1), 
                                          quantity_max=Fraction(20, 1), 
                                          unit='', 
                                          text='20', 
                                          confidence=0.887524, 
                                          starting_index=9, 
                                          unit_system=<UnitSystem.NONE: 'none'>, 
                                          APPROXIMATE=True, 
                                          SINGULAR=False, 
                                          RANGE=False, 
                                          MULTIPLIER=False, 
                                          PREPARED_INGREDIENT=False)],
                 preparation=IngredientText(text='peeled and cooked', 
                                            confidence=0.999523, 
                                            starting_index=2),
                 comment=IngredientText(text='or 1 cup canned, unsweetened chestnuts', 
                                        confidence=0.925894, 
                                        starting_index=12),
                 purpose=None,
                 foundation_foods=[],
                 sentence='1 cup peeled and cooked fresh chestnuts (about 20), or 1 cup canned, unsweetened chestnuts')

  12. Understand implicit quantities

    master

    The parser identifies implicit quantities where the amount is not explicitly stated but implied by the unit or context. For example:

    • '15 oz can black beans': Implies 1 can.
    • 'Rosemary sprig': Implies a quantity of 1 because 'sprig' is singular.

    In these cases, the quantity in the IngredientAmount object is explicitly set to 1. To prevent errors, the parser checks that the unit is not plural and that no indefinite quantifiers (like 'few' or 'some') precede it.