DOCSEARCHbyAlgolia
Submit Docs

smart_importer

repository·main·Indexed 43 days ago

https://github.com/beancount/smart_importer

A tool that augments Beancount and Beangulp importers with machine learning capabilities using scikit-learn. It provides PredictPayees and PredictPostings to predict transaction attributes like payees and account postings based on historical ledger data. It can be integrated via Beangulp hooks or by wrapping specific importers using the ImporterWrapper class.

Tokens
4.1K
Snippets
14
Records
17
Agent score
54%

What's inside smart_importer

  1. How PredictPostings and PredictPayees work

    main

    The smart_importer package provides two primary machine learning tools:

    • PredictPostings: Predicts the list of postings (accounts) for a transaction.
    • PredictPayees: Predicts the payee of a transaction.

    Requirements for use:

    1. Importer Interface: Your importer's extract method must follow the latest beangulp interface and include an existing_entries argument.
    2. Single-legged Postings: The tool only works by appending to incomplete single-legged postings. It will not work on postings that already have accounts like Expenses:TODO assigned.

  2. Apply smart_importer by wrapping specific importers

    main

    If you only want to apply machine learning to a specific importer rather than all of them, you can use the .wrap() method. This modifies only the importer instance it is called on.

    from your_custom_importer import MyBankImporter
from smart_importer import PredictPayees, PredictPostings

CONFIG = [
    PredictPostings().wrap(
        PredictPayees().wrap(
            MyBankImporter('whatever', 'config', 'is', 'needed')
        )
    ),
]

HOOKS = []

  3. Provide training data for smart_importer

    main

    The machine learning model requires existing Beancount transactions to learn from. You can provide this training data in two ways:

    1. CLI: When running bean-extract, use the -e argument to reference your existing transactions file: import.py extract -e existing_transactions.beancount

    2. Fava: If you are using Fava, it automatically uses existing entries as training data.

  5. Apply smart_importer using Beangulp hooks

    main

    You can apply machine learning capabilities to all your importers by defining a HOOKS list in your configuration. This is the recommended way to use smart_importer with beangulp.

    from your_custom_importer import MyBankImporter
from smart_importer import PredictPayees, PredictPostings

CONFIG = [
    MyBankImporter('whatever', 'config', 'is', 'needed'),
]

HOOKS = [
    PredictPostings().hook,
    PredictPayees().hook
]

  6. Configure a custom tokenizer for smart_importer

    main

    By default, smart_importer uses a standard tokenizer. You can provide a custom string_tokenizer (a callable that takes a string and returns a list of tokens) to support different languages, such as Chinese using jieba.

    from smart_importer import PredictPostings
import jieba

jieba.initialize()
tokenizer = lambda s: list(jieba.cut(s))

predictor = PredictPostings(string_tokenizer=tokenizer)

  8. Merge modified transactions with `merge_non_transaction_entries`

    main

    Use merge_non_transaction_entries to combine a list of original imported entries with a list of enhanced (modified) transactions.

    This is useful during the import process when you have a sequence of Beancount Directive objects (which includes both Transaction and non-transaction entries like Open or Balance) and you want to replace only the Transaction objects with your newly processed/enhanced versions while keeping the other directives in their original positions.

    Logic:

    • It iterates through imported_entries.
    • If an entry is an instance of Transaction, it pulls the next available entry from enhanced_transactions to replace it.
    • If the entry is not a Transaction, it keeps the original entry from imported_entries.
    from smart_importer.entries import merge_non_transaction_entries

# imported_entries: Sequence[Directive] (contains Transactions and other Directives)
# enhanced_transactions: Sequence[Directive] (contains only the modified Transactions)

final_entries = merge_non_transaction_entries(imported_entries, enhanced_transactions)

  9. Update transaction postings with `update_postings`

    main

    Use update_postings to synchronize a Beancount Transaction's postings with a specific list of accounts.

    Behavior:

    • The function only acts if the transaction currently has exactly one posting. If it has zero or multiple postings, it returns the transaction unchanged.
    • It creates a new list of postings where every account in the provided accounts list gets a posting.
    • If the original posting's account is in the accounts list, that specific posting is preserved at its correct index.
    • If the original posting's account is NOT in the accounts list, it is appended to the end of the new postings list.
    • It returns a new Transaction object (via _replace) with the updated postings.
    from beancount.core.data import Transaction
from smart_importer.entries import update_postings

# Assuming 'tx' is a Transaction object with one posting
new_tx = update_postings(tx, ["Assets:Checking", "Expenses:Groceries", "Income:Salary"])

  10. Use ImporterWrapper to enrich existing importers

    main

    The ImporterWrapper class is a decorator-style wrapper used to apply smart_importer prediction logic to an existing beangulp.importer.Importer.

    It works by wrapping a standard importer and passing its extracted entries through an EntryPredictor.hook. This allows the predictor to modify, enrich, or correct the entries (such as adjusting accounts or dates) during the extraction process.

    When calling extract(), the wrapper:

    1. Calls the underlying importer's extract method.
    2. Retrieves the account from the underlying importer.
    3. Passes the filepath, entries, account, and the original importer to the predictor.hook method.
    4. Returns the modified entries provided by the predictor.
    from beangulp.importer import Importer
from smart_importer.predictor import EntryPredictor
from smart_importer.wrapper import ImporterWrapper

# Assuming you have an existing importer and a predictor
my_importer = MyExistingImporter()
my_predictor = MyEntryPredictor()

# Wrap the importer
wrapped_importer = ImporterWrapper(my_importer, my_predictor)

# Use the wrapped importer as you would a normal one
entries = wrapped_importer.extract("path/to/statement.csv", existing_directives)

  11. Use EntryPredictor to wrap an importer

    main

    The EntryPredictor class is the core engine for machine learning-based attribute prediction in smart_importer. To use it, you should create an instance of EntryPredictor and use its wrap method to transform a standard beangulp.importer.Importer into an ImporterWrapper. This wrapper will automatically handle training the model on existing entries and applying predictions to new ones during the import process.

    from smart_importer.predictor import EntryPredictor

# Assuming 'my_importer' is a standard beangulp Importer
predictor = EntryPredictor(predict=True, overwrite=True)
smart_importer = predictor.wrap(my_importer)

  12. Configure EntryPredictor

    main

    When initializing EntryPredictor, you can configure how the machine learning model behaves using the following parameters:

    • predict (bool): Whether to add predictions to the entries. Defaults to True.
    • overwrite (bool): If True, when an attribute is predicted but already exists on an entry, the existing value will be overwritten. Defaults to False.
    • string_tokenizer (Callable[[str], list[str]]): A custom callable that takes a string and returns a list of tokens. Useful for supporting non-English languages.
    • string_token_pattern (str): A regex used for tokenizing text if no custom tokenizer is provided. Default is r"(?u)\b\w\w+\b".
    • denylist_accounts (list[str]): A list of account names. Transactions containing any of these accounts will be excluded from the training data.
    predictor = EntryPredictor(
    predict=True,
    overwrite=True,
    string_tokenizer=my_custom_tokenizer,
    denylist_accounts=["Assets:Internal:Test"]
)