tesstrain

Ground truth data must be placed in a specific directory structure: data/MODEL_NAME-ground-truth.

File Requirements

• Images: Must be TIFF (.tif) or PNG (.png, .bin.png, or .nrm.png).
• Transcriptions: Must be single-line plain text files with the same name as the image but with the extension replaced by .gt.txt (e.g., line01.png and line01.gt.txt).

Data Splitting

The files in this directory are split into training and evaluation sets based on the RATIO_TRAIN variable (default is 0.90).

You can interact with tesstrain in two ways:

1. Terminal Interface: Use the module as a CLI tool to interact with the training tools directly.
2. Python API: Import the package and use the high-level tesstrain.run() function within your own Python scripts.

# Using the terminal interface
python -m tesstrain --help

# Using the Python API (conceptual)
import tesstrain
tesstrain.run()

To install the tesstrain package, you can use pip. Note that this package requires the Tesseract training tools to be installed and available on your system. A supported Python version of at least 3.7 is required.

To install from PyPI:

pip install tesstrain

To install from a local source checkout:

pip install .

Tesseract requires configuration data (such as radical-stroke.txt and *.unicharset for all scripts) in the DATA_DIR. You can fetch these stock unicharsets using the following command. This is often done once before training, though it is implicitly included in the training target.

make tesseract-langdata

You can visualize the training and evaluation progress by generating plots from the training log using the plot target. This target can be run even while training is still in progress.

Generate plots

make plot

Evaluate models independently

To evaluate .checkpoint models on the evaluation dataset without generating plots:

make evaluation

# Generate plots for a specific model
make plot MODEL_NAME=ocrd

To start the training process, use the make training target. This is a shortcut that runs unicharset, lists, proto-model, tesseract-langdata, and training.

Basic Command

make training MODEL_NAME=your_model_name

Training Regimes

• From Scratch: Specify a NET_SPEC (network specification in VGSL).
• Fine-tuning: Provide a START_MODEL (the name of an existing model to continue from).

make training MODEL_NAME=name-of-the-resulting-model

If you do not explicitly provide input data via CLI flags, tesstrain automatically attempts to locate files within your specified --langdata_dir using the following pattern based on your --lang code:

• Training Text: {langdata_dir}/{lang_code}/{lang_code}.training_text
• Wordlist: {langdata_dir}/{lang_code}/{lang_code}.wordlist
• Bigrams: {langdata_dir}/{lang_code}/{lang_code}.word.bigrams
• Numbers: {langdata_dir}/{lang_code}/{lang_code}.numbers
• Punctuation: {langdata_dir}/{lang_code}/{lang_code}.punc

Additionally, frequency files like bigram_freqs, unigram_freqs, and train_ngrams are derived from the path of the chosen training_text file by replacing its suffix.

To run tesstrain successfully, ensure the following requirements are met:

1. Language Data: You must provide --langdata_dir.
2. Tessdata: You must provide --tessdata_dir OR have the TESSDATA_PREFIX environment variable defined.
3. Language Code: You must specify a language using --lang.

If --output_dir is not provided, the tool will create a temporary directory named trained-{lang_code}-{timestamp}.

Use make help to see all available targets. The primary targets are:

• unicharset: Create unicharset
• charfreq: Show character histogram
• lists: Create lists of lstmf filenames for training and eval
• training: Start training (creates .checkpoint files)
• traineddata: Create best and fast .traineddata files from checkpoints
• proto-model: Build the proto model
• tesseract-langdata: Download stock unicharsets
• evaluation: Evaluate .checkpoint models on eval dataset via lstmeval
• plot: Generate train/eval error rate charts from training log
• clean: Clean all generated files