NLP4J Java Library

The NLPDemo class serves as a demonstration entrypoint for using the NLP4J decoding capabilities. It demonstrates two primary ways to use the NLPDecoder:

1. Decoding a single string: Pass a raw string to the decode(String sentence) method to receive an array of NLPNode objects representing the linguistic structure.
2. Decoding a file to an output stream: Use decode(InputStream inputStream, PrintStream printStream, String format) to process an entire file and stream the results directly to a destination (like System.out).

Note that the decoder requires a configuration file (typically an XML file) to initialize the linguistic models and pipeline.

// Initialize decoder with a configuration file
NLPDecoder nlp4j = new NLPDecoder(IOUtils.createFileInputStream("path/to/config-decode-en.xml"));

// Option 1: Decode a single sentence
NLPNode[] nodes = nlp4j.decode("John bought a car for Mary.");

// Option 2: Decode an entire file to standard output using a specific format
nlp4j.decode(IOUtils.createFileInputStream("path/to/input.txt"), System.out, AbstractNLPDecoder.FORMAT_RAW);

The RadiologyDecode class serves as a programmatic entry point for decoding radiology-specific data using the NLPDecoder. It iterates through a directory of input files (e.g., .txt) and applies decoding logic to produce output files (e.g., .tsv) in a specified format.

To use this logic, you need:

1. A configuration file (XML format) passed to the NLPDecoder constructor.
2. An input directory containing the files to be processed.
3. An input file extension to filter files.
4. An output file extension for the results.
5. An output format, such as AbstractNLPDecoder.FORMAT_LINE.

// Configuration and paths
final String configFile   = "/path/to/config-decode-deident.xml";
final String inputDir     = "/path/to/radiology/input";
final String inputExt     = "txt";
final String outputExt    = "tsv";
final String outputFormat = AbstractNLPDecoder.FORMAT_LINE;

// Initialize the decoder with the config file
NLPDecoder nlp4j = new NLPDecoder(IOUtils.createFileInputStream(configFile));

// Process files in the input directory
for (String inputFile : FileUtils.getFileList(inputDir, inputExt))
{
    String outputFile = inputFile + "." + outputExt;
    nlp4j.decode(
        IOUtils.createFileInputStream(inputFile), 
        IOUtils.createFileOutputStream(outputFile), 
        outputFormat
    );
}

The TokenizeIt class is a utility designed to process TSV files containing tokens and their features. It uses an EnglishTokenizer to segmentize tokens and an NLPDecoder to decode them into an NLPNode array based on a provided configuration file.

Key Functionalities:

• Batch Conversion: The convert(String inputDir, String outputDir) method processes all .tsv files in a specified input directory and saves the results to the output directory.
• Stream Processing: The convert(InputStream in, OutputStream out) method allows for processing individual input streams.
• Validation: The check(NLPNode[] nodes) method performs a specific validation check on the decoded nodes, looking for inconsistencies between the lemma (e.g., "it" or "its") and the associated features.

Input Format

The input TSV files are expected to have a format where each line contains a token form and its features separated by a tab: [token_form]\t[features]

// Example of manual instantiation and batch conversion
String configFile = "path/to/config-decode-en.xml";
String inputDir = "/path/to/input_tsv_dir";
String outputDir = "/path/to/output_dir";

TokenizeIt tokenizer = new TokenizeIt(configFile);
tokenizer.convert(inputDir, outputDir);

The CSVSentiment class provides utilities for processing sentiment-labeled CSV files. It can convert CSV data into different formats (TSV or TXT) or perform NLP decoding to categorize sentiment.

Input CSV Format Requirements: Based on the implementation, the class expects a CSV where:

• The first column (index 0) contains the sentiment label (e.g., positive, negative, or objective).
• The seventh column (index 6) contains the text content to be processed.
• The first row is treated as a header and is skipped.

Available Methods:

• toTSV(String inputFile): Converts the CSV to a Tab-Separated Values format. Each line in the output follows the pattern: [integer_label]\t[tokenized_text]. Labels are mapped as follows:
  • positive $\rightarrow$ 0
  • negative $\rightarrow$ 1
  • objective $\rightarrow$ 2
• toTXT(String inputFile): Extracts only the text content (column 6) from the CSV and saves it to a .txt file.
• categorize(String inputFile): Uses an NLPDecoder to process the text. It attaches the sentiment label as a feature (sent) to the NLP nodes and saves the decoded structure to individual .nlp files for each record.

// Example: Converting a CSV to TSV format
CSVSentiment cvs = new CSVSentiment();
cvs.toTSV("path/to/input.csv");

// Example: Categorizing sentiment using a configuration file
CSVSentiment cvs = new CSVSentiment("path/to/config-decode-en.xml");
cvs.categorize("path/to/input.csv");

The tokenize method processes a radiology CSV file by segmenting the text in the first column of each record using an EnglishTokenizer.

Key behaviors:

• Input Format: Expects a CSV file where the first column (index 0) contains the text to be tokenized.
• Output Generation: For each record in the CSV, a new .txt file is created in the same directory as the input file. The files are zero-padded (e.g., 0001.txt, 0100.txt) based on the outputStart index.
• Text Normalization: During the printing process, the tool applies regex patterns to specific radiology headers (defined in BEFORE and AFTER arrays) to ensure consistent newline placement. It also trims whitespace and removes empty lines.
• Parameters:
  • inputFile: The path to the source CSV file.
  • outputStart: An integer used to offset the starting filename index for the generated text files.

CSVRadiology cvs = new CSVRadiology();
// Tokenizes the CSV and starts naming output files from index 500
cvs.tokenize("/path/to/radiology_reports.csv", 500);

The AmbiguityClassGenerator class provides methods for reading training data and exporting the resulting ambiguity vocabulary.

readTrain(Bigram<String,String> bigram, String dirpath, boolean uncaptialized)

Reads training files from the specified directory and populates the provided bigram object.

• dirpath: Directory containing training files. Files are processed using FileUtils.getFileList(dirpath, "*").
• uncaptialized: If true, words are converted to a simplified form using StringUtils.toSimplifiedForm.
• File Format: Expects tab-separated files where the word is at index 1 (t[1]) and the tag is at index 3 (t[3]).
• Filtering: Automatically skips forms containing underscores (_) or non-ASCII characters (>= 128).

printVocab(Bigram<String,String> bigram, String outputFile, int cutoff, double threshold)

Processes the collected bigrams to generate a vocabulary map and writes it to a file.

• cutoff: Only words with a total unigram count $\ge$ cutoff are considered.
• threshold: Uses unigram.toList(threshold) to filter associated tags.
• Exclusion Rule: If a word has only one tag and that tag is NNP or NNPS, it is excluded from the output.
• Output Format: A serialized Map<String, List<String>> where the key is the word and the value is a list of its possible tags, sorted by frequency. The file is written using IOUtils.createObjectXZBufferedOutputStream.