DOCSEARCHbyAlgolia
Submit Docs

OpenCog Link Grammar

repository·master·Indexed 19 days ago

https://github.com/opencog/link-grammar

A linguistic tool and library that models natural language structure as a graph of typed links between words, supporting languages such as English, Thai, and Russian. It provides a Node.js interface via the LinkGrammar and Linkage classes, a link-parser CLI, and bindings for Python, Vala, Perl, and Ruby.

Tokens
25.2K
Snippets
66
Records
128
Agent score
64%

What's inside opencog-link-grammar

  1. Overview of Link Grammar components and directories

    master

    The Link Grammar repository is organized into several functional modules that handle the pipeline from raw text to parsed linkages:

    • Dictionary Handling: dict-common, dict-ram (RAM-based), dict-file (file-based), dict-sql (SQL DB), and dict-atomese (OpenCog AtomSpace).
    • Preprocessing: tokenize (converts strings to word-graphs) and prepare (prepares tokenized sentences for parsing).
    • Parsing: parse (the classic Sleator/Temperley/Lafferty parser) and sat-solver (an experimental Boolean-SAT based parser).
    • Analysis & Output: linkage (constructs/analyzes linkages), post-process (applies constraint rules and constituent-tree construction), and print (converts parses to human-readable strings).

  2. Overview of Link Grammar Parser capabilities

    master

    The Link Grammar Parser models the linguistic structure of natural languages (English, Thai, Russian, Arabic, Persian, and others) as a graph of typed links (edges) between words in a sentence.

    Key features include:

    • Syntactico-semantic depth: Provides more fine-grained information than conventional HPSG or dependency parsers.
    • Multi-language support: Includes robust support for English, Thai, and Russian.
    • Morphological analysis: Supports morphology, including alternative splittings for ambiguous words and suffix/stem connections (e.g., in Russian).
    • Thread-safety: Dictionaries can be updated at run-time while parsing is occurring.
    • Advanced features: Supports morphology, dialects, a fine-grained weight (cost) system, regex-based word class recognition, and random planar graph parsing.
    • Sentence Generation: Includes an experimental system (since v5.9.0) for generating grammatically valid sentences using a "fill in the blanks" API.

  3. AtomSpace Dictionary Encoding Format

    master

    The AtomSpace dictionary uses the concept of disjuncts to encode language data. These are stored as Atoms of type ConnectorSeq.

    Key Atom Types

    • ConnectorSeq: Represents a sequence of connectors.
    • Section: Represents word-disjunct pairings.

    Example Encoding

    To encode the phrase "level playing field", the Atomese format uses Section atoms to define how words connect:

    (Section
    (Word "level")
    (ConnectorSeq
        (Connector (Word "playing") (ConnectorDir "+"))))

(Section
    (Word "playing")
    (ConnectorSeq
        (Connector (Word "level") (ConnectorDir "-"))
        (Connector (Word "field") (ConnectorDir "+"))))

(Section
    (Word "field")
    (ConnectorSeq
        (Connector (Word "playing") (ConnectorDir "-"))))

    Grammatical Classes

    Membership in a grammatical class is denoted using a Member atom:

    (Member (Word "word") (WordClass "foo"))
    (Section
    (Word "level")
    (ConnectorSeq
        (Connector (Word "playing") (ConnectorDir "+"))))

  4. Link Grammar and Type Theory

    master
    Link Grammar is compatible with and can be converted to Categorial Grammar. The link types in LG form a type system that is easier to use than standard categorial grammar but remains mathematically equivalent. The diagrammatic approach used in LG is also closely related to the category theoretic approaches used in linguistics.

  5. Support multi-word structures and set phrases with 'holes'

    master

    Link Grammar supports complex multi-word structures, collocations, and set phrases that contain 'holes' (slots where other words can be inserted). These can be implemented using:

    1. Ordinary links: Using specific link types like V, XJ, and RJ.
    2. Post-processing rules: Using SF rules (e.g., the filler-it SF rules).

    Examples of structures with holes include:

    • Adpositions: ... from [xxx] on.
    • Conditionals: If ... then ... or If ... only ...
    • Correlatives: ... not only, ... but also ..., Either ... or ..., Both ... and ...
    • Relative structures: Someone ... who ...

  6. Use the SQLite-based dictionary for automated management

    master
    The SQLite-based dictionary provides a machine-readable interface for Link Grammar dictionaries. While traditional dictionaries are stored in hand-edited text files, the SQLite backend is designed to facilitate automated dictionary maintenance, such as updates driven by machine-learning algorithms or language-learning tools.

  7. Handle dialect and sociolect variations using cost vectors

    master
    Link Grammar supports handling different linguistic styles (like newspaper headlines vs. casual speech) by using cost vectors. Instead of hard-coding grammar rules to allow ungrammatical structures (like missing determiners in headlines), you can specify a dialect during the parse. This causes the parser to employ a specific set of link costs for that dialect during the ranking process. This allows for a more natural way to enable or disable specific parsing behaviors without polluting the core dictionary with high-cost null links.

  8. Understand Debugging and STDIO streams

    master

    The link-parser program directs messages to different streams based on severity:

    • stderr: Messages with severity Info and higher (Warning, Error, Fatal).
    • stdout: Messages with severity Debug and below (Trace, None).

    This separation ensures that debugging traces (sent to stdout) do not interfere with error reporting when stdout is redirected to a file. If you are debugging errors during a batch run, you may want to redirect stderr to the same file as stdout to preserve the chronological order of output.

  9. What are Uniformly Distributed Random Parse Trees

    master
    The any dictionary is designed to parse any language by exploring all combinatoric possibilities of word arrangements into a parse tree. The sampling method provides a uniform distribution in the space of parse trees, meaning each tree occurs with equal probability because they are exhaustively enumerated. This capability is specifically useful for certain machine learning tasks that require unbiased sampling of syntactic structures.

  10. Understanding Link Grammar output and linkage types

    master

    The parser output displays a graph of links between words. Each link type (e.g., Ss*b, Ost, WV, Wd, Xp) represents a specific grammatical relationship.

    • Link Types: These define how words connect (e.g., Ss*b might connect a verb and a singular subject; Ost might connect a verb and an object).
    • Disjuncts: The bottom of the output lists "disjuncts," which are the specific connectors used for each word. These act as a fine-grained form of "part of speech" (e.g., S- O+ indicates a transitive verb).
    • Cost: Each disjunct has a floating-point value representing its "cost," which roughly captures the log-probability of that specific grammatical usage.
    • Morphology: In languages like Russian, links like LL connect stems to suffixes, and MVA links may connect only to suffixes to capture syntactic structure.
    linkparser> This is a test!
	Linkage 1, cost vector = (UNUSED=0 DIS= 0.00 LEN=6)

+-------------Xp------------+
    +----->WV----->+---Ost--+
    +---Wd---+-Ss*b+  +Ds**c+   |
    |        |     |  |     |   |
LEFT-WALL this.p is.v a  test.n ! 

(S (NP this.p) (VP is.v (NP a test.n)) !)

LEFT-WALL    0.000  Wd+ hWV+ Xp+
               this.p    0.000  Wd- Ss*b+
                 is.v    0.000  Ss- dWV- O*t+
                    a    0.000  Ds**c+
               test.n    0.000  Ds**c- Os-
                    !    0.000  Xp- RW+
           RIGHT-WALL    0.000  RW-

  11. Understand the RAM-based dictionary implementation

    master

    The dict-ram directory provides methods for managing dictionary entries stored directly in RAM. This implementation serves two primary purposes:

    1. File-based dictionary support: It is used by dict-file to load the entire contents of dictionary files into memory upon startup, ensuring that subsequent lookups do not require further file I/O.
    2. Caching for remote dictionaries: It acts as a local cache for remote server dictionaries (such as dict-atomese). When a lookup is performed, the system first checks the local RAM cache; if the word is not found, it queries the remote server and then adds the result to the local RAM cache for future use.

  12. Thread Safety in Link Grammar

    master

    Link Grammar is safe to use in multi-threaded environments.

    • Shared Resources: Multiple threads can safely share the same dictionary.
    • Per-thread settings: Parse options can be set independently for each thread.
    • Global settings: The verbosity setting is the only global variable; it is shared by all threads.