DOCSEARCHbyAlgolia
Submit Docs

ConceptNet 5

repository·master·Indexed 25 days ago

https://github.com/commonsense/conceptnet5

A Python toolset for building the ConceptNet 5 multilingual semantic network and serving its web interfaces and JSON Web APIs. The library includes utilities for importing data from sources like WordNet, Wiktionary, and DBpedia, processing word vectors (GloVe, FastText, Word2Vec, Polyglot), and managing the knowledge graph database.

Tokens
3.6K
Snippets
2
Records
40
Agent score
84%

What's inside conceptnet5

  3. Access the Stanford Rare Word (RW) Similarity Dataset

    master
    The Stanford Rare Word (RW) Similarity Dataset is a collection of word pairs used for evaluating word representations. The dataset was originally collected by researchers at Stanford University and is described in the paper "Better Word Representations with Recursive Neural Networks for Morphology" (Luong et al., 2013).

  5. License and Attribution for ConceptNet 5 Data

    master

    The complete data in ConceptNet 5 is available under the Creative Commons Attribution-ShareAlike 4.0 license (CC-By-SA 4.0).

    When using ConceptNet 5 data in your work, it is suggested to use the following attribution text:

    This work includes data from ConceptNet 5, which was compiled by the
Commonsense Computing Initiative. ConceptNet 5 is freely available under
the Creative Commons Attribution-ShareAlike license (CC-By-SA 4.0) from
http://conceptnet.io.

The included data was created by contributors to Commonsense Computing
projects, contributors to Wikimedia projects, Games with a Purpose,
Princeton University's WordNet, DBPedia, Unicode, Jim Breen, MDBG, and
Cycorp's OpenCyc.

  6. Install and build ConceptNet

    master

    This Python package provides a toolset for building the ConceptNet 5 knowledge graph (including custom data) and serving its HTML interface and JSON Web API.

    Requirements for building:

    • A Unix command line (Ubuntu 16.04 is recommended)
    • Python 3.5 or later
    • 30 GB of RAM
    • Various other dependencies (refer to the build process documentation on the wiki for details)

  7. Understand the Stanford Rare Word (RW) Similarity Dataset format

    master

    The dataset file (rw.txt) contains 2034 pairs of words. Each line in the file is tab-delimited and follows this structure:

    word_pair_1 <tab> average_similarity_rating <tab> individual_rating_1 <tab> ... <tab> individual_rating_n (up to 10 individual ratings).

    Each line represents one pair of words and its associated similarity scores.

  9. Test ConceptNet libraries and build process

    master

    Use pytest to run tests for the ConceptNet libraries and a small-scale version of the build process. You can use specific flags to control the scope of the testing:

    • pytest: Runs standard tests.
    • pytest --quick: Re-runs tests quickly, assuming a small test database is already built.
    • pytest --fulldb: Runs additional tests against the fully built ConceptNet database.
    pytest
pytest --quick
pytest --fulldb

  10. Reference SimLex-999 column schema

    master

    The SimLex-999.txt file contains the following tab-separated columns:

    ColumnDescription
    word1The first concept in the pair.
    word2The second concept in the pair. (Note: Order only affects Assoc(USF) which is asymmetric; all other columns are symmetric).
    POSThe majority part-of-speech of the concept words (matching POS only).
    SimLex999The similarity rating, linearly mapped from [0,6] to [0,10].
    conc(w1)Concreteness rating of word1 (scale 1-7).
    conc(w2)Concreteness rating of word2 (scale 1-7).
    concQThe quartile the pair occupies based on concreteness ratings.
    Assoc(USF)Strength of free association from word1 to word2 (asymmetric).
    SimAssoc333Binary indicator if the pair is among the 333 most associated (based on Assoc(USF)).
    SD(SimLex)Standard deviation of annotator scores (low values indicate high agreement).