texthero

repository·master·Indexed 25 days ago

A Python package for efficient text mining and data analysis on large text-based datasets, designed to work seamlessly with Pandas Series. It provides tools for text cleaning, tokenization, TF-IDF representation, PCA dimensionality reduction, K-means clustering, and visualization (scatterplots, top words, and word clouds). The library also includes a texthero.nlp module for extracting named entities and noun chunks, and provides the Superheroes NLP Dataset for practicing NLP tasks.

Tokens: 9.1K
Snippets: 20
Records: 92
Agent score: 83%

What's inside texthero

Overview of the Texthero Python package
master
Texthero is a Python package designed for efficient text mining and data analysis on large text-based datasets. It is built to work seamlessly with Pandas, specifically targeting the processing of text corpora (collections of documents) rather than single elements.
Key characteristics include:
- Pandas-centric: It is designed to work with Pandas Series. Most functions accept a Pandas Series as an argument and return a Pandas Series, allowing for easy function chaining or appending to DataFrame columns.
- Exploratory focus: It is intended to be used as a tool for understanding underlying data before applying complex machine learning models like Transformers.
- Efficiency: Designed to handle large quantities of text data efficiently.
- Non-Neural Network: Texthero does not implement its own neural network solutions, instead acting as a complementary tool to libraries like PyTorch or TensorFlow.
https://github.com/jbesomi/texthero/blob/master/PURPOSE.md
Overview of the Superheroes NLP Dataset
master
The Superheroes NLP Dataset is a playground dataset designed for practicing NLP, text mining, and data analysis. It contains data scraped from the Superhero Database and includes information on 1,447 superheroes across 81 columns.
https://github.com/jbesomi/texthero/blob/master/dataset/Superheroes%20NLP%20Dataset/README.md
Visualize text data with texthero.visualization
master
The texthero.visualization module provides tools for visual analysis of text datasets. It includes functions for generating scatterplots, identifying top words, and creating word clouds.
https://github.com/jbesomi/texthero/blob/master/docs/source/visualization.rst
Use the texthero.nlp module for NLP tasks
master
The texthero.nlp module provides high-level functions for Natural Language Processing (NLP) tasks, specifically for extracting named entities and noun chunks from text. Use these functions to perform entity recognition and linguistic analysis on your datasets.
https://github.com/jbesomi/texthero/blob/master/docs/source/nlp.rst
Map words into vectors using texthero.representation
master
The texthero.representation module provides various algorithms to map text-based Pandas Series into vector representations or perform dimensionality reduction and clustering. Supported methods include TF-IDF, term frequency, PCA, NMF, and several clustering algorithms.
https://github.com/jbesomi/texthero/blob/master/website/docs/api-representation.md
Common usage patterns for Texthero
master
Texthero is primarily used to perform the following tasks on a text Pandas Series:
- Cleaning: Clean a text Pandas Series.
- Tokenization: Tokenize a text Pandas Series.
- Representation: Represent a text Pandas Series (Note: representation functions may return a Sparse Pandas Series due to sparsity).
- Benchmarking: Benchmark on simple models (e.g., Naive Bayes) to see if text preprocessing improvements affect model performance.
- Data Understanding: Understand text without the need for complex models like Transformers.
- Fact Extraction: Extract main facts from a Pandas Series.
https://github.com/jbesomi/texthero/blob/master/PURPOSE.md
Install Docusaurus
master
To set up a new Docusaurus website, ensure you have Node (>= 10.9.0) and Yarn (>= 1.5) installed. You can initialize a project by running the docusaurus-init script in your project's root directory. This will create docs and website directories.
```
npx docusaurus-init
```
https://github.com/jbesomi/texthero/blob/master/website/docs/getting-started-installation.md
Update Docusaurus version
master
To check for updates, navigate to the website directory and run yarn outdated docusaurus or npm outdated docusaurus. To upgrade to the latest version, use the following commands:
If you encounter errors after an upgrade, try clearing the Babel cache by running the server with the BABEL_DISABLE_CACHE=1 environment variable.
https://github.com/jbesomi/texthero/blob/master/website/docs/getting-started-installation.md
Perform stemming with hero.stem()
master
Use the hero.stem() method to perform stemming on text data. The method accepts a pd.Series and a stem parameter specifying the stemmer to use (e.g., "snowball").
Best Practice: For optimal results, call do_stem after using remove_punctuation. Stemming works more effectively when punctuation has been removed from the text first.
```
>>> text = "I love climbing and running."
>>> hero.stem(pd.Series(text), stem="snowball")
   0    i love climb and running.
   dtype: object
```
https://github.com/jbesomi/texthero/blob/master/website/docs/getting-started-preprocessing.md
Install Texthero via pip
master
Install the texthero package using pip. To upgrade to the latest version, use the -U flag.
```
pip install texthero

# To upgrade
pip install texthero -U
```
https://github.com/jbesomi/texthero/blob/master/website/docs/getting-started.md
Import the Superheroes NLP Dataset using pandas
master
You can load the dataset directly into a pandas DataFrame using the raw GitHub URL. This is the quickest way to start working with the data in a Python environment.
```
import pandas as pd

df = pd.read_csv("https://raw.githubusercontent.com/jbesomi/texthero/master/dataset/superheroes_nlp_dataset.csv")

df.head()
```
https://github.com/jbesomi/texthero/blob/master/dataset/Superheroes%20NLP%20Dataset/README.md
Run the Docusaurus example website
master
After initialization, you can run the local development server by navigating to the website directory and using yarn start or npm start. The site will typically be available at http://localhost:3000. The server includes LiveReload, so changes to files in the website directory will automatically refresh the browser.
```
cd website
yarn start
# or
npm start
```
https://github.com/jbesomi/texthero/blob/master/website/docs/getting-started-installation.md