tidytext R Package

The tidytext package follows tidy data principles where text data is structured such that:

• Each variable is a column.
• Each observation is a row.
• Each type of observational unit is a table.

Specifically, the tidy text format is defined as being one-token-per-document-per-row. This structure allows you to use standard tidy tools like dplyr, ggplot2, and broom for text mining and natural language processing tasks.

The tidytext package provides functionality to convert objects from popular text mining R packages into tidy formats. This allows you to use dplyr for filtering and processing before converting data into formats like document-term matrices for machine learning.

Supported conversion targets include objects from:

• tm
• quanteda

By using the tidy() function (consistent with the broom package pattern), you can switch between traditional text mining structures and tidy data frames for easier interpretation and visualization with ggplot2.

You can install the stable version of tidytext from CRAN, or the development version from GitHub using the remotes package.

# Install from CRAN
install.packages("tidytext")

# Install development version from GitHub
library(remotes)
install_github("juliasilge/tidytext")

Both MadanText and MadanTextNetwork fail installation due to an rJava loading error. This is caused by an architecture mismatch between the installed Java Virtual Machine (JVM) and the system architecture (e.g., trying to load an x86_64 library on an arm64 system).

Error Signature: .onLoad failed in loadNamespace() for 'rJava'... incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')

The package genius fails during the lazy loading stage of installation (both in Devel and CRAN environments). The error indicates a breaking change in the tibble package where the x argument to as_tibble() can no longer be missing.

Error Signature: Error : The x argument of as_tibble() can't be missing as of tibble 3.0.0.

The package benchdamic may trigger a NOTE during R CMD check regarding unused imports. Specifically, a namespace declared in the Imports field is not actually used in the code.

Error Signature: Namespace in Imports field not imported from: ‘microbiome’

The unnest_tokens() function converts a dataframe containing a text column into a "one-token-per-row" format. This is the core step for transitioning from raw text to a tidy dataset suitable for analysis with dplyr and ggplot2.

By default, it tokenizes text into words, but you can specify other token types such as characters, n-grams, sentences, lines, or paragraphs. It uses the tokenizers package under the hood.

library(tidytext)

# Assuming 'original_books' has a column named 'text'
tidy_books <- original_books |> 
  unnest_tokens(word, text)

Sentiment analysis in tidytext is typically performed by joining your tidy token dataframe with a sentiment lexicon obtained via get_sentiments().

Common lexicons include "bing" (which categorizes words as positive or negative). You can use an inner_join() to match words in your text to the sentiment scores in the lexicon.

library(tidytext)

# Get the Bing lexicon
bing_lexicon <- get_sentiments("bing")

# Join with your tidy data
janeaustensentiment <- tidy_books |> 
  inner_join(
    get_sentiments("bing"),
    by = "word",
    relationship = "many-to-many"
  )

Once text is tokenized, you can remove common stop words (words that carry little semantic meaning) by performing an anti_join() between your tidy token dataframe and the output of get_stopwords().

# Assuming 'tidy_books' is a one-token-per-row dataframe
tidy_books <- tidy_books |> 
  anti_join(get_stopwords())

If you are working with DocumentTermMatrix objects (from the tm package), you can convert them into a tidy, one-row-per-term data frame using the tidy() function from the broom package. The resulting dataframe will contain columns for document, term, and count.

library(tidytext)
library(tm)

# Assuming 'AssociatedPress' is a DocumentTermMatrix
tidy_dtm <- tidy(AssociatedPress)

To investigate specific failures or details regarding a package's reverse dependencies, use the revdep_details function from the revdepcheck package. This provides more granular information than the summary reports.

revdepcheck::revdep_details(, "package_name")