Workflow

quanteda has three basic types of objects:

  1. Corpus

    • Saves character strings and variables in a data frame
    • Combines texts with document-level variables
  2. Tokens

    • Stores tokens in a list of vectors
    • More efficient than character strings, but preserves positions of words (string-of-words)
  3. Document-feature matrix (DFM)

    • Represents frequencies of features in documents in a matrix
    • The most efficient structure, but it does not have information on positions of words (bag-of-words)

Text analysis with quanteda goes through all those three types of objects either explicitly or implicitly.

graph TD D[Text files] V[Document variables] C(Corpus) T(Tokens) AP["Positional analysis (string-of-words)"] AN["Non-positional analysis (bag-of-words)"] M(DFM) style C stroke-width:4px style T stroke-width:4px style M stroke-width:4px D --> C V --> C C --> T T --> M T -.-> AP M -.-> AN

For example, if character vectors are given to dfm(), it internally constructs corpus and tokens, before a DFM.