Transform data to be able to use it in algorithms

Main data structures for documents in NLP:

• As raw text
• As a sequence of tokens for each document
• As a vector, stored in a matrix (document-term matrix, embedding matrix)

Massive matrices (\(n_{docs} \times n_{vocab}\))

Even if sparse (contains mostly zeros), it might take up a lot of RAM to run algorithms on these

Plus, we often do not care about the words in a document but about the underlying meaning

Want vectors to be able to capture meaning

Reduce the dimensionality of the matrices to capture this meaning and lower computing load:

• Apply Principal Component Analysis (PCA) to the matrix: called Latent Semantic Analysis
• Latent Dirichlet Allocation (LDA)
• Word embeddings: many approaches. We will discuss them a bit later.

After transformation, documents are represented as sequences of tokens or as vectors

Can now use these representations for our intended tasks

Compute document similarity, for instance by calculating cosine between two document-vectors (cosine-similarity)

Identify presence of concepts:

• Count occurrences of words or dictionaries, build RegEx, train a ML algorithm, etc
• Identify clusters of documents (ie of vectors)
• Some of theses tasks only involve matrix products

Methods that use predefined lists of words or phrases for analysis

Applications:

• Count (co-)occurrences of certain words/dictionaries
• Compute metrics based on values associated to each gram (eg sentiment score)
• Classify text into categories defined by dictionaries

Match your text data with the dictionary, typically with RegEx

Defining your dictionary:

• Using external sources

• Using domain expertise (pre-existing topic specific dictionaries)

• Choosing terms depending on how well they predict human-annotation

Use of pre-existing dictionaries:

• AFINN, NRC, bing for sentiment analysis

• WordNet (synonyms, antonyms, etc)

• LIWC (Linguistic Inquiry and Word Counts): words sorted into categories (eg anger, familly, etc)

Expanding dictionaries:

• With words that are close by in an embedding space
• Use LLMs

Project documents into a vector space

Building dense matrices, ie non-sparse

Uses the context

• Words that often occur jointly will be close in the embedding space

• eg we probably want pen and pencil to be relatively close in the space

Captures relations between words

“Classic” embeddings do not take polysemy into account: collapse each word to one vector

We are going to talk about more complex, deep learning based, embeddings later in this session

Ready to use, capture relationships between words in common language

Trained on large corpora (Common Crawl, Wikipedia)

Different algorithms: FastText (Facebook), word2vec (Google), GloVe (Stanford)

Computation methods:

• CBOW (Continuous Bag Of Words), Skip-gram
• Basically, compute words that co-occur in a \(k\)-words window

Will capture the relationships between words in your corpus

Take specificities of your corpus into account

Need a lot of text data to train properly train your own embeddings

Use the algorithms (FastText, word2vec, GloVe) and methods (CBOW, skipgram) described above to train you embedding

Specify:

• Desired dimension of the embedding space
• Number of words in the training window

Use unlabeled data \(\neq\) many of the algorithms discussed in class

Do not try to fit to make the model predict a “ground truth”

For text data, unsupervised ML is mostly used for topic models :

• They infer latent topics in the corpus
• Supervised: define topics, write algorithms to predict which category they belong to
• Semi-supervised: give anchor-words (eg CorEx)
• Unsupervised: let the algorithm discover topics by itself (eg LDA)

Every document is a mixture of topics

• eg doc 1 is 20% topic A, 80% topic B

Every topic is a mixture of words

• eg topic A is composed of words “banana”, “apple”, etc

LDA estimate both at the same time, it is probabilistic ML

Only one parameter to specify: the number of topics (\(k\))

LDA + Metadata

Includes contextual information by:

• Making topic prevalence vary with metadata, eg left wing parties talk more about inequality than right wing parties

• Topic content can vary with metadata, eg left wing parties talk more about education inequality than right wing parties

Bidirectional Encoder Representation from Transformers

Introduced by Google in 2018

Masked Language Model: trained by trying to discover words that are randomly masked

Good at text classification, answering questions

Mainly focused on understanding language

Generative Pretrained Transformer

Introduced by OpenAI in 2018

Good at text generation, having conversations

Autoregressive Language Modeling: trained to predict the next word

Mainly focus on generating language

Very good at certain tasks

Easily accessible but might be expensive to run

May not be a good option for sensitive private data (need to share data)

Can use models installed locally

Computationally intensive

Not that interpretable (black-boxy) but can be validated against hand-labelled data

May not be very robust to prompt variations + non-replicable

Need to transform unstructured text data to numeric format (typically to sequences of grams or matrices)

Embeddings and dense vector representations allow to capture relations between words

These representations allow to apply:

• Direct analyses on these representations, eg similarity analyses via cosine-similarity
• Supervised ML models learned in the rest of the class
• Unsupervised ML algorithms such as topic modeling (eg LDA)

Recent developments in NLP allow to compute context-specific representations and to implement more complex analyses

Typically, NLP allows to build new metrics to plug into econometrics models

There are many steps involved in text analysis

Gathering data may be one of the most time consuming ones

Once we have transformed our text data, we can apply other typical ML tools to these representations

There are currently tons of developments in NLP, hard to keep up

Nowadays, we can basically do anything we want on text data \(\Rightarrow\) importance of identifying good research questions