Lecture 1 - Use in economics and pre-processing

Objective

Discuss some of the main uses of text as data in economics and describe the necessary steps to prepare your text data for analysis.

Summary

The ubiquity of text data and recent advances in text analysis allow us to study economic questions that were previously out of reach. The first part of this lecture describes some of the types of analyses that text data make possible. Before running any analysis, however, one needs to gather and pre-process the data. We discuss these steps in the second part of the lecture.

Session Outline

  1. Introduction
    • Specificity of text data
    • Why using text data in economics?
  2. Outline of the course and resources
  3. Applications in economics
    • Measuring document similarity
    • Concept detection
    • Relation between concepts
    • Associating text with metadata
  4. Workflow for analysis
  5. Gathering data
    • Common data sources and ways to gather data
    • Introduction to web scraping
    • Optical Character Recognition
  6. Pre-processing
    • Tokenization
    • Capitatilization and punctuation
    • Stemming/lemmatization
    • Stopwords

Materials

Open slides in html

Open slides in pdf

Exercise

The assignment is available on the Portail des Etudes.

Specific resources for this lecture

If you should read only one thing

Ash and Hansen (2023) gives a great overview of the use of text analysis in economics and is the reference the first part this lecture is built on

Gathering data

  • Curated lists of text data: here, here, here, and there
  • Web scraping: read the documentation and tutorials of the main webscraping package, Beautiful Soup in Python and rvest in R

Pre-processing

References

Ash, Elliott, and Stephen Hansen. 2023. “Text Algorithms in Economics.” Annual Review of Economics 15 (1): 659–88. https://doi.org/10.1146/annurev-economics-082222-074352.
Hvitfeldt, Emil, and Julia Silge. 2022. Supervised Machine Learning for Text Analysis in R.