This course will focus on MLE-based models (Logit, Poisson, etc)
But this division is controversial
Inside political science there are two wolves
Economics
Causal inference, estimation of effects (Angrist and Pischke (2009))
“Use the simplest model that identifies the effect”
Statistics
Modeling the data-generating process (Gelman, Hill, and Vehtari (2020))
“Use models that match the structure of your data”
We will revisit this tension throughout the course—but you must choose your own path
Research is collaborative
We will practice working collaboratively
Git tracks every change you (and your colleagues) make—easy to keep track of changes, rewind, branch off
GitHub makes that history shareable
In this course:
You maintain a private repository for all your work
Assignments are pushed to GitHub, not Canvas
The final project lives in a shared repo with your partner
You will present as a group to the class
Not everything is worth learning anymore
AI reduces technical barriers to research, (arguably) unavoidable—BUT! can impede/erode learning
Big question → what should we learn, post-AI?
Hand off to AI
Boring, technical things
Git commands, package errors
Typesetting, formatting
ggplot aesthetics
Use with supervision
Important, difficult things
Data wrangling, cleaning
Analysis
Visualization
You must do this
Creative things
Model selection
Statistical reasoning
Research design
We scaffold AI use across the quarter: logistics only → directed analysis → full use on the final project
AI slop will be penalized → same as writing that is full of clichés
📝 In-class activity: pull → fix → push
Working as your coauthor, I made some crappy edits to your repo
On your machine, pull the change (have your AI agent do it, or do it yourself)
Open the file I edited locally, fix the mistake, and have your AI agent commit and push the fix
Verify on Github that your repo now shows two new commits: the mistake and the fix
A review of OLS
A working example
Question Does becoming wealthier tend to make people more conservative?
Premise 1 Citizens vote in their economic self-interest
Premise 2 The Democratic Party platform supports redistributive policies
Expectation Increasing income will shift support away from Democrats
The data: 2020 US county-level election results
County-level electoral data from the 2020 presidential election. Source:
name
state
per_dem_2020
income_10k
white_pct
black_pct
Autauga County
AL
27.02
5.37
78.1
18.4
Baldwin County
AL
22.41
5.02
87.3
9.5
Barbour County
AL
45.79
3.29
50.2
47.6
Bibb County
AL
20.70
3.64
76.3
22.1
Blount County
AL
9.57
4.41
96.0
1.8
Bullock County
AL
74.70
3.20
27.2
69.9
Our terminology
Row (blue) and column (red)
name
state
per_dem_2020
income_10k
white_pct
black_pct
Autauga County
AL
27.02
5.37
78.1
18.4
Baldwin County
AL
22.41
5.02
87.3
9.5
Barbour County
AL
45.79
3.29
50.2
47.6
Bibb County
AL
20.70
3.64
76.3
22.1
Blount County
AL
9.57
4.41
96.0
1.8
Bullock County
AL
74.70
3.20
27.2
69.9
Observations are rows in a dataset
Variables are columns
The unit of analysis defines what we are comparing → counties
What are we trying to estimate?
Outcome (blue), treatment (orange), and controls (grey)
name
state
per_dem_2020
income_10k
white_pct
black_pct
Autauga County
AL
27.02
5.37
78.1
18.4
Baldwin County
AL
22.41
5.02
87.3
9.5
Barbour County
AL
45.79
3.29
50.2
47.6
Bibb County
AL
20.70
3.64
76.3
22.1
Blount County
AL
9.57
4.41
96.0
1.8
Bullock County
AL
74.70
3.20
27.2
69.9
Treatment variable causes a change in outcome variable
We use control variables to (attempt to) identify this effect
Goal estimate the effect of county income on Democratic vote share, controlling for racial composition
What do we mean by “effect”? My view
Our theories are always causal, our evidence can only be interpreted causally under certain assumptions
We are trying to approximate the ideal → the experimental setting
Think of coefficient estimates as comparisons:
We interpret the regression slopes as comparisons of individuals that differ in one predictor while being at the same levels of the other predictors. (Gelman, Hill, and Vehtari 2020)
In our case: “when comparing two counties with the same racial demographics, counties with $10,000 higher median household incomes voted for Democrats at \(\beta\) more percentage points”
In this course, we will use two approaches to interpret effect estimates from models:
Simple calculus
Visualization
Takeaway Calculus gives us the effect formula; visualization shows us what that formula means in context
With OLS, the formula is just a number → visualization is optional. With other models, we need visuals
Using calculus to identify effects
We can use calculus to find the effect of income on vote choice (according to our model)
The first derivative of a function is the mathematical definition of rate of change: “if X increases, how much does Y change?”—this is exactly what we mean by “effect”
We only need two rules:
Power rule:\(\frac{d}{dx} c \cdot x^n = n \cdot c \cdot x^{n-1}\)
The second rule extends to any term that doesn’t involve \(x\) — including other variables like \(z\), which are just constants when we differentiate with respect to \(x\)
For a linear model, the effect equals the coefficient
Outcome: Democratic vote share (%)
(1)
+ p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001
Intercept
62.769***
(2.962)
Income ($10k)
4.769***
(0.190)
% White
-0.602***
(0.029)
% Black
0.022
(0.032)
Num.Obs.
3111
R2
0.423
The effect of wealth on vote share is simply the coefficient estimate
“when comparing two counties with the same racial demographics, counties with $10,000 higher median household incomes voted for Democrats at 4.8 more percentage points”
As long as \(X\) is untransformed (see: next week), the effect is constant across all values of \(X\) and \(\mathbf{Z}\)
This convenience won’t last
Transforming \(X\) (squaring it, logging it, interacting) changes this, even within OLS. It’s the specification that determines whether the effect is constant, not the estimator. (More on this next week)
In most MLE-based models we’ll cover later on (logit, Poisson, etc.), the effect of \(X\) will always depend on where you are in the data
This is why we will visualize
Approach 2: visualization
In this course, we will use two approaches to interpret models:
Basic calculus
Visualization
Visualizing effects (by hand)
A simple way to visualize the effect of \(X\) on \(Y\) is to compare the estimated value of \(Y\) across levels of \(X\)—these estimates are denoted \(\widehat{Y}\)
We can use these predictions to visualize the marginal effect of county income on Democratic vote share—the difference in vote share between counties of different income levels but similar control variable levels
visualize the distribution of X
geom_rug() will show you the distribution of income
The effect of income is always constant
Move anywhere along the line: a $10k increase in income always changes vote share by β = 4.77 percentage points
This is because income is linear (untransformed) in our model
Even with constant effects, visualization earns its keep
If the effect is simply \(\beta\) why visualize?
Even when the effect is constant, visualization shows you:
Where in the data you have coverage
The uncertainty across the range of \(X\)
Whether predictions go out of bounds
For most models, the effect is not just \(\beta\)—visualization will become necessary
Building the calculus + visualization habit now pays off when the models get harder
📝 Activity: constant across controls
We just showed the effect of income is the same no matter where you are in the income distribution—let’s show the same is true for controls
Steps:
Use crossing() to build a grid where both income and one control variable vary
Generate predictions with augment()
Plot the marginal effect of income with separate lines across your chosen control variable
We know what OLS estimates. Now the how—what problem is lm() actually solving?
There are two broad classes of modeling approaches:
Statistical theory make assumptions and use math to find the set of \(\beta\) that minimize the loss function - these typically have a closed-form solution—there is one set of \(\beta\) that minimize the loss function - that set can be mathematically derived
Computational / numerical approaches use constrained, educated guessing to identify the set of \(\beta\) - these typically do not have a closed-form solution—there is no way to nicely derive the set of \(\beta\) that minimize the loss function
OLS has a closed-form solution
We can use calculus (or matrix algebra) to find the values for \(\beta\) that will minimize the sum of squared residuals
Three steps: write the loss, differentiate, solve
The recipe for finding a minimum is the same one from calculus:
Write down the thing you want to minimize (the loss function)
Take the derivative with respect to the thing you’re choosing (\(\beta\))
Set it equal to zero and solve
If you can do these three steps, you can derive OLS.
Step 1: Write down the loss function
One predictor, with intercept. The model is \(\widehat{Y}_i = \beta_0 + \beta_1 X_i\)
The formula on the board and the number R gives you are the same thing—no black box.
Multiple predictors: same recipe, matrix bookkeeping
We derived \(\hat{\beta}_1\) for one predictor. With multiple predictors (\(\beta_0, \beta_1, \ldots, \beta_k\)), the recipe is the same—take partial derivatives, set to zero—but the system of equations grows
We use matrix algebra to write and solve that system compactly:
You don’t need to memorize this. The recipe is the same—the matrices are just bookkeeping for handling \(k\) unknowns at once. The point is that a closed-form solution exists.
The big picture
A model is a recipe for predicting \(Y\) from \(X\)—the coefficients (\(\beta\)) are the ingredients
OLS finds the \(\beta\) that minimize the sum of squared residuals—a closed-form solution exists
For linear models, the effect of \(X\) equals its coefficient—but this convenience disappears with transformations and non-linear models
We interpret effects using calculus (what’s the formula?) and visualization (what does it mean in context?)
This course: models where there is no closed-form solution—we’ll need to search for the optimum numerically (maximum likelihood)
Appendix: extra slides
(Cover only if time permits)
Why does the recipe work?
Hold \(\beta_0\) fixed and think of SSR as a function of \(\beta_1\) alone. Expand the square: