library(tidyverse)
library(broom)
library(modelsummary)
library(marginaleffects)
library(estimatr)
library(juanr)Problem Set 3: Binary Outcomes and the Logit Model
Due: Tuesday, April 21 by 11:59 PM. Push your completed .qmd and rendered .html to your GitHub repository.
Format: Write a self-contained Quarto document (.qmd) that renders to HTML. Your document should read as a short report — include brief prose explaining your reasoning, not just code and output.
Libraries you will likely need:
Question 1: Your own data
Browse Data Is Plural and find a dataset that interests you. Your outcome needs to be binary — either a 0/1 variable that’s already in the data, or a variable you can convert to 0/1 in a way that’s substantively meaningful (for example, a continuous vote share can become “did candidate X win”). Download the data and place it in a data/ folder in your class folder.
Describe the dataset briefly: what are the observations? What variables are available? How big is it?
Identify your binary outcome (\(Y\)), a binary treatment (\(X\)), and at least one control variable. If your natural treatment is continuous, dichotomize it in a way that makes substantive sense. Tell a short causal story: why might \(X\) change the probability of \(Y = 1\)?
Compute the difference-in-means estimator for your treatment by hand and standard error of that difference using the two-sample formula from lecture. Describe what you find.
Now fit two models of \(Y\) on your treatment and control: a linear probability model, and a logistic regression. Present both in a regression table. Confirm that the treatment coefficient from the LPM matches your by-hand difference from (c).
Using your logit model, generate predicted probabilities of \(Y = 1\) at the two levels of your treatment, holding the controls at a reasonable value. Report the two predicted probabilities and interpret their difference.
Compute the marginal effect of your treatment on \(\Pr(Y=1)\) across the range of one control variable, and plot it with confidence bounds.
Report the average marginal effect of your treatment under the logit model. How does it compare to the LPM’s coefficient on the treatment from (d)? Why might the two numbers be similar?
Question 2: Classical vs. robust standard errors
In lecture we argued that robust (HC2) standard errors should be the default for the linear probability model because binary outcomes are always heteroskedastic. But the choice between classical and robust SEs isn’t always one-sided. Let’s compare them directly on real data, then use a simulation to see a case where classical SEs are actually better.
Shared data. For parts (a)–(b), use the rebel_leader dataset from {juanr}. This is a cross-section of leaders of armed rebel groups. Our outcome of interest is whether a leader was killed. You’ll need to construct the outcome yourself: a leader counts as killed == 1 if they are deceased (dead == 1) and their death_cause is one of KIA, Assassinated by government, Assassinated by rival group, Assassinated by external state, or Executed. Living leaders (dead == 0) and those who died of other causes (disease, accident, suicide, etc.) count as killed == 0. Drop observations where dead is missing, or where dead == 1 but death_cause is missing — in those cases you can’t tell whether the leader was killed. Also drop observations with missing values on any of the three predictors below.
Your predictors are birth_year, western (educated in the West), and train_abroad (received military training abroad).
Fit a single linear probability model of
killedon all three predictors. For each of the three predictor coefficients (you can ignore the intercept), report both the classical SE and the HC2 robust SE.Compute the percent difference between the HC2 and classical SEs for each predictor — i.e., \(100 \times (\text{SE}_{\text{HC2}} - \text{SE}_{\text{classical}}) / \text{SE}_{\text{classical}}\). Are the robust SEs systematically bigger than, smaller than, or about the same as the classical ones?
Simulation: a case where classical SEs win.
Robust SEs are consistent — with enough data they converge to the truth — but they have poor finite-sample properties. In small samples with genuinely homoskedastic errors, robust SEs can produce confidence intervals that under-cover the true parameter.
Generating one dataset from the linear model below and fitting lm() to it is something you can write by hand using rnorm(), runif(), and lm(). The trickier part is doing it 2,000 times and collecting the results into a tidy data frame. You are welcome to use AI assistance for the repetition part.
If you’d rather work it out yourself, here’s a hint: in the tidyverse, the natural tool for “do this same thing N times” is purrr::map() (or map_dfr() / map() |> list_rbind()), which applies a function to each element of a vector. You write a function that runs one iteration and returns a one-row tibble with the quantities you care about (point estimate, classical SE, HC2 SE); then map() over 1:2000 and stack the results.
This problem calls for repeating the same process (fitting a model to randomly generated data) 2,000 times. Here’s how I approach any task that requires repetition.
First, nail down the thing I want to happen. For example, pick 10 rows at random, take the average:
library(tidyverse)
library(juanr)
bot |> sample_n(size = 10) |> summarise(avg = mean(age, na.rm = TRUE))# A tibble: 1 × 1
avg
<dbl>
1 51.7
Next, turn this into a function that executes what I want in one swoop. Notice that I’ve replaced bot with a generic data argument that allows me to run this function on any dataset (so long as that data has a variable called age!)
# write function
get_10 = function(data) {
data |> sample_n(size = 10) |> summarize(avg = mean(age, na.rm = TRUE))
}
# run once
get_10(bot)# A tibble: 1 × 1
avg
<dbl>
1 56.6
Now, use map() or an apply() function to map this function onto one of 10 runs:
results = crossing(runs = 1:10) |>
mutate(average = map(runs, ~ get_10(bot))) |>
unnest(average)
results# A tibble: 10 × 2
runs avg
<int> <dbl>
1 1 47.5
2 2 40.7
3 3 56.1
4 4 55
5 5 47.9
6 6 46.9
7 7 54.9
8 8 45
9 9 38.9
10 10 40.2
Write a simulation that generates a small dataset (\(n = 30\)) from a linear model with homoskedastic errors:
\[X_i \sim \text{Uniform}(0, 10), \qquad \varepsilon_i \sim N(0, 1), \qquad Y_i = 2 + 0.5 \cdot X_i + \varepsilon_i\]
Fit a linear model of \(Y\) on \(X\) and extract both the classical SE and the HC2 robust SE for \(\hat\beta_1\). Repeat 2,000 times and collect the results.
For each iteration, construct a 95% confidence interval from each SE type (point estimate \(\pm 1.96 \times\) SE).
For each iteration, create a column that tells you whether the true value \(\beta_1 = 0.5\) is in the confidence interval from (d). Report the percent of the time that the CI covers the true value (the coverage rate) for classical SEs and for HC2 SEs. A well-calibrated 95% CI should cover the truth 95% of the time.
Interpret: which SE produced better-calibrated CIs at \(n = 30\)?
Question 3: Decomposing a standard error
In lecture we focused on the SE of \(\beta\) when our model only has one predictor, but noted that the SE of \(\beta\) when there are multiple predictors in the model takes the form: \[\text{SE}(\hat{\beta}_j) \;=\; \sqrt{\frac{\hat{\sigma}^2_{\varepsilon}}{\sum_i (X_{ij} - \bar{X}_j)^2 \, \cdot \, (1 - R^2_j)}}\]
where the denominator is the residual variance of \(X_j\) — the part of \(X_j\) that isn’t already explained by the other predictors in the model. Let’s compute this ourselves for a concrete case, and watch what happens to the SE when we add a control.
Data setup. Use the affairs dataset from {juanr} (Fair 1978; the same data we looked at in lecture). You can use the code below to prepare the data:
library(juanr)
library(tidyverse)
affairs_clean = affairs |>
mutate(had_affair = as.numeric(n_affairs > 0)) |>
select(had_affair, kids, years_married)You’ll compare two LPMs: one with only kids, and one with kids + years_married.
Bivariate model. Fit a linear probability model of
had_affaironkidsalone. Report \(\text{SE}(\hat{\beta}_{\text{kids}})\) from the standardlm()output.Compute the same SE by hand. Extract the residual variance \(\hat{\sigma}^2_{\varepsilon}\) from the model, compute \(\sum_i (\text{kids}_i - \overline{\text{kids}})^2\) (the sum of squared deviations of the treatment around its mean), and plug them into
\[\text{SE}(\hat{\beta}_{\text{kids}}) \;=\; \sqrt{\frac{\hat{\sigma}^2_{\varepsilon}}{\sum_i (\text{kids}_i - \overline{\text{kids}})^2}}\]
Hint: to get \(\hat\sigma^2_\varepsilon\), use
augment(mod) |> summarize(SSR = sum(.resid^2), ...)as in lecture.Verify that your by-hand SE matches the one from
lm().Multivariate model. Now add
years_marriedto the LPM. Report the new \(\text{SE}(\hat{\beta}_{\text{kids}})\) fromlm().Compute the new SE by hand. The residual variance \(\hat{\sigma}^2_{\varepsilon}\) comes from the multivariate LPM exactly as in (b). The denominator is the new part: it turns out to be the sum of squared residuals from regressing
kids ~ years_married— i.e., the parts ofkidsthatyears_marriedcan’t explain.- Regress
kidsonyears_married. Save the residuals. - Sum the squared residuals — call this \(\text{SS}_{\text{kids}}\).
- Compute \(\text{SE}(\hat{\beta}_{\text{kids}}) = \sqrt{\hat{\sigma}^2_{\varepsilon} \,/\, \text{SS}_{\text{kids}}}\).
Verify this matches
lm().- Regress
Compare the two standard errors and the inputs used to construct them.
- Did \(\hat{\sigma}^2_{\varepsilon}\) increase or decrease when you added
years_married? Why? - How much of the variation in
kidsdoesyears_marriedexplain? (What is \(R^2\) from thekids ~ years_marriedregression in (d)?) - Net effect: did \(\text{SE}(\hat{\beta}_{\text{kids}})\) get bigger, smaller, or stay about the same between (a) and (c)?
Question 4: Derive the logit’s marginal effect
This question is optional. It is intended for students building a methods portfolio for the quantitative methods subfield.
In lecture we took as given that the derivative of the sigmoid is
\[\frac{d\sigma}{dz} \;=\; \sigma(z)\,(1 - \sigma(z)) \;=\; p\,(1 - p)\]
Let’s derive this from scratch, then use it to re-derive the logit marginal-effect formula for a multi-predictor model.
Same as PS1: wrap your math in $...$ for inline or $$...$$ for display math. Or write it out on paper and include a photo in your repo.
If \(f(z) = \dfrac{u(z)}{v(z)}\), then
\[\frac{df}{dz} \;=\; \frac{u'(z)\,v(z) \;-\; u(z)\,v'(z)}{v(z)^2}\]
where \(u'(z)\) and \(v'(z)\) are the derivatives of \(u\) and \(v\) with respect to \(z\). A mnemonic: “low d-high minus high d-low, over low squared” — “low” is the denominator, “high” is the numerator, “d-” means “derivative of.”
Worked example. For \(f(z) = \dfrac{z^2}{1 + z}\): take \(u = z^2\) (so \(u' = 2z\)) and \(v = 1 + z\) (so \(v' = 1\)). Plugging in:
\[\frac{df}{dz} \;=\; \frac{(2z)(1+z) \;-\; (z^2)(1)}{(1+z)^2} \;=\; \frac{z^2 + 2z}{(1+z)^2}.\]
Starting from \(\sigma(z) = \dfrac{1}{1 + e^{-z}}\), compute \(\dfrac{d\sigma}{dz}\).
Hint: Use the quotient rule with numerator \(u = 1\) and denominator \(v = 1 + e^{-z}\). Remember that \(\dfrac{d}{dz}\,e^{-z} = -e^{-z}\) (chain rule: derivative of \(e^{g(z)}\) is \(e^{g(z)} \cdot g'(z)\), and here \(g(z) = -z\) so \(g'(z) = -1\)).
Your derivative from (a) should look something like \(\dfrac{e^{-z}}{(1 + e^{-z})^2}\). Show that this expression equals \(\sigma(z) \cdot (1 - \sigma(z))\).
Hint: Write \(\sigma(z)\) and \(1 - \sigma(z)\) as fractions with the same denominator, then multiply them.