library(tidyverse)
library(broom)
library(modelsummary)
library(marginaleffects)
library(survival)
library(survminer)
library(pscl)
library(VGAM)
library(juanr)Problem Set 6: Duration Models
Due: Tuesday, May 12 by 11:59 PM. Push your completed .qmd and rendered .html to your GitHub repository.
Format: Write a self-contained Quarto document (.qmd) that renders to HTML. Your document should read as a short report — include brief prose explaining your reasoning, not just code and output.
Libraries you will likely need:
Question 1: A hurdle model for extramarital affairs
The focal predictor is marriage_rating — a self-reported rating of marital happiness (higher = happier). Controls are age, kids (whether the respondent has children), and years_married.
A quick look at affairs will show what makes this outcome interesting: most respondents report zero affairs, but among those who report any, the count can be substantial. That mixture — many zeros plus a long right tail — is the situation a hurdle model is designed for.
We didn’t cover hurdle models in lecture. The key idea is that your treatment variable might have a different effect on the participation (or extensive) margin — i.e., whether the person has an affair or not — than on the intensive margin — i.e., how many affairs they have.
- Theorize the two stages. Before fitting anything, write a short paragraph explaining why the process generating “zero vs. any affairs” might be different from the process generating “1 affair vs. many affairs” in this setting. What kinds of factors might shape the decision to ever have an affair? And then, once someone has decided to, what might drive how many? There’s no one right answer. Simply motivate why a two-part model is worth fitting here.
For both stages below, we’ll visualize the effect of marriage_rating by building a counterfactual grid with tidyr::crossing() (varying marriage_rating across its observed range while holding the other covariates at sensible values like means/modes), then adding predictions with predict(model, newdata = ..., type = "response"). We use predict() rather than broom::augment() or marginaleffects::predictions() because VGAM::vglm() (used in stage 2) doesn’t have methods for either of those — predict() is the lowest-common-denominator that works for both glm and vglm.
Fit the hurdle by hand: stage 1 (the hurdle). Create a binary version of the outcome — call it
any_affair— equal to 1 ifaffairs > 0and 0 otherwise. Fit a logistic regression ofany_affaironmarriage_rating,age,kids, andyears_married, using all of the data. Report the model. Then build a counterfactual grid withcrossing(marriage_rating = 1:5, ...)(holdingage,kids, andyears_marriedat sensible values), add predicted probabilities withpredict(...), and plot Pr(any affair) across the marriage-rating scale. What does this stage tell you?Fit the hurdle by hand: stage 2 (the count, given positive). Now fit the count stage on the subset of respondents who report at least one affair. The right model here is a zero-truncated Poisson, fit with
VGAM::vglm(..., family = pospoisson()). Fit the truncated-Poisson regression ofaffairsonmarriage_rating,age,kids, andyears_married. Report the model. Then repeat the samecrossing()+predict()workflow to plot the predicted count of affairs (given participation) across the marriage-rating scale. What does this stage tell you?
A standard Poisson PMF puts probability mass on \(y = 0\), but we’ve conditioned on \(Y > 0\) by subsetting the data — so the model’s likelihood needs to be renormalized to put zero mass on zero, otherwise the coefficients will be biased. (A plain glm(family = poisson) on the positives subset would silently use the wrong likelihood.)
Fit the hurdle in one step. The two-stage logic can be bundled into a single call with
pscl::hurdle(). Look through its help file to figure out how to use it. Show that the count-block coefficients match your stage-2 truncated Poisson, and the zero-block coefficients match your stage-1 logit.Substantive read. In a few sentences, summarize what the hurdle model tells you about the role of
marriage_rating. Does it operate primarily on the participation margin (whether any affair happens), the intensity margin (how many, given some), or both?
Question 2: Estimating the dispersion parameter by hand
In lecture we said that the Poisson model rests on a strong assumption: that the conditional variance equals the conditional mean, \(\text{Var}(Y_i \mid X_i) = \mathbb{E}[Y_i \mid X_i] = \mu_i\). When that assumption fails, Poisson SEs are too small and \(z\)-stats too large. The dispersion parameter \(\phi\) measures how badly this assumption is violated. In this question you’ll compute it by hand from a fitted Poisson and use it to diagnose the affairs model.
The Pearson dispersion statistic is:
\[\hat\phi \;=\; \frac{1}{n - k} \sum_{i=1}^n \frac{(y_i - \hat\mu_i)^2}{\hat\mu_i}\]
where \(n\) is the sample size and \(k\) is the number of estimated coefficients.
Fit Poisson and compute \(\hat\phi\) by hand. Refit the Poisson model from Question 1 (
n_affairs ~ marriage_rating + age + kids + years_married). Extract the fitted values \(\hat\mu_i\) withfitted(), compute the sum of squared Pearson residuals \(\sum (y_i - \hat\mu_i)^2 / \hat\mu_i\), divide by residual df (\(n - k\)), and report \(\hat\phi\). Show your code.Hint:
summary(mod)$dispersionreturns 1, not your estimate — R fixes \(\phi = 1\) by assumption when you specifyfamily = poisson. To test that assumption you have to compute \(\hat\phi\) yourself.Interpret. What does the \(\hat\phi\) you computed tell you about the affairs model? In one or two sentences, translate the number into a substantive statement about the conditional variance vs. the conditional mean.
Refit as NB and compare. Fit the same model with
MASS::glm.nb(). Put the Poisson and NB fits side-by-side in amodelsummary()table. Focus on the standard errors (not the coefficients): roughly how much do they change? Why are NB SEs larger, and why is that the correct answer when the data are overdispersed?Where is the overdispersion coming from? \(\hat\phi\) tells you the Poisson variance assumption is wrong, but not why. The two usual suspects are (1) more zeros than Poisson can accommodate given the implied \(\hat\mu_i\), and (2) a heavier right tail among the positives than Poisson allows. Compare the sum of observed zeros in your data against the sum of zeros your Poisson model expects. Hint:
sum(dpois(x, lambda = fitted(model)))will give you the sum of expected \(x\) across all your observations. Report both numbers. How big is the gap?
This question is optional. It is intended for students building a methods portfolio for the quantitative methods subfield.
Build the dispersion parameter from first principles. \(\hat\phi\) is the sample version of a population quantity: \[\phi \;=\; \frac{1}{\mu}\,\mathbb{E}\!\left[(Y - \mu)^2\right] \;=\; \frac{1}{\mu}\sum_{y=0}^{\infty} (y - \mu)^2 \, P(Y = y).\]
In other words, \(\phi\) is the probability-weighted average of \((y - \mu)^2\), divided by \(\mu\). We’re going to compute that sum by hand for three different distributions that all have the same mean \(\mu = 3\) but very different shapes, and watch \(\phi\) move around. Each part below asks for the same three-column table:
Build the table below for \(y = 0, 1, \ldots, 20\) (20 is enough — by then \(P(Y = y)\) is negligible for all three distributions). Fill in the three right-hand columns, sum the last column, and divide by \(\mu = 3\). That gives you \(\phi\) for the population from which a hypothetical dataset was drawn.
\(y\) \(P(Y = y)\) \((y - \mu)^2\) \(P(Y = y)\cdot(y - \mu)^2\) 0 ? ? ? 1 ? ? ? 2 ? ? ? The Poisson case (\(\phi = 1\)). Take \(Y \sim \text{Poisson}(\mu = 3)\). Compute \(P(Y = y)\) with
dpois(y, lambda = 3), build the table, and report your \(\phi\). You should land at exactly 1 (up to truncation error from stopping at \(y = 20\)). In one sentence: which rows of the table contribute most to the sum?The overdispersed case (\(\phi > 1\)). Now take \(Y \sim \text{NegBin}(\text{mean} = 3,\ \text{size} = 2)\). Compute \(P(Y = y)\) with
dnbinom(y, mu = 3, size = 2), rebuild the table, and report your \(\phi\). (Algebraically, \(\text{Var}(Y) = \mu + \mu^2/\text{size} = 3 + 9/2 = 7.5\), so you should get \(\phi = 2.5\).) Compare the \(P(Y = y)\) column to the Poisson one: which \(y\) values have more mass under NB than under Poisson? Connect that to why the sum gets bigger.The underdispersed case (\(\phi < 1\)). Now take \(Y \sim \text{Binomial}(n = 10, p = 0.3)\), which also has mean \(np = 3\). Compute \(P(Y = y)\) with
dbinom(y, size = 10, prob = 0.3), rebuild the table, and report your \(\phi\). (Algebraically, \(\text{Var}(Y) = np(1-p) = 2.1\), so \(\phi = 0.7\).) Why is the binomial underdispersed relative to Poisson? Look at the table: what’s \(P(Y = y)\) for \(y > 10\), and how does that compare to NB or Poisson?
Question 3: Censoring with real data
In lecture we used a simulation to show what right-censoring does to OLS: we generated a true spell length for each unit, censored at \(C\), and compared the “oracle” estimate to two naïve OLS approaches (clip at \(C\), or drop the censored). The simulation made the bias visible because we knew the truth.
We can play the same trick with the leader data without simulating anything. The data run through 2015, so we already know each leader’s true spell length. Pretend instead that data collection stopped at the end of 1991 — anyone still in office on January 1, 1992 is now right-censored at 1991, and we’ll compare what we’d estimate from that truncated view to what the full data say.
Get the data. The code below collapses
juanr::leader(one row per leader-year) into one row per leader-spell, builds a “truncated-at-1991” version of the duration and event indicator alongside the true (1840–2015) versions, and drops leaders whose spell starts after 1991. Run it as-is — the resultingleaderstibble has everything you need for the rest of the question.leaders = juanr::leader |> group_by(country, leader) |> summarise( start_year = min(year), end_year = max(year), duration = max(yr_office), # true spell length (1840–2015 view) rebel = first(rebel, order_by = year), .groups = "drop" ) |> drop_na(duration, rebel) |> filter(start_year <= 1991) |> # drop leaders whose spell begins after the cutoff mutate( event = as.integer(end_year < 2015), # event under the true 2015 cutoff duration_1991 = pmin(duration, 1991 - start_year + 1), # what we'd observe if data ended in 1991 event_1991 = as.integer(end_year <= 1991) # 1 if exited by 1991, 0 if still in office )Report (i) how many leader-spells you have and (ii) what share are right-censored under the 1991 cutoff vs. the 2015 cutoff.
Visualize the three views as histograms. Plot three histograms of leader spell durations, one for each of the views below, on a common x-axis so they are directly comparable. Fill or color the bars by
rebelso you can see how the censoring is hitting rebel vs. non-rebel leaders differently.- Oracle — the true (1840–2015) spell length for every leader
- Observed (clip at 1991) — what we’d observe if data collection ended in 1991, treating censored leaders as if they had exited
- Drop censored — the same as (2), but now we throw out the leaders who were still in office at the end of 1991
Run the three OLS regressions. Now run the analog of the histograms above as regressions: a regression of duration on
rebelfor each of the three views. Report the threerebelcoefficients side-by-side. Which way does each naïve estimate move relative to the oracle? Does the direction of the bias match what the lecture simulation predicted?Interpret. In a short paragraph, connect what you see in (b) and (c). Where in the histograms can you see the bias coming from — which kinds of leaders (rebel vs. non-rebel) are getting their spells truncated, and how does that map onto the direction of the bias in (c)? What does this exercise say about why we need the survival-analysis machinery (KM, exponential, Weibull, Cox) rather than just OLS on
duration?
Question 4: Does financial aid reduce recidivism?
For this question we’ll use the Rossi dataset (in the carData package), a classic experiment in criminology. 432 inmates were released from Maryland state prisons in the early 1970s and randomly assigned to receive either financial aid (a small weekly stipend during transition) or no aid. They were then followed for 52 weeks, and the outcome of interest is the time until first re-arrest. Anyone not re-arrested by week 52 is right-censored.
The variables you’ll need:
week— duration in weeks until first re-arrest (or 52 if never re-arrested)arrest— event indicator (1 = re-arrested during follow-up; 0 = censored at week 52)fin— the focal predictor;"yes"if assigned financial aid,"no"otherwise- Controls:
age(at release),wexp(had full-time work experience prior to incarceration),mar(married at time of release),paro(released on parole),prio(number of prior convictions)
Kaplan-Meier by financial aid. Plot KM curves for the aid (
fin == "yes") and no-aid (fin == "no") groups on the same axes. What does the picture say about whether financial aid extends time-to-rearrest at the descriptive level?Fit three parametric/semi-parametric models. Using
finas the focal predictor andage,wexp,mar,paro, andprioas controls, fit (i) an exponential AFT, (ii) a Weibull AFT, and (iii) a Cox PH model. Put all three side-by-side in amodelsummary()table.Interpret the
fincoefficient in each model. Write one or two sentences for each fit, translating the coefficient onfininto substantive language. Be explicit about the direction of the effect and the units it lives in (time ratio for the AFT models, hazard ratio for Cox), and note that the AFT and Cox parameterizations should give you the same substantive story with opposite signs.