Lesson 14: Model Building
With an emphasis on prediction
2026-05-13
Bias-variance trade off
Recall from 512/612: MSE can be written as a function of the bias and variance
\[ MSE = \text{bias}\big(\widehat\beta\big)^2 + \text{variance}\big(\widehat\beta\big) \]
No longer use MSE in logistic regression, BUT the idea between the bias and variance trade off holds!
For the same data, if our model has…
More covariates: less bias, more variance of coef. estimates
- Potential overfitting: with new data does our model still hold?
Less covariates: more bias, less variance of coef. estimates
- More bias bc more likely that were are not capturing the true underlying relationship with less variables
Think of it as a balance between flexibility and stability of our model
Visual of overfitting and underfitting data
From Data Science in a Box:
Step 1: Splitting data
Training: act of creating our prediction model based on our observed data
- Supervised: Means we keep information on our outcome while training
- Testing: act of measuring the predictive accuracy of our model by trying it out on new data
When we use data to create a prediction model, we want to test our prediction model on new data
- Helps make sure prediction model can be applied to other data outside of the data that was used to create it!
- So an important first step in prediction modeling is to split our data into a training set and a testing set!
Step 1: Splitting data
When splitting data, we need to be conscious of the proportions of our outcomes
Is there imbalance within our outcome?
We want to randomly select observations but make sure the proportions of No and Yes stay the same
We stratify by the outcome, meaning we pick Yes’s and No’s separately for the training set
- Side note: I took out some variables
Cross-validation (specifically k-fold)
Prevents overfitting to one set of training data
Split training set into folds that train and validate model selection
We can train and validate several times within the training set
![]()
Step 2: Fit LASSO: Main effects: Identify variables
# A tibble: 9 × 3
Variable Importance Sign
<chr> <dbl> <chr>
1 momfracYes 0.799 POS
2 priorfracYes 0.766 POS
3 rateriskGreater 0.644 POS
4 rateriskSame 0.414 POS
5 premenoYes 0.284 POS
6 armassistYes 0.267 POS
7 smokeYes 0.231 NEG
8 height 0.0365 NEG
9 age_c 0.0340 POS Looks like nothing is removed! (Importance would be 0)
Step 2: Fit LASSO interaction model: Variable importance
# A tibble: 44 × 3
Variable Importance Sign
<chr> <dbl> <chr>
1 momfracYes:rateriskSame 1.13 POS
2 priorfracYes 0.494 POS
3 premenoYes:rateriskGreater 0.476 POS
4 priorfracYes:armassistYes 0.354 POS
5 armassistYes:rateriskSame 0.285 POS
6 momfracYes:rateriskGreater 0.198 POS
7 height 0.0226 NEG
8 smokeYes:age_c 0.00272 POS
9 height:rateriskGreater 0.00162 POS
10 height:momfracYes 0.000409 POS
# ℹ 34 more rows
Step 3: Prediction on testing set
Area under the curve: 0.6672Why is this AUC worse than the one we saw with prior fracture, age, and their interaction?
- Did not split training and testing before
- Our testing set only has 100 observations!
- None of these things really predict fracture?
